什么是SystemTap以及如何使用它？

环境

• Red Hat Enterprise Linux (all versions)

问题

• What is SystemTap and how to use it?

决议

SystemTap is an innovative tool which allows for simplified information gathering on the running Linux kernel. The purpose of using SystemTap is to obtain information on either performance issues or functional problems (bugs). By using SystemTap,developers and system administrators can debug problems and gather profiling and performance data without having to create and install instrumented kernels or other packages.

In essence,SystemTap provides the infrastructure (a command line interface and scripting language) needed to gather information. The actual job that SystemTap performs relies on user-developed scripts tailored to a specific purpose. Currently,there are a number of example SystemTap scripts pre-made for general use.

The operation of SystemTap is quite simple. The?stap?command takes as an argument the name of a SystemTap file (called a probe). There may be additional command line arguments passed depending on the probe. SystemTap translates the probe into C and compiles the C source as a kernel module. It then inserts the resulting kernel module into the running kernel to perform the probe functions defined in the script. The output is printed to the console,or can be redirected to file.

Requirements

Because SystemTap compiles scripts from C code and launches probes for kernel instrumentation,it requires several packages in order to function. See the?Installing SystemTap?chapter of the?SystemTap Beginners Guide?for detailed installation instructions.

The easiest way to satisfy the requirements is to simply?subscribe the system to the relevant debuginfo channels?in RHN,then run the following commands which should set up the environment for SystemTap:

yum install systemtap
stap-prep

To set the environment up manually,in addition to the?systemtap?package,the following packages must also be installed:

• kernel-devel?for the running kernel
• kernel-debuginfo?for the running kernel
• kernel-debuginfo-common?for the running kernel
• gcc
• systemtap

For example,for a?2.6.32-71.18.2.el6.x86_64?kernel you‘ll need:

kernel-debuginfo-2.6.32-71.18.2.el6.x86_64
kernel-debuginfo-common-x86_64-2.6.32-71.18.2.el6.x86_64
kernel-devel-2.6.32-71.18.2.el6.x86_64

As well as the?gcc?and?systemtap?packages.

The?gcc?and?kernel-devel?packages are available on Red Hat Network and can be installed using?yum?(RHEL 5 onwards).

Legacy

On RHEL 4,use?up2date. For Red Hat Enterprise Linux (RHEL) 4,the?kernel-debuginfo?package is not available via?up2date?and must be installed from the ISO image available on Red Hat Network or from?ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4AS/en/os/Debuginfo/?(substitute?4AS?in the URL with?4WS,?4ES,or?4Desktop,depending on the variant of of RHEL you have installed. Note: Red Hat Enterprise Linux 4 does not have?kernel-debuginfo-common?package.

For RHEL 5,you can download the?kernel-debuginfo?and?kernel-debuginfo-common?packages from?ftp://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/.

Using Systemtap

You can verify the SystemTap environment with the?stap-report?command.

An example script is shown below:

#! /usr/bin/env stap

# Using statistics and maps to examine kernel memory allocations

global kmalloc

probe kernel.function("__kmalloc") {
        kmalloc[execname()] <<< $size
}

# Exit after 10 seconds
probe timer.ms(10000) { exit () }

probe end {
        foreach ([name] in kmalloc) {
                printf("Allocations for %sn",name)
                printf("Count:   %d allocationsn",@count(kmalloc[name]))
                printf("Sum:     %d Kbytesn",@sum(kmalloc[name])/1000)
                printf("Average: %d bytesn",@avg(kmalloc[name]))
                printf("Min:     %d bytesn",@min(kmalloc[name]))
                printf("Max:     %d bytesn",@max(kmalloc[name]))
                print("nAllocations by size in bytesn")
                print(@hist_log(kmalloc[name]))
                printf("-------------------------------------------------------nn");
        }
}

This script,drawn from the SystemTap project wiki,can be used to print information kernel memory allocations of the system. The script can be invoked as follows:

stap kmalloc2.stp

For issues during the compile or loading of the module within the?stap?command,append the parameter?-vv?to show more verbose output.

SystemTap will then translate the probe into C,compile the C program,and insert the probe into the running kernel. Truncated output is below:

-------------------------------------------------------

Allocations for httpd
Count:   10 allocations
Sum:     0 Kbytes
Average: 0 bytes
Min:     0 bytes
Max:     0 bytes

Allocations by size in bytes
value |-------------------------------------------------- count
    0 |@@@@@@@@@@                                         10
    1 |                                                    0
    2 |                                                    0

-------------------------------------------------------

Allocations for sendmail
Count:   2 allocations
Sum:     0 Kbytes
Average: 24 bytes
Min:     24 bytes
Max:     24 bytes

Allocations by size in bytes
value |-------------------------------------------------- count
    4 |                                                   0
    8 |                                                   0
   16 |@@                                                 2
   32 |                                                   0
   64 |                                                   0

-------------------------------------------------------

This is a simple example that just touches the surface of the capabilities offered by SystemTap. System Administrators could use this information to better understand kernel memory allocation on the running system and adjust kernel tuning parameters accordingly. Application developers can use this information as an overview of which applications are receiving more kernel memory allocations,which can be used as a starting point for deeper application profiling.

Additional Information about SystemTap

• Complement to this guide
• SystemTap Beginners Guide
• The SystemTap project Wiki:?http://sourceware.org/systemtap/wiki/HomePage
• The SystemTap project Homepage:?http://sourceware.org/systemtap/
• Red Hat Magazine overview of SystemTap:?http://www.redhat.com/magazine/011sep05/features/systemtap/