1.1 Why a Linux trace toolkit?

    The Linux operating system is a multiprogramming, multiuser system. This means that it is able to handle multiple running programs at once. On a uniprocessor system (a computer with only one microprocessor), this is achieved by sharing this precious resource among multiple tasks, letting each execute for a certain period of time and then switching to another. The selection and switching of processes is handled by the Linux kernel, which also is a program and therefore also needs some time from the processor. It is also responsible for fulfilling certain requests by the programs it manages, dealing with error conditions, etc.

    One could have the need to know exactly what these scheduling decisions, process switches and various management tasks are, how they are handled, how long they take and to which process the processor is allocated. Spread out over a certain period of time, we call this an execution trace.

   The Linux trace toolkit is a suite of tools designed to do just that; extract program execution details from the Linux operating system and interpret them. Specifically, it enables its user to extract processor utilization and allocation information for a certain period of time. It is then possible to perform various calculations on this data and dump this in a text file. the list of probed events can also be included in this. The integrated environment, based on the GTK+ library, can also plot these results and perform specific searches.

1.2 Software requirements

    In its simplest form, the Linux trace toolkit does not need any specific libraries. Any properly configured Linux system will do. By this, we mean a Linux system where the Linux header files, source code and libraries are present. No, this will NOT work on any other operating system, although you might be able to compile and use the graphical environment. This is obviously not very useful if you cannot get any data from the kernel!

   As of 11/23/2004, however, the data acquisition components require version 2.6.9 of the Linux kernel. You may be able to update/downgrade the patch provided with the package to other kernel versions. As a matter of fact, many people have succesfully done so and the project's ftp site contains a directory called "ExtraPatches" which contains the patches submitted by some of these people.

   In any case, you can very well do your data acquisition with a patched (more about that in the next section) kernel and then reboot using any other kernel to view the results. The performance hit for a patched kernel compared to a "normal" one is, however, minimal; Testing has shown that this is less than 2.5% on a normal use system (measured using batches of kernel makes). On algorithm intensive tasks, the impact is close to zero. On file intensive systems (measured using batches of tar), the impact is less than 5%. The kernels can be found at  www.kernel.org.

    This is fine if all you need is to get results and dump them in a text file. However, you would be missing out on one of LTT's most powerful features: the plotting of the trace. All of the integrated environment including the event graph is implemented with the GTK+ library. You can download it for free from  www.gtk.org. This version of LTT was coded and successfully tested with GTK+ version 1.2.10, although any 1.2.x version will most probably do.

    Many Linux distributions include packages for GTK+. In Mandrake and Fedora Core, you can install the necessary gtk+ development materials in order to build the graphical parts of TraceToolkit by installing the packages: gtk+-devel and libglib-devel. Please make sure these packages are installed on your system prior to building the TraceToolkit user space components.

1.3 Architecture

    The Linux trace toolkit is made up of the following components:
 

For more information on the project's architecture, take a peak at the Documentation section of LTT's web site.

1.4 Real-Time Linux support

    Along-side support for Linux, LTT, starting version 0.9.3, has support for the DIAPM RTAI (Dipartimento di Ingegneria Aerospaziale - Politecnico di Milano Real Time Application Interface) real-time linux extension. Although you could use LTT with an RTAI patched kernel, LTT didn't record any information about RTAI's decision or the behavior of the tasks it scheduled. Now, LTT is able to get this information and to analyze it to retrieve important information to the understanding of the dynamic behavior of an RTAI/Linux system. Most interesting are the event graphs drawn according to the collected information. They enable a unique view a real-time Linux system. This is most important when trying to isolate all types of timing problems.

    If you do not use RTAI or do not plan to do so, you can skip sections discussing using LTT with RTAI.