New release: ELF Toolchain v0.6.1

I am pleased to announce the availability of version 0.6.1 of the software being developed by the ElfToolChain project.

This new release supports additional operating systems (DragonFly BSD, Minix and OpenBSD), in addition to many bug fixes and documentation improvements.

This release also marks the start of a new "stable" branch, for the convenience of downstream projects interested in using our code.

Comments welcome.

PmcTools: Motivation and Future Steps

This (long delayed) post describes the original motivation for my PmcTools whole-system profiling toolkit, and touches on some of the possible next steps for the project.

Motivation

Around the year 2000, I happened to read a paper titled "Continuous Profiling: Where Have All the Cycles Gone?". The techniques described by the paper were inspiring, but the DEC Alpha™ systems they were implemented on were out of reach for a hobbyist living in India. A FreeBSD™-equivalent of those tools and techniques, running on affordable hardware, seemed a good idea.

From the outset, my goal was to create a programming toolkit for using in-CPU performance counters:

• I wanted an API that would permit tools to fully use the features provided by the hardware.
• I wanted tools that had low overheads, in order not to disturb the behaviour of the system being measured.
• I wanted tools that were non-disruptive—usable without needing to restart running processes, rebooting the system or requiring recompilation, etc.
• I wanted to analyse the "whole system" at once; i.e., to simultaneously analyze userspace applications, the top-half of the kernel, and the kernel's interrupt handlers.
• I wanted the toolset to be SMP-ready, since SMP seemed affordable in the future.

When affordable systems using AMD Athlon™ CPUs (with publically documented in-CPU performance counters) entered the Indian market in early 2003, I built myself a machine, and started on the project.

(Pre-)History

• 2003: Initial work, which was managed using homebrew tools tuned for dialup speeds (shell scripts running RCS layered over CVS/CVSup).
• 2004: With the arrival of broadband access, development moved to FreeBSD's Perforce™ server.
• 2005: The first check-in into the FreeBSD source tree in April 2005.

Current Status

At the time of writing, PmcTools is being actively maintained and extended by the FreeBSD community.

The design of the toolkit is briefly described in a tech talk presented at ACM Bangalore in 2009 (slides).

Future Steps

Platforms, simplicity and portability are likely to be the focus of future work.

Platforms

PmcTools would be useful on popular hobby platforms such as the BeagleBoard. PmcTools already supports 'remote' data collection on embedded systems. However, the specific PMCs on these systems would need to be supported.

Simplicity

Based on the experience gained so far, both the programming APIs and the implementation of PmcTools could be simplified without losing useful functionality.

Portability

PmcTools would be a useful addition to other open-source operating systems.

In addition to the above, many innovative tools can be created: in the paper "Exploiting hardware performance counters with flow and context sensitive profiling", the authors show how to add PMC-based instrumentation to program binaries for fine-grained analyses. To be able to add such instrumentation, we need tools to parse and modify binary instruction streams—one of the motivations for the proposed libmc library, part of the Elftoolchain project.

Comments welcome.

A build system for the Elftoolchain project

Summary

This post describes the cross-platform build and test system being written for the Elftoolchain project.

The need for a cross-platform build and test system

In the upcoming v1.0 release of the Elftoolchain project we plan to support 6 operating system families---prominent *BSD OSes such as FreeBSD and NetBSD, Ubuntu GNU/Linux and Minix. We may additionally support an Illumos-derived operating system.

For each supported OS, we may need to test our sources on multiple released OS versions. We may need to build and test on multiple architectures (e.g., ARM, MIPS, i386 and X64/AMD64), depending on the target operating system.

Apart from mainline development, we would need to support maintenance branches of our source tree on these OS instances.

Clearly, some kind of automation is needed to help manage this combinatorial increase in support load.

Design goals

In brief, the design goals are:

• The system should support builds of our source tree, on the target operating systems and machine architectures of our interest.
• The system should support builds on non-native architectures (relative to the build host).
• The system should allow a source tree that is in-development to be built and tested, prior to a check-in.
• The system should be deployable with the minimum of software dependencies, and should be easy to configure.
• The system should be able to run entirely on a relatively power and resource constrained system such as a laptop, i.e., without needing a beefy build box, or architecture-specific hardware.

Related Projects

Continuous integration systems such as Buildbot, Bitten and Hudson are commonly used to manage automated builds. While these tools are featureful, their large resource requirements, and the additional dependencies needed to run them (a Java/Python runtime, along with other dependencies) make these tools difficult to use in the Elftoolchain context.

The QEMU and VirtualBox programs are popular machine emulators. When running on X86/X64 hardware, these programs support the emulation of i386 and x86_64/amd64 CPUs. Additionally, QEMU can emulate non-native architectures using dynamic translation techniques. The GXemul project is a BSD-licensed machine emulator, similar to QEMU.

The Design

In the current design, the build system comprises of two major parts:

• A simple daemon---a portable C program built using libevent, that runs inside the target OS in the machine emulator. This 'slave' component connects to a 'master/despatcher' component that runs on the build host and executes commands issued to it.
• A 'master' component that is responsible for managing the build process at the top-level: starting up the relevant machine emulators, waiting for the OS inside to boot and for the 'slave' inside to connect back, transferring the source tree of interest into the slave, running the build/test cycle, collecting output files and output status, and shutting down the emulator cleanly.

Note that the actual build (& test) within a source tree is controlled using BSD make.

The protocol between the 'slave' and the 'master' components is spartan: it supports the execution of arbitrary shell script fragments on the slave with redirection of input and output, and supports simple data transfer between the 'slave' and the 'master'. In order to maintain responsiveness, the protocol between the 'slave' and the 'master' is asynchronous. Multiple 'slaves' could be connected to the 'master' concurrently.

The 'master' would be controlled by a set of configuration files and shell scripts.

The Implementation

The implementation (a work-in-progress) may be found in the tools/build-automation directory of the Elftoolchain project's source tree.

The implementation is being written as a literate program.

Comments welcome.

NetBSD

I am pleased to mention here that I have joined the NetBSD project as a developer.  Alistair Crooks (agc@) and Cherry Mathew (cherry@) are my sponsors.


My initial focus will be on integrating the tools from the ELF Toolchain project into the NetBSD base system.