Friday, September 19, 2008

Timed measurements with pmcstat(8)

Every once in a while I get email asking for an option to pmcstat(8) that would allow hardware events to be measured for a specified time interval.

The good news is: pmcstat(8) already supports timed measurements — using /bin/sleep.

Here is how you would do it:

  • Timed system sampling would be done in the following manner:

    % pmcstat -S instructions -O logfile /bin/sleep 42

    This invocation allocates a system-scope sampling PMC (-S) and profiles the whole system while /bin/sleep executes, i.e., for 42 seconds.

  • Timed measurements on groups of processes are performed in a similar fashion:

    % pmcstat -d -p dc-misses -t '1234' /bin/sleep 24

    This invocation would allocate a process-scope counting PMC (-p) that counts data cache misses, attach it (-t) to the process with pid 1234 and its descendants (-d), and count for 24 seconds.

    Note that the '-t' option also takes regular expressions, so you don't need to know process ids beforehand. To count instructions executed by processes named 'httpd' for an hour you would use:

    % pmcstat -p instructions -t 'httpd' /bin/sleep 3600

The "Unix way" uses small tools, each of which does a defined task reasonably well and which are combined to perform more complex tasks. In the examples above, /bin/sleep manages time intervals and pmcstat(8) manages PMC based measurements.

When you combine them, voilĂ , you get timed PMC based measurements.

Monday, September 15, 2008

PMCTools meets KCachegrind

Fabien Thomas sent in a screenshot showing the result of his experiments at integrating KCachegrind and PmcTools.

Thank you Fabien, for sharing!

Sunday, August 31, 2008

Unbreaking CVSup/amd64

When I recently upgraded my FreeBSD6/amd64 ``-STABLE'' machine, the CVSup binary on the system stopped working. CVSup would dump core consistently, shortly after connecting to the remote server. This rather unwelcome development took away my ability to keep my CVS trees upto-date; fixing the bug became top priority. The bug also turned out to be an interesting one.

The bug

Running CVSup after the upgrade to 6.3-PRERELEASE would result in a core dump shortly after connection establishment.

Program received signal SIGBUS, Bus error.
0x0000000800682d4f in fcntl () from /lib/
(gdb) disassemble fcntl
... snip ...
0x0000000800682d3f <fcntl+79>:  movaps %xmm4,0xffffffffffffffc1(%rax)
0x0000000800682d43 <fcntl+83>:  movaps %xmm3,0xffffffffffffffb1(%rax)
0x0000000800682d47 <fcntl+87>:  movaps %xmm2,0xffffffffffffffa1(%rax)
0x0000000800682d4b <fcntl+91>:  movaps %xmm1,0xffffffffffffff91(%rax)
0x0000000800682d4f <fcntl+95>:  movaps %xmm0,0xff
0x0000000800682d53 <fcntl+99>:  lea    0x110(%rsp),%rax
0x0000000800682d5b <fcntl+107>: movl   $0x10,0x20(%rsp)
0x0000000800682d63 <fcntl+115>: movl   $0x30,0x24(%rsp)
0x0000000800682d6b <fcntl+123>: mov    %rax,0x28(%rsp)
... snip ...

The faulting instruction was trying to save SSE registers to memory; and this was odd since there was no reason for this particular code path to be using SSE registers in the first place.

Rebuilding Modula-3 and CVSup from source did not fix the core dump, though the builds of these tools themselves completed without error. A search through the PR database revealed that other FreeBSD users had also been tripped by the bug: PR bin/124353.

A peek at the solution

Modula-3's runtime needed to be patched in the following way to fix this fault.

  • First, in $M3SRC/libs/m3core/src/unix/freebsd-4.amd64/Unix.i3, we declare the Modula-3 function Unix.fcntl() as being implemented externally by C function ufcntl().
    ... snip ...
    <*EXTERNAL "ufcntl"*> PROCEDURE fcntl (fd, request: int; arg: long): int;
    ... snip ...
  • Matching this declaration, an implementation of ufcntl() was provided in $M3SRC/libs/m3core/src/runtime/FBSD_AMD64/RTHeapDepC.c:
    #include <fcntl.h>
    ufcntl(int fd, int cmd, long arg)
           return (fcntl(fd, cmd, arg));

On the surface, this "fix" does not seem to be doing anything. The ufcntl() entry point takes 3 arguments but it passes these down to fcntl() unchanged, and in the same order.

Yet, despite the apparent ``no op''-like nature of the change, the core dumps were gone.

Why this works

To understand why this fix works, we have to delve into the ABI; into the C calling conventions used for AMD64 code.

For normal function calls, the AMD64 calling convention passes upto 6 integer arguments in registers. Thus register %rdi would hold the first argument (fd in our case), register %rsi the second, cmd, register %rdx the third and so on. However, the C prototype for fcntl() is: int fcntl(fd, cmd, ...);, i.e., fcntl is a varargs function. Varargs functions use a different calling convention on the AMD64: register %rax is a ``hidden'' input parameter for these functions.

So, prior to the fix, the Modula-3 runtime was invoking fcntl() directly, but with registers set up for a non-varargs function call.

Now, as it turns out, in FreeBSD 6.2 and earlier, fcntl() in libc was not a C language function; rather it was implemented as an assembly language stub that invoked the SYS_fcntl system call. On the AMD64, FreeBSD's argument passing convention for system calls is close enough to the non-varargs C calling convention that the processor's registers happened to be correctly setup for a direct system call.

When fcntl() in libc was changed in FreeBSD 6-STABLE on 24 Apr 2008 to be a C function instead of a system call, things broke.

Though not obvious from just looking at the C code, the no-op like fix above works by using the C compiler to translate between the two calling conventions.

What's worrying

The relevant change to libc was in CVS/SVN HEAD for about 20 days before it was merged to -stable. CVSup is also a critical tool for the FreeBSD project. This bug was however only detected in -stable, and not in -current.

Monday, August 18, 2008

Getting online using FreeBSD and BSNL DataOne

BSNL's DataOne service (DSL) is straightforward to use in FreeBSD. The following recipe shows how to get online using PPPoE.

  • First, you need to configure your DSL modem as specified by the ISP. This procedure is modem-specific and your BSNL representative should be able to assist you here.
  • You would need the name of the ethernet interface to which your modem is attached. If you are unsure of what this is, use ifconfig(8) to find out.
  • Next, you need to add the following template text to /etc/ppp/ppp.conf:
     set device "PPPoE:*INTERFACE*"
     set authname "*YOUR-USERNAME*"
     set authkey "*YOUR-PASSWORD*"
     set dial
     enable dns
     add default HISADDR

    Replace *YOUR-USERNAME* and *YOUR-PASSWORD* with your DataOne user name and password respectively. Replace *INTERFACE* with the name of your network interface (i.e., "rl0" or "fxp0" or whatever).

  • Finally, invoke ppp(8) in the usual way:
    % ppp dataone
    ppp> dial
    ppp> ... the prompt changes as PPP negotiation proceeds ...

Thats, it! You should be online.

Saturday, August 16, 2008

Tuesday, July 29, 2008

Getting online with GPRS in India

In order to get online using GPRS you would need:

  1. A device capable of handling GPRS; usually a cell phone or a GPRS modem. I've used a Nokia 3110c with some success. This particular cellphone model offers a USB port using which it may be connected to a host computer with a compatible USB cable.
  2. A subscription to GPRS service. Contact your cell service provider on the way to enable this on your cell phone. The examples below describe the configuration needed for using AirTel's GPRS service.
  3. The appropriate configuration for your OS. This post describes how to use GPRS using FreeBSD and NetBSD.

FreeBSD Configuration

  1. Ensure that the relevant drivers are present in the kernel. The Nokia 3110c is supported by the stock umodem(4) driver. Use kldload(8) to load this into your kernel if it is not already present:
    # kldload umodem
  2. On connecting the cellphone to the computer, you should see a message similar to the following on the console (or in /var/log/messages):
    ... kernel: ucom0: Nokia Nokia 3110c, rev x.yy/, addr 2, iclass 2/2
    ... kernel: ucom0: data interface 2, has CM over data, has break
    ... kernel: ucom0: status change notification available
    If all goes well, a new device entry such as /dev/cuaU0 should be present under /dev.
  3. Add the following to /etc/ppp/ppp.conf (note the leading whitespace on most lines):
     set device /dev/cuaU0
     enable dns
     set phone "*99***1#"
     set authname "airtel"
     set authkey "airtel"
      \"\" AT OK-AT-OK ATQ0V1E1S0=0&C1&D2+FCLASS=0 OK \
      AT+CGDCONT=1,\\\"IP\\\",\\\"\\\" OK \\dATDT\\T \
      add default HISADDR
      set ifaddr

    For the curious, the ppp(8) manual page fully describes the syntax and semantics of the configuration commands above.

    Note that the contents of the authname and authkey configuration items do not seem to matter. You could put the name of your pet dog there if you wish.

  4. Invoke the ppp(8) daemon in the usual way:
    % ppp airtel
    ppp> dial
    ... the prompt changes as PPP setup progresses

    And you should be done.

NetBSD Configuration

In NetBSD as in FreeBSD, this particular cellphone model is supported by the umodem(4) driver. Once recognized, the modem will be accessible using a path name such as /dev/ttyU0.

The following recipe uses the pppd(8) daemon. This recipe requires the NetBSD kernel to have been compiled with in-kernel PPP support.

  1. Create a file /etc/ppp/peers/airtel with the following contents:
    ttyU0 115200
    connect '/usr/sbin/chat -v -f /etc/ppp/chat-airtel -T "*99***1#"'
    user "airtel"
    password "airtel"
  2. Create a file /etc/ppp/chat-airtel with the following contents:
    "" "AT"
    OK "ATQ0V1E1S0=0&C1&D2+FCLASS=0"
    OK AT+CGDCONT=1,\"IP\",\"\"
    OK \dATDT\T
    TIMEOUT 40
  3. Start ppp using:
    # pppd call airtel

    Monitor /var/log/messages for error messages.

  4. Once the connection is setup, IP addresses for the ISP's DNS servers would be placed in /etc/ppp/resolv.conf. Add these to /etc/resolv.conf if needed.

Connection performance

In my experience, connection speeds vary widely from a few bytes/second to a peak of a few KB/sec. Connection latencies also tend to be variable, ranging from a few hundred milliseconds upwards. There are frequent long pauses, a minute or more in duration, during which no traffic flows (though the connection remains up).

Thus GPRS based connectivity is best used for batch tasks such as downloading and sending email.

Friday, July 18, 2008

Slowly getting back online

One of the prices paid when I moved out to rural India last year was loss of connectivity.

For many months, there was no reliable way for people to call and reach me. As for internet access, I used to go to the nearby town (20km away) to an internet parlour from where I would access the 'Net---if there was power in the town to do so. Keeping my FreeBSD CVSup repository in sync used to be a interesting challenge, as was keeping up the communication flow with my two GSoC'07 mentees.

Connectivity options began to improve about a month ago: AirTel's cell service started up in the area so I could get online using GPRS. And then, last week, BSNL started offering DSL service in this area.

In the past year I explored various options for getting online as a FreeBSD user; I will describe these here by and by. Stay tuned!