EDOOFUS

Choosing an open-source project to work with

2010-07-22T11:22:00.000-07:00

In the article "Why you should contribute to open-source" Shalin Mangar lists some of the benefits of participating in an open source project.

However, there are a large number of open-source projects out there; indeed, quite a few tools have multiple active implementations. How does a new contributor choose a project to contribute to?

The process I usually suggest is:

Start by choosing a domain of interest—an area which you would be comfortable spending a few years of your life learning about.
Short list two to five interesting open-source projects in this domain.
Research the functioning of each project using the check list given later in this post.
Narrow your list down to one project that matches your values.
Dive in and start!

Find open-source projects in a domain of interest.

Whether this is compilers, operating systems, GUI tools, Web 2.0 tools or something else, you should choose a domain that fascinates you.

Search for existing open-source projects—collaboration sites like Freshmeat or SourceForge can help you find an active project in the area you are interested in. Sites like ohloh.net can help you guage the project's activity levels and other vital statistics.

Study the selected projects in depth

Spend some time examining each community. Visit their home pages and browse their code. Keep a look out for the key values held by the community. The following set of questions could help get you started:

What is the quality of the project's source code and documentation? Is doing work to a high standard a value in the community? What will you learn by reading their code?

Warning signs: incomprehensible code, documentation that is sparse or poorly written, or that is out of sync with the code.
What are the project's communication channels? Do they maintain public archives of their communication? What is the signal to noise ratio observed on their channels? What tone of communication is commonly seen?

Warning signs: snooty behaviour towards newbies, excessive bureaucracy within the project, contributions being silently dropped.
What are the project's development processes like? For example, they have a culture of performing code reviews? Do they use a ticket tracking system? What kind of automation do they use?
How open is decision making in the project? Do you see evidence of outside input being welcomed? How do conflicts get resolved?

Warning signs: private communication channels between select developers, small groups of developers taking ad-hoc decisions without input from the other stake holders, frequent conflicts that escalate.
What is the project's maturity? Are there experienced hands around to act as mentors? Do you see evidence of such mentoring on the project's communications channels?
How active is the project? Projects that show a steady (or steadily increasing) rate of progress over a long period are usually in healthy shape.
Who is behind the project?

Some open-source projects are dominated by people employed in a small number of companies. Such projects run the risk of sudden death when corporate priorities change, as happened to the OpenSolaris project, one of the best operating system projects of its time.

Look at the profiles of the key developers; keep a look out for projects that lack sufficient diversity.
What is the licensing of the project? Some projects use permissive licenses like the Apache license, while some prefer the more restrictive GPL. Are you comfortable with the license being used?

Match projects to your own values

Write down what you expect to gain from participating in the projects in your short list.

Some of the reasons people participate in open-source are:

Wanting to contribute and make a difference to society.
Wanting to learn and become an expert in some aspect of computing.
For some, open-source provides a creative outlet that their employment does not provide.
For some, open-source is a way to join a community---they value the company of people sharing a common interest in a domain.

Use your values as a guide to selecting the project that would fit you the best.

Dive in and get started

Hopefully the suggestions above would have helped you narrow down to one, attractive open-source project. Your next steps would be to dive in and learn your way around the project's code. Pick a bug to be fixed, or a small feature that needs implementing.

Good luck, and happy hacking!

New discussion group for PmcTools

2009-06-12T20:13:00.000-07:00

The PmcTools project now has a new discussion group pmctools-discuss, hosted on groups.google.com.

If you have a question on how to use PmcTools, or about its internals, please ask it there.

PmcTools talk at the Bangalore chapter of the ACM

2009-04-19T21:58:00.000-07:00

In April 2009 I was invited to speak on FreeBSD/PmcTools by the Bangalore chapter of the ACM. The slides for this talk are available at: http://people.freebsd.org/~jkoshy/download/acm-apr-09.pdf (550KB, PDF).

This was an overview talk. The talk briefly touched upon: the motivations and goals of the project, the programming APIs, some aspects of the implementation and on possible future work.

Short updates

2009-02-09T23:05:00.001-08:00

A couple of (lagged!) updates on my FreeBSD/PmcTools work:

In Nov'08, I added support for Intel™ Core 2™ PMCs to PmcTools (SVN #185363 and later changesets).

More recently, Nokia, via Jeff Roberson, contributed basic support for the PMCs in the Core/i7™ CPU, the next member in the Core™ family of CPUs (SVN #187761).

With these additions PmcTools supports most Intel CPUs that have PMCs in them. (I have plans to finish support for 166Mhz Pentium MMX™ CPUs soon.)
A couple of tricky bugs were tracked down and fixed:
- The NMI handler code I had written for the amd64 architecture needed to be more robust than it was. Fixed in SVN #188065.
- A bug in callchain capture code that would appear only under high loads and only on SMP machines. Fixed in SVN #186037.
Thanks to Artem Belevich, Fabien Thomas, George Neville-Neil and Jeff Roberson for their assistance with debugging.

Timed measurements with pmcstat(8)

2008-09-19T10:06:00.000-07:00

Every once in a while I get email asking for an option to pmcstat(8) that would allow hardware events to be measured for a specified time interval.

The good news is: pmcstat(8) already supports timed measurements — using /bin/sleep.

Here is how you would do it:

Timed system sampling would be done in the following manner:
```
% pmcstat -S instructions -O logfile /bin/sleep 42
```
This invocation allocates a system-scope sampling PMC (-S) and profiles the whole system while /bin/sleep executes, i.e., for 42 seconds.
Timed measurements on groups of processes are performed in a similar fashion:
```
% pmcstat -d -p dc-misses -t '1234' /bin/sleep 24
```
This invocation would allocate a process-scope counting PMC (-p) that counts data cache misses, attach it (-t) to the process with pid 1234 and its descendants (-d), and count for 24 seconds.

Note that the '-t' option also takes regular expressions, so you don't need to know process ids beforehand. To count instructions executed by processes named 'httpd' for an hour you would use:
```
% pmcstat -p instructions -t 'httpd' /bin/sleep 3600
```

The "Unix way" uses small tools, each of which does a defined task reasonably well and which are combined to perform more complex tasks. In the examples above, /bin/sleep manages time intervals and pmcstat(8) manages PMC based measurements.

When you combine them, voilà, you get timed PMC based measurements.

PMCTools meets KCachegrind

2008-09-14T23:25:00.000-07:00

Fabien Thomas sent in a screenshot showing the result of his experiments at integrating KCachegrind and PmcTools.

Thank you Fabien, for sharing!

Unbreaking CVSup/amd64

2008-08-31T02:23:00.000-07:00

When I recently upgraded my FreeBSD6/amd64 ``-STABLE'' machine, the CVSup binary on the system stopped working. CVSup would dump core consistently, shortly after connecting to the remote server. This rather unwelcome development took away my ability to keep my CVS trees upto-date; fixing the bug became top priority. The bug also turned out to be an interesting one.

The bug

Running CVSup after the upgrade to 6.3-PRERELEASE would result in a core dump shortly after connection establishment.

Program received signal SIGBUS, Bus error.
0x0000000800682d4f in fcntl () from /lib/libc.so.6
(gdb)
(gdb) disassemble fcntl
... snip ...
0x0000000800682d3f <fcntl+79>:  movaps %xmm4,0xffffffffffffffc1(%rax)
0x0000000800682d43 <fcntl+83>:  movaps %xmm3,0xffffffffffffffb1(%rax)
0x0000000800682d47 <fcntl+87>:  movaps %xmm2,0xffffffffffffffa1(%rax)
0x0000000800682d4b <fcntl+91>:  movaps %xmm1,0xffffffffffffff91(%rax)
0x0000000800682d4f <fcntl+95>:  movaps %xmm0,0xff
ffffffffffff81(%rax)
0x0000000800682d53 <fcntl+99>:  lea    0x110(%rsp),%rax
0x0000000800682d5b <fcntl+107>: movl   $0x10,0x20(%rsp)
0x0000000800682d63 <fcntl+115>: movl   $0x30,0x24(%rsp)
0x0000000800682d6b <fcntl+123>: mov    %rax,0x28(%rsp)
... snip ...

The faulting instruction was trying to save SSE registers to memory; and this was odd since there was no reason for this particular code path to be using SSE registers in the first place.

Rebuilding Modula-3 and CVSup from source did not fix the core dump, though the builds of these tools themselves completed without error. A search through the PR database revealed that other FreeBSD users had also been tripped by the bug: PR bin/124353.

A peek at the solution

Modula-3's runtime needed to be patched in the following way to fix this fault.

First, in $M3SRC/libs/m3core/src/unix/freebsd-4.amd64/Unix.i3, we declare the Modula-3 function Unix.fcntl() as being implemented externally by C function ufcntl().
```
... snip ...
<*EXTERNAL "ufcntl"*> PROCEDURE fcntl (fd, request: int; arg: long): int;
... snip ...
```
Matching this declaration, an implementation of ufcntl() was provided in $M3SRC/libs/m3core/src/runtime/FBSD_AMD64/RTHeapDepC.c:
```
...
#include <fcntl.h>
...
int
ufcntl(int fd, int cmd, long arg)
{
       return (fcntl(fd, cmd, arg));
}
    
```

On the surface, this "fix" does not seem to be doing anything. The ufcntl() entry point takes 3 arguments but it passes these down to fcntl() unchanged, and in the same order.

Yet, despite the apparent ``no op''-like nature of the change, the core dumps were gone.

Why this works

To understand why this fix works, we have to delve into the ABI; into the C calling conventions used for AMD64 code.

For normal function calls, the AMD64 calling convention passes upto 6 integer arguments in registers. Thus register %rdi would hold the first argument (fd in our case), register %rsi the second, cmd, register %rdx the third and so on. However, the C prototype for fcntl() is: int fcntl(fd, cmd, ...);, i.e., fcntl is a varargs function. Varargs functions use a different calling convention on the AMD64: register %rax is a ``hidden'' input parameter for these functions.

So, prior to the fix, the Modula-3 runtime was invoking fcntl() directly, but with registers set up for a non-varargs function call.

Now, as it turns out, in FreeBSD 6.2 and earlier, fcntl() in libc was not a C language function; rather it was implemented as an assembly language stub that invoked the SYS_fcntl system call. On the AMD64, FreeBSD's argument passing convention for system calls is close enough to the non-varargs C calling convention that the processor's registers happened to be correctly setup for a direct system call.

When fcntl() in libc was changed in FreeBSD 6-STABLE on 24 Apr 2008 to be a C function instead of a system call, things broke.

Though not obvious from just looking at the C code, the no-op like fix above works by using the C compiler to translate between the two calling conventions.

What's worrying

The relevant change to libc was in CVS/SVN HEAD for about 20 days before it was merged to -stable. CVSup is also a critical tool for the FreeBSD project. This bug was however only detected in -stable, and not in -current.

Getting online using FreeBSD and BSNL DataOne

2008-08-17T22:01:00.000-07:00

BSNL's DataOne service (DSL) is straightforward to use in FreeBSD. The following recipe shows how to get online using PPPoE.

First, you need to configure your DSL modem as specified by the ISP. This procedure is modem-specific and your BSNL representative should be able to assist you here.
You would need the name of the ethernet interface to which your modem is attached. If you are unsure of what this is, use ifconfig(8) to find out.
Next, you need to add the following template text to /etc/ppp/ppp.conf:
```
dataone:
 set device "PPPoE:*INTERFACE*"
 set authname "*YOUR-USERNAME*"
 set authkey "*YOUR-PASSWORD*"
 set dial
 enable dns
 add default HISADDR
```
Replace *YOUR-USERNAME* and *YOUR-PASSWORD* with your DataOne user name and password respectively. Replace *INTERFACE* with the name of your network interface (i.e., "rl0" or "fxp0" or whatever).

Finally, invoke ppp(8) in the usual way:

% ppp dataone
ppp> dial
ppp> ... the prompt changes as PPP negotiation proceeds ...
PPP>

Thats, it! You should be online.

Draft #5 of the indic-computing "Design Axes" article

2008-08-15T19:56:00.000-07:00

Draft #5 of the "Design Axes for Indian Language Computing" article has been made available for review at the Indic-Computing web site.

Please do take a look and let me know your comments.

Getting online with GPRS in India

2008-07-29T10:34:00.000-07:00

In order to get online using GPRS you would need:

A device capable of handling GPRS; usually a cell phone or a GPRS modem. I've used a Nokia 3110c with some success. This particular cellphone model offers a USB port using which it may be connected to a host computer with a compatible USB cable.
A subscription to GPRS service. Contact your cell service provider on the way to enable this on your cell phone. The examples below describe the configuration needed for using AirTel's GPRS service.
The appropriate configuration for your OS. This post describes how to use GPRS using FreeBSD and NetBSD.

FreeBSD Configuration

Ensure that the relevant drivers are present in the kernel. The Nokia 3110c is supported by the stock umodem(4) driver. Use kldload(8) to load this into your kernel if it is not already present:
```
# kldload umodem
```
On connecting the cellphone to the computer, you should see a message similar to the following on the console (or in /var/log/messages):
```
... kernel: ucom0: Nokia Nokia 3110c, rev x.yy/a.bb, addr 2, iclass 2/2
... kernel: ucom0: data interface 2, has CM over data, has break
... kernel: ucom0: status change notification available
```
If all goes well, a new device entry such as /dev/cuaU0 should be present under /dev.

Add the following to /etc/ppp/ppp.conf (note the leading whitespace on most lines):

airtel:
 set device /dev/cuaU0
 enable dns
 set phone "*99***1#"
 set authname "airtel"
 set authkey "airtel"
 set dial "ABORT BUSY ABORT NO\\sCARRIER TIMEOUT 5 \
  \"\" AT OK-AT-OK ATQ0V1E1S0=0&C1&D2+FCLASS=0 OK \
  AT+CGDCONT=1,\\\"IP\\\",\\\"airtelgprs.com\\\" OK \\dATDT\\T \
  TIMEOUT 40 CONNECT"
  add default HISADDR
  set ifaddr 10.0.0.1/0 10.0.0.2/0 255.255.255.0 0.0.0.0

For the curious, the ppp(8) manual page fully describes the syntax and semantics of the configuration commands above.

Note that the contents of the authname and authkey configuration items do not seem to matter. You could put the name of your pet dog there if you wish.

Invoke the ppp(8) daemon in the usual way:

% ppp airtel
ppp> dial
... the prompt changes as PPP setup progresses
PPP>

And you should be done.

NetBSD Configuration

In NetBSD as in FreeBSD, this particular cellphone model is supported by the umodem(4) driver. Once recognized, the modem will be accessible using a path name such as /dev/ttyU0.

The following recipe uses the pppd(8) daemon. This recipe requires the NetBSD kernel to have been compiled with in-kernel PPP support.

Create a file /etc/ppp/peers/airtel with the following contents:

ttyU0 115200
connect '/usr/sbin/chat -v -f /etc/ppp/chat-airtel -T "*99***1#"'
defaultroute
usepeerdns
ipcp-accept-local
ipcp-accept-remote
user "airtel"
password "airtel"

Create a file /etc/ppp/chat-airtel with the following contents:

ABORT "BUSY"
ABORT "NO CARRIER"
ABORT "ERROR"
ABORT "NO ANSWER"
"" "AT"
OK "ATQ0V1E1S0=0&C1&D2+FCLASS=0"
OK AT+CGDCONT=1,\"IP\",\"airtelgprs.com\"
OK \dATDT\T
TIMEOUT 40
CONNECT

Start ppp using:
```
# pppd call airtel
```
Monitor /var/log/messages for error messages.
Once the connection is setup, IP addresses for the ISP's DNS servers would be placed in /etc/ppp/resolv.conf. Add these to /etc/resolv.conf if needed.

Connection performance

In my experience, connection speeds vary widely from a few bytes/second to a peak of a few KB/sec. Connection latencies also tend to be variable, ranging from a few hundred milliseconds upwards. There are frequent long pauses, a minute or more in duration, during which no traffic flows (though the connection remains up).

Thus GPRS based connectivity is best used for batch tasks such as downloading and sending email.

Slowly getting back online

2008-07-18T02:07:00.000-07:00

One of the prices paid when I moved out to rural India last year was loss of connectivity.

For many months, there was no reliable way for people to call and reach me. As for internet access, I used to go to the nearby town (20km away) to an internet parlour from where I would access the 'Net---if there was power in the town to do so. Keeping my FreeBSD CVSup repository in sync used to be a interesting challenge, as was keeping up the communication flow with my two GSoC'07 mentees.

Connectivity options began to improve about a month ago: AirTel's cell service started up in the area so I could get online using GPRS. And then, last week, BSNL started offering DSL service in this area.

In the past year I explored various options for getting online as a FreeBSD user; I will describe these here by and by. Stay tuned!

Edoofus blocked by the Government of India

2006-07-19T10:11:00.000-07:00

Against stupidity the gods themselves contend in vain.
-- Friedrich von Schiller

(For those of you who are wondering this post is about: at the time the post was written, the Indian government had forced Indian ISPs to restrict internet access to all the blogs under the blogger.com domain. The rationale for this edict remains unclear, even today.)

Robert Watson: "How FreeBSD Works"

2006-05-17T09:42:00.001-07:00

An excellent summary of the FreeBSD project, by Robert Watson: http://www.watson.org/~robert/freebsd/2006bsdcan/20060512-bsdcan2006-how-freebsd-works.pdf

D-Link

2006-04-09T09:32:00.000-07:00

In 2005, I had the opportunity to visit D-Link India's development office in Bangalore. I found that they were using Linux on a few routers that they were developing. This surprised me since I had not found mention either of Linux, or for their obligations to their customers under the GPL, on the main D-Link website. My understanding of the GPL was that GPL-derived binaries need to be accompanied with a prominent offer of source code. Most companies that use Linux for embedded work just put their sources up on the web.

The matter got stranger. In response to a direct question, the VP of Development (at Bangalore) indicated that he didn't think that they needed to make the Linux source code that went into their products available to their customers. I didn't have the opportunity to ask why he thought that the GPL didn't apply to his product line. I do know that some D-Link routers are Linux based and that you can find GPL'ed source tarballs for these by assiduously googling for them. However, it is not clear to me whether source code is available for all of D-Link's Linux based routers and whether the provisions of the GPL are being correctly honored by this company.

And now, there is the news that D-Link's routers have been abusing NTP by directly connecting to Tier-1 NTP servers around the world, in violation of the way NTP is designed to work. The Slashdot thread on the topic revealed a few interesting tidbits:

D-Link's products had previously affected the free DynDNS service. This anonymously posted comment is particularly revealing: instead of taking time to fix the bugs in their code, D-Link apparently invested energy to work around the restrictions placed by DynDNS.org (this should be easy to verify).
Another comment expressed frustration at the D-Link/India firewall :) used to protect management from customer feedback.

The impression I came away with after going through the material on the Internet is that D-Link, as a company, gives short shrift to the network of gentlemen's agreements that hold the internet (and modern society) up. They have demonstrated that they are not above abusing a free service if they can find one, and if the anonymous poster's information is correct, that they are willing to work-around technical protective blocks with impunity. They use GPL'ed code without honoring its copyright fully and completely.

Whether this attitude arises due to malice or due to plain incompetence is not clear yet. Either way, this is one company whose products I'm personally going to avoid in the future, following the principle of giving my business to the least sucky corporation that I can find.

New Indian BSD developer

2006-03-29T18:38:00.000-08:00

India now has another BSD developer: Cherry G. Mathew <cherry at netbsd dot org>. Cherry is porting NetBSD to the ia64 architecture.

Congratulations, Cherry!

hwpmc as a basecamp project

2006-03-18T08:59:00.000-08:00

I'm trying out Basecamp as a communication/project management tool for hwpmc work.

37 Signals, the company behind Basecamp, uses FreeBSD servers to host Basecamp.

Embedded Space

2006-02-02T05:32:00.000-08:00

The past few weeks I've been hacking a small MIPS-based router, getting a glimpse into the workings of the embedded 'Linux' world. This is life at the edge of chaos: open-source, yes, but nearly bereft of sane software development practices. The FreeBSD world is so much nicer:

I found FreeBSD to be an excellent platform for development. The tools I needed were already part of the FreeBSD ports collection, and building applications via ports allowed me to tweak their configuration as I wished (e.g., make WITH_PYTHON=1 WITHOUT_PERL=1); a future port upgrade would preserve such settings automatically. User interaction remains smooth and glitch-free even running large compiles in the background.
Cross-development in FreeBSD is straightforward to get started off with ...
# cd /usr/src # make buildworld TARGET_ARCH=other-architecture # make buildenv TARGET_ARCH=other-architecture
and in one fell swoop one has a cross-compiler, standard libraries, and all the standard utilities synchronized with the kernel and ready for use. This is very different from the typical 'Linux' project where one has to do a lot of work to reach this point.
Then there is source history for every file in the base system. Figuring out "what changed?" isn't an exercise in frustration, scanning a myriad diffs and rummaging inside terse changelogs.
Similarly, is no need to go hunting on the 'net for the current version of common utilities, hoping that these would work with today's cross-toolchain and patch set. Most of the common utilities that one needs for an embedded product are part of our base system—code and have recieved a fair degree of testing on our supported architectures, the i386, sparc64, arm, amd64, powerpc, ia64 and the alpha. For 3rd-party tools too, patch management is easy with FreeBSD: if a FreeBSD port exists for the application then a
# cd PORTS-DIR && make patch
would give you source code that is (a) patched to compile on FreeBSD and is (b) upto-date or close to upto-date.
Tracking security vulnerabilities is straightforward on FreeBSD, both for the base system and for 3rd-party ports.

It was sobering to realize how much we FreeBSD folk just take for granted.

I also now understand how embedded Linux vendor companies manage to charge hefty sums for their embedded OS offerings. If I were a manager charged with implementing an 'embedded' Linux device, I would pay too. The risk of a project slipup because of some bizarre interaction between lightly tested upstream source bases is all too real here.

There are a number of ways by which FreeBSD could be made even better for embedded development:

It would be great if we could cross-compile 3rd-party applications as easily as we do the base system today. A general fix is probably hard, but there are ways to get the near ubiquitous "configure" shell script to handle cross builds reasonably well.
We need a way to quickly cobble together complete system images (kernel, configuration files, basic utilities, 3rd party tools) for common target platforms. For example, what if we could say ``I'd like an image from the RELENG_6 branch suitable for a Linksys WRT54GS clone, with pf and sshd from the base system, and clamav and smtp-proxy from ports and with the following local patches'', and have the system churn out a flashable image? The NanoBSD and PicoBSD tools are a start in this direction.
We need our system analysis and performance measurement tools to be truly cross-platform and embedded development friendly. For example, it would be cool to be able to run gprof on an AMD64 platform to analyse profiles for ARM binaries. It would be useful to be able to take kernel core dumps over the network and have seamless cross-architecture kernel debugging over a local LAN.
Multiple implementations of system functionality would be useful (e.g., some embedded applications could benefit from a less profilgate malloc(3) implementation that the default).
Documentation on scaling FreeBSD down would be useful to new adopters. Right now this knowledge is scattered in our list archives.

It would be great to see FreeBSD running on more embedded & resource constrained platforms.

Building Products with FreeBSD

2005-11-14T04:48:00.000-08:00

The availability of high-quality source code from the FreeBSD project, under its liberal license, is a boon to organizations (particularly startups) creating software products and appliances.

However, most organizations that I've had the oppurtunity to observe have not been looking at long term collaboration with the project. They take in FreeBSD source code as a one-time drop and forget to keep an eye on where things are going. These are the kinds of mistakes that I've seen being made:

One organization developed a kernel module that relied on kernel threads being non-preemptible (despite PREEMPTION being present in -CURRENT). They had to scramble when kernel preemption appeared in -STABLE.
One organization started with FreeBSD 5.X and kept fixing bugs that they encountered in their private copy of the code. They didn't feed anything back upstream, and had to redo their patches when they took the next code drop from upstream.
One organization started an ambitious FreeBSD project but never spoke about it in public. They advertised for candidates, but had a few excellent candidates walk away unconvinced that they would find good FreeBSD work in that company.

Traditionally, you wouldn't think of collaborating with your OS vendor. You would buy their software, either as a binary or as source, and start using it. You wouldn't have the ability to influence your vendors strategy and decision making process (unless you were IBM, of course). While some vendors do let you view their bug-list, bug prioritization remains their prerogative, so if you are hurting because of a nasty bug, you just have to cross your fingers and wait. You would be constrained by your vendor's platform support policies too; they could stop supporting a platform that is unprofitable for them but still profitable for you.

Collaboration with volunteer developed projects is something new, something that is nearly never done in the closed source world. Collaboration is also something that is not possible with every open source project out there (just being "open-source" is not enough). Effective collaboration requires projects to be complete, transparent, to have good engineering practices, and to have fair and tranparent ways for you to contribute and influence their evolution.

FreeBSD has good code, and, perhaps more importantly, follows many practices that make it a solid foundation to develop products on.

So I wrote an article to introduce FreeBSD as a toolbox for product development. I've listed the benefits of taking the long-term view, and examined best-practices to follow when collaborating with the FreeBSD project.

This article on the FreeBSD website now: http://www.freebsd.org/doc/en_US.ISO8859-1/articles/building-products/.

Read, enjoy, spread the word!

Cross building FreeBSD

2005-10-26T08:34:00.000-07:00

A Tale of OS Bootstrapping

What do you do when an AMD64 box comes by your way but you only have an 32-bit operating system to run on it? With FreeBSD, you can bootstrap your way into a 64 bit world. Here is what you do:

Install FreeBSD in the usual way. When doing so, keep a spare disk partition for holding the new amd64 OS. In this example, we'll assume that this is partition 'd'. Mount this partition on some convenient mount point, say /1. The full FreeBSD system occupies about ~250MB these days; so make this partition a few GB in size. Don't forget to install the sources to the FreeBSD system of course, otherwise there won't be anything to cross-build.
Change directory to the root of the source tree /usr/src and execute the following commands:
```
# cd /usr/src
# make world TARGET_ARCH=amd64 DESTDIR="/1"
```
This step builds an amd64 userland and populates it into /1. Several interesting things happen as part of this step.
- Since the host OS at this point is running in 32-bit mode it has to build a full cross-toolchain first: tools running on the i386 architecture but manipulating libraries and executables for the amd64 architecture.
- It then needs to compile the whole source tree for the amd64 architecture using tools running on the i386.
- Once compilation succeeds, the build process transfers the newly built binaries into the filesystem rooted at /1.
Other options for TARGET_ARCH include powerpc, sparc64, alpha, ia64.
The next step is to populate a default /etc. This is how you do this:
```
# make distribution TARGET_ARCH=amd64 DESTDIR=/1
```
This step populates /1/etc with the default set of startup scripts.
Now we need an amd64 kernel to boot from:
```
# make kernel KERNCONF=GENERIC TARGET_ARCH=amd64 DESTDIR="/1"
```
This builds a kernel following the configuration specified in the configuration file named /usr/src/sys/amd64/conf/GENERIC and installs it under /1/boot/kernel.
Copy over /etc/fstab to /1/etc/fstab and change the definition of the / (root) mount point to use partition 'd' (the default is to mount / on the 'a' partition).
Copy over /etc/rc.conf to /1/etc/rc.conf, as this contains the machine's hostname, and IP addresses of its interfaces.
Create a file boot.config in the 'a' disk partition containing the following text:
```
0:ad(0,d)/boot/loader
```
This command informs the first stage boot loader (see boot(8)) that we want to boot from partition 'd' instead of the usual partition 'a'. If you've booted from a SCSI disk, you may need to specify the boot device as da(0,d).
Reboot. Viola! You boot into an AMD64 kernel running an AMD64 userland.