tl;dr

ARMv6 Raspberry Pi mini-computers offer an power-economical soultion to provision networking appliances for small to medium scale networks. This article shows how to circumvent the low computation power of these machines when building software in a cross-compilation setup.

Problem

A serious computing infrastructure depends on a set of network services available to all connected clients. Networked computers in a small to medium (home?) office depend on services for network address assignment and address resolution, i.e., naming, as a minimum. Other services such as central authentication or authorization follow.

Addresses can be assigned statically and naming can be provided by entries in the host-DB or any other database available to nsdispatch(3), which is configurable in nsswitch.conf(5). With rising numbers of networked computers, the necessity for (manual?) configuration poses a problem. Therefore, usually name- and DHCP-servers are used for naming and address assignment respectively. The same is true for many other domains of application.

Most small home or small office routers offer basic DHCP and naming services. However, usually these do not offer the same possibilities for configuration as required. Consequently, administrators may wish to run dedicated servers for DHCP and DNS.

Setting up a separate PC for DHCP and DNS is dubious, however. Even when ignoring hardware costs when salvaging some low end machine from the scrap heap, costs for electric power can (and will) quickly reach proportions where such a setup is seriously debatable.

With a maximum power intake of 3.5 W, the Raspberry Pi ARMv6 machines offers an economically feasible alternative to provision low capacity network appliances on. Linux distributions like ArchLinux or Raspbian exist and are able to install and run various network appliances.

For several reasons, I wish to run FreeBSD. On FreeBSD, binary packages are available only for tier-1 architectures, which are i386 and amd64. For armv6, packages need to be compiled from FreeBSD ports. While I like to customize my packages anyway, building packages from source on an ARMv6 machine is a tedious task.

Solution

FreeBSD offers a different solution. With clang as the standard compiler, the FreeBSD sources can be compiled for different processor architectures. In addition, since January 2014, snapshot builds for ARMv6 are available via ftp. Regional mirrors exist. Installing the set of ARMv6-binaries in a separate $DESTDIR and running these binaries from emulators/qemu-devel allows to compile on vastly more powerful machines.

Discussion

The QemuUserModeHowTo on the FreeBSD wiki pages offers a basic introduction in the process. However, I found these to be at times stale and at times lacking vital information. I suspect my problems are so basic or obvious to developers that no-one thought them worthwhile mentioning.

The task is relatively straight-forward:

Virtual Base System

Download (or compile) a recent snapshot of FreeBSD for ARMv6 Raspberry.

Set up an installation of FreeBSD for i386 in a virtual machine. I use SmartOS and KVM. (Careful: Packages will not link properly when cross-compiling on amd64. ARMv6 is 32b, so linking in a 64b address space will fail!)

emulators/qemu-devel

Install emulators/qemu-devel on the i386. Uncheck all X11 and graphics related options, check STATIC_LINK.

[root@ritchie ~] cd /usr/ports/emulators/qemu-devel
[root@ritchie ~] make config
[root@ritchie ~] make configure
[root@ritchie ~] cd work/qemu-*
[root@ritchie ~] gmake

Set the environment in your virtual i386 machine to

export TARGET=arm
export TARGET_ARCH=armv6
export DESTDIR=/vms/chroots/`uname -r`arm

Extract the ARMv6 binaries to $DESTDIR on the i386 virtual machine, copy a ports tree to $DESTDIR and mount devfs.

Then, install the emulators/qemu-devel ARMv6 emulator to $DESTDIR:

[root@ritchie ~] mkdir -p $DESTDIR/usr/local/bin
[root@ritchie ~] cp /usr/ports/emulators/qemu-devel/work/qemu-*/arm-bsd-user/qemu-arm \
                    $DESTDIR/usr/local/bin

Start the chroot

Chroot into $DESTDIR, check your architecture and initialize ldconfig(8):

[root@ritchie ~] mdconfig -a -t vnode -f <snapshot_file> -u 0
[root@ritchie ~] mkdir /mnt/rpi
[root@ritchie ~] mount /dev/ufs/<snapshot_ufsid> /mnt/rpi
[root@ritchie ~] cp -R /mnt/rpi $DESTDIR
[root@ritchie ~] umount /mnt/rpi
[root@ritchie ~] mdconfig -d -u 0

[root@ritchie ~] cp -R /usr/ports $DESTDIR/usr/ports
[root@ritchie ~] mount -t devfs devfs $DESTDIR/dev
[root@ritchie ~] cp /etc/resolv.conf $DESTDIR/etc/resolv.conf
[root@ritchie ~] chroot $DESTDIR /usr/local/bin/qemu-arm /bin/sh
[root@ritchie ~] file /bin/ls
[root@ritchie ~] service ldconfig onestart

Compile

From this point on, you can start compiling. Note that cross-architecture jails do not work. So, compiling with ports-mgmt/poudriere is no option when cross-building.

Native Tool-Chain

To speed things up, it is possible to use a native, i.e., i386 clang inside the ARMv6 chroot. To do so, compile a cross-build toolchain on the i386 host and copy that toolchain into the chroot. Replace the toolchain’s cross-compiler’s include-dir with the chroot ARMv6 inlude-dir and set the chroot’s environment to use the native cross-compiler by .profile.

[root@ritchie ~] cd /usr/src
[root@ritchie ~] mkdir -p /usr/obj
[root@ritchie ~] export MAKEOBJDIRPREFIX=/usr/obj
[root@ritchie ~] make -j 4 toolchain TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH}

[root@ritchie ~] mkdir -p ${DESTDIR}/usr/obj
[root@ritchie ~] cd /usr/obj
[root@ritchie ~] mv $TARGET.$TARGET_ARCH ${DESTDIR}/usr/obj

[root@ritchie ~] chroot -u root $DESTDIR /usr/local/bin/qemu-arm /bin/sh
[root@ritchie ~] cd /usr/obj/${TARGET}.${TARGET_ARCH}/usr/src/tmp/usr
[root@ritchie ~] rm -rf include
[root@ritchie ~] ln -s ../../../../../../include ./include

Inside the chroot, append this

CC=/usr/obj/arm.armv6/usr/src/tmp/usr/bin/cc
export CC
CPP=/usr/obj/arm.armv6/usr/src/tmp/usr/bin/cpp
export CPP
CXX=/usr/obj/arm.armv6/usr/src/tmp/usr/bin/c++
export CXX
AS=/usr/obj/arm.armv6/usr/src/tmp/usr/bin/as
export AS
NM=/usr/obj/arm.armv6/usr/src/tmp/usr/bin/nm
export NM
LD=/usr/obj/arm.armv6/usr/src/tmp/usr/bin/ld
export LD
OBJCOPY=/usr/obj/arm.armv6/usr/src/tmp/usr/bin/objcopy
export OBJCOPY
SIZE=/usr/obj/arm.armv6/usr/home/src/tmp/usr/bin/size
export SIZE
STRIPBIN=/usr/obj/arm.armv6/usr/src/tmp/usr/bin/strip
export STRIPBIN

to root.profile. Source the .profile ( cd /root; . .profile) and start compiling.