Environment

Joyent’s SmartOS is essentially a Solaris-based hypervisor. It employs Solaris Zones as very lightweight OS-level virtualization containers. In addition, Linux KVM has been ported to the (open-sourced, OpenSolaris-derived) Illumos-flavour of Solaris, adding type-I hypervisor virtualization.

SmartOS is canonically booted from USB-sticks and any “work”, i.e., anything requiring persistent state, is done in virtual machines. Irrespective of any discussion of the (non-) merits of such an approach, most of us would be inclined to agree that an operating system on it’s own is in most circumstances not very useful and typically requires further software to fulfil it’s function.

For SmartOS, Joyent provides an extensive repository of pre-compiled pkgsrc packages. For many settings, this may well suffice. However, in some situations users or administrators desire to compile their software themselves. A pkgsrc-based system enables them to do precisely that.

Various guides on how to efficiently compile packages using pkgsrc can be found. The main manual http://www.netbsd.org/docs/pkgsrc/ is the canonical source of information. Regarding Illumos/SmartOS specifically, Jonathan Perkin has written a series of blog posts starting with http://www.perkin.org.uk/pages/pkgsrc-binary-packages-for-illumos.html.

While all methods have their valid applications, coming from FreeBSD, I myself would like a mechanism which allows me to build software in a pristine environment, i.e. I would like to completely destroy the environment after a compile and compile the next package from a clean slate. I have been nastily bitten by implicit dependencies introduced by autoconf, which happily links against anything to be found on a given system, which can cause quite a headache when things break.

On FreeBSD, Baptiste Daroussin has developed a package called poudriere, which effectively is a testing and compile environment leveraging FreeBSD jails and ZFS. Ignoring many great features of poudriere for the moment, the idea is simple: Populate a chroot directory from a clean base system on a separate dataset. Clone that dataset for n systems, give them ports trees, jail them and compile the packages therein. Afterwards, destroy the clones and start the process anew.

The huge advantage is that any compile system is completely devoid of any packages apart from those specified in the depends-list of said pkgsrc package. So, no implicit autoconf dependencies are introduced, so that I do not get more grey hairs.

While solutions exit which can accomplish that, I wanted to emulate the FreeBSD poudriere solution, as it is clean, high performant and can be accomplished using only system tools. The idea is to create chroot dirs on ZFS datasets and use these to compile pkgsrc packages. Zones cannot, although favourable, be used, as a zone cannot be run from inside a zone.

Infrastructure

I use a Hewlett-Packard ProLiant NL40 and Joyent’s SmartOS. I have violated the “all persistent state in zones” principle a bit and have installed puppet on the hypervisor. I use a self-cooked puppet provider for SmartMachines (Joyent’s Solaris zones), which can be obtained from http://github.com/cruwe/puppet-smartosvm. Using the 0.0.x branch. I define a SmartMachine thus:

smartosvm {"pkg-master" :
  ensure => 'present',
  state => 'running',
  aliasname => 'pkg-master',
  brand => 'joyent',
  image_uuid => '17c98640-1fdb-11e3-bf51-3708ce78e75a',
  hostname => "pkg-master.cruwe.de",
  dns_domain => 'cruwe.de',
  resolver => '192.168.178.1',
  nics_0_ip => '192.168.178.5',
  nics_0_netmask => '255.255.255.0',
  nics_0_gateway => '192.168.178.1',
  nics_0_nic_tag => 'admin',
  nics_0_primary => 'true',
  cpu_cap => '100',
  cpu_shares => '100',
  max_lwps => '2000',
  max_locked_memory => '4096',
  max_physical_memory => '4096',
  max_swap => '8192',
  tmpfs => '1024',
  quota => '50',
  zfs_io_priority => '200',
  zfs_root_compression => 'off',
  zfs_root_recsize => '128K',
}

On SmartOS you would use vmadm, which yields on my system

[root@ritchie ~]# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
bdf1898a-1e29-4a4d-9075-b60a037f2838  KVM   256      stopped           davical
b63eb831-1ed2-4c13-8b08-5d8db6c34e91  OS    2048     running           wordpress
ca6e9139-217a-49f3-a7b7-d8b589c0cdfa  OS    2048     running           filesrv
b51ebbff-6587-49d6-8ef6-e51abf702039  OS    4096     running           pkg-master
c596758d-d75c-4944-8703-1f9da8125cbb  OS    32768    running           puppet

Note the UUID of your freshly configured machine.

Create a dataset to delegate, delegate and reboot:

[root@ritchie ~]# zfs create zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld

[root@ritchie ~]# zonecfg -z b51ebbff-6587-49d6-8ef6-e51abf702039

[root@ritchie ~]# zonecfg:b51ebbff-6587-49d6-8ef6-e51abf702039> add dataset
[root@ritchie ~]# zonecfg:b51ebbff-6587-49d6-8ef6-e51abf702039:dataset> set name=zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld
[root@ritchie ~]# zonecfg:b51ebbff-6587-49d6-8ef6-e51abf702039:dataset> end
[root@ritchie ~]# zonecfg:b51ebbff-6587-49d6-8ef6-e51abf702039> exit

[root@ritchie ~]# vmadm reboot b51ebbff-6587-49d6-8ef6-e51abf702039

Note that in order to have working NICs, you need to reboot the VM anyway, my puppet provider is slightly quirky. Use zlogin -z or ssh to work in the zone.

Build

Inside the zone, create further datasets which will later be used by the pkgsrc build environment:

Install the Compiler

[root@pkg-master ~]# $DELEGATE=zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld

[root@pkg-master ~]# zfs set mountpoint=/pkgbld $DELEGATE

[root@pkg-master ~]# for ds in pkgsrc distfiles packages var var/wrkobjdirs chroot chroot/proto
do
    zfs create $DELEGATE/$ds
done

Install the necessary software

[root@pkg-master ~]# pkgin in gcc47 scmgit-base

Remember to snapshot the zone itself from the outside.

[root@ritchie ~]# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039@w_sw

Get pkgsrc

I have a local copy of joyent-pkgsrc to reduce outbound network traffic, so I clone from the “inside”. I advise for such a setup, as usually one is bound make to make mistakes and local Ethernet is way faster than outbound DSL.

[root@pkg-master ~]# git clone root@filesrv:/export/mirrors/joyent-pkgsrc /pkgbld/pkgsrc 
[root@pkg-master ~]# cd /pkgbld/pkgsrc
[root@pkg-master /pkgbld/pkgsrc]# git checkout -b pkgsrc_2013Q2
[root@pkg-master /pkgbld/pkgsrc]# git status
# On branch pkgsrc_2013Q2
nothing to commit, working directory clean

Build the chroot

Create, populate and start a chroot dir

[root@pkg-master ~]# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/pkgsrc@clean

[root@pkg-master ~]# for dir in bin etc lib opt sbin tmp usr var
> do
>    rsync -avh /$dir /pkgbld/chroot/proto
> done 
[root@pkg-master ~]# for dir in \
>     dev \
>     pkgbld \
>     pkgbld/pkgsrc \
>     pkgbld/distfiles \
>     pkgbld/packages \
>     pkgbld/var \
>     pkgbld/var/wrkobjdirs \
>     proc
> do
>     mkdir /pkgbld/chroot/proto/$dir
> done

[root@pkg-master ~]# mount -F lofs /dev /pkgbld/chroot/proto/dev
[root@pkg-master ~]# mount -F lofs /proc /pkgbld/chroot/proto/proc
[root@pkg-master ~]# mount -F lofs -o ro /pkgbld/pkgsrc /pkgbld/chroot/proto/pkgbld/pkgsrc
[root@pkg-master ~]# for rw_dir in distfiles packages var/wrkobjdirs
> do
> mount -F lofs -o rw /pkgbld/$rw_dir /pkgbld/chroot/proto/pkgbld/$rw_dir
> done

[root@pkg-master ~]# chroot /pkgbld/chroot/proto /bin/bash

Configure the mk.conf

It is necessary to adapt the configuration file mk.conf, which is to be found in /opt/local/etc/mk.conf. We will later move it when we detach the old prefix and switch to the new.

.ifdef BSD_PKG_MK       # begin pkgsrc settings

ABI=                    64
PKGSRC_COMPILER=        gcc
GCCBASE=                /opt/local/gcc47

PKG_TOOLS_BIN=          /opt/local/sbin

PKGSRCDIR=              /pkgbld/pkgsrc
DISTDIR=                /pkgbld/distfiles
PACKAGES=               /pkgbld/packages/joyent-2013Q3
WRKOBJDIR=              /pkgbld/var/wrkobjdirs

PREFIX=                 /opt/pkg
LOCALBASE=              /opt/pkg
VARBASE=                /opt/pkg/var
PKG_DBDIR=              /opt/pkg/var/db/pkg

BINPKG_SITES=           file:///pkgbld/packages/joyent-2013Q3
DEPENDS_TARGET=         bin-install

PKGINFODIR=             info
PKGMANDIR=              man

PREFER_PKGSRC=          yes
PREFER_NATIVE=          libexecinfo solaris-pam terminfo

TOOLS_PLATFORM.install?=        /opt/pkg/bin/install-sh
TOOLS_PLATFORM.sh?=             /opt/pkg/bin/pdksh
TOOLS_PLATFORM.ksh?=            /opt/pkg/bin/pdksh
TOOLS_PLATFORM.awk?=            /opt/pkg/bin/nawk
TOOLS_PLATFORM.sed?=            /opt/pkg/bin/nbsed
TOOLS_PLATFORM.sh?=             /usr/bin/bash

ACCEPTABLE_LICENSES+=   gnu-gpl-v2 ruby-license

.endif                  # end pkgsrc settings

Bootstrap

Save your clean state and bootstrap

bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@clean
bash-4.1# cd /pkgbld/pkgsrc/bootstrap
bash-4.1# ./bootstrap \
    --abi=64 \
    --full \
    --pkgdbdir=/opt/pkg/var/db/pkg \
    --pkgmandir=man \
    --prefix=/opt/pkg \
    --sysconfdir=/opt/pkg/etc \
    --varbase=/opt/pkg/var \
    --workdir=/pkgbld/var/wrkobjdirs/bootstrap

Wait. Drink some mate. Check your configuration when you are done.

bash-4.1# cd /pkgbld/pkgsrc/lang/gcc47
bash-4.1# bmake show-var VARNAME=CONFIGURE_ENV
LDFLAGS_FOR_TARGET=
DL_CFLAGS=
DL_LDFLAGS= 
DL_LIBS= 
PTHREAD_CFLAGS=-D_REENTRANT 
PTHREAD_LDFLAGS= 
PTHREAD_LIBS=-lpthread\ -lrt
PTHREADBASE=/usr
INSTALL_INFO=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.tools/bin/install-info
MAKEINFO=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.tools/bin/makeinfo
FLEX=
BISON=
AWK=/opt/pkg/bin/nawk
CAT=/usr/bin/cat
ac_cv_path_CAT=/usr/bin/cat
CHMOD=/usr/bin/chmod
CMP=/bin/cmp
CP=/bin/cp
DIFF=/usr/bin/diff
ECHO=echo
ac_cv_path_ECHO=echo
EGREP=/usr/xpg4/bin/grep\ -E
ac_cv_path_EGREP=/usr/xpg4/bin/grep\ -E
SETENV=/usr/bin/env
ENV_PROG=/usr/bin/env
ac_cv_path_ENV=/usr/bin/env
FALSE=false
ac_cv_path_FALSE=false
FIND=/usr/bin/find
GREP=/usr/xpg4/bin/grep
ac_cv_path_GREP=/usr/xpg4/bin/grep
HOSTNAME=/bin/hostname
LN=/usr/bin/ln
LS=/usr/bin/ls
MKDIR=/usr/bin/mkdir\ -p 
MV=/usr/bin/mv
PERL=/opt/pkg/bin/perl
PERL_PATH=/opt/pkg/bin/perl
RM=/usr/bin/rm
RMDIR=/usr/bin/rmdir
SED=/opt/pkg/bin/nbsed
SORT=/usr/bin/sort
TAR=/usr/bin/gtar
TEST=test
ac_cv_path_TEST=test
TOUCH=/usr/bin/touch
TR=/usr/xpg4/bin/tr
TRUE=true
ac_cv_path_TRUE=true
PKG_CONFIG=
PKG_CONFIG_LIBDIR=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.buildlink/lib/pkgconfig:\
/pkgbld/var/wrkobjdirs/lang/gcc47/work/.buildlink/share/pkgconfig
PKG_CONFIG_LOG=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.pkg-config.log
PKG_CONFIG_PATH=
MAKE=make
CC=gcc
CFLAGS=-O
CPPFLAGS=
CXX=g++
CXXFLAGS=-O
COMPILER_RPATH_FLAG=-Wl,-R
F77=f77
FC=f77
FFLAGS=-O
LANG=C
LC_ALL=C
LC_COLLATE=C
LC_CTYPE=C
LC_MESSAGES=C
LC_MONETARY=C
LC_NUMERIC=C
LC_TIME=C
LDFLAGS=
LINKER_RPATH_FLAG=-R
PATH=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.wrapper/bin:/pkgbld/var/wrkobjdirs/lang/gcc47/work/.buildlink/bin:\
/pkgbld/var/wrkobjdirs/lang/gcc47/work/.gcc/bin:/pkgbld/var/wrkobjdirs/lang/gcc47/work/.tools/bin: \
/opt/pkg/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbin:/opt/pkg/bin:/opt/pkg/bin
PREFIX=/opt/pkg
PKG_SYSCONFDIR=/opt/pkg/etc
CXXCPP=cpp
HOME=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.home
CONFIG_SHELL=/opt/pkg/bin/pdksh
LIBS= 
ac_given_INSTALL=/opt/pkg/bin/install-sh\ -c\ -o\ root\ -g\ root

It is important here to check whether $PREFIX and other paths are set correctly.

After the bootstrap build, move the old mk.conf to the new prefix (/opt/pkg) and link it to /opt/local. Edit mk.conf to reflect the new $PKG_TOOLS_BIN. Snapshot it.

bash-4.1# mv /opt/local/etc/mk.conf /opt/pkg/etc/  
bash-4.1# ln -s /opt/pkg/etc/mk.conf /opt/local/etc/
bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@bootstrapped

Build Your Own Compiler

Build the compiler, libarchive and rebuild libtool-base. Note that perl5 has difficulties building against dtrace. Building gcc47 can take a long time.

bash-4.1# cd /pkgbld/pkgsrc/lang/perl5
bash-4.1# bmake PKG_OPTIONS.perl="-dtrace" bin-install
bash-4.1# cd /pkgbld/pkgsrc/lang/gcc47
bash-4.1# bmake bin-install
bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@bootstrapped-w_gcc 
bash-4.1# PATH=/opt/pkg/gcc47/bin:/opt/pkg/gnu/bin:/opt/pkg/bin:/opt/pkg/sbin:/usr/bin:/usr/sbin 
bash-4.1# cd /pkgbld/pkgsrc/archivers/libarchive/
bash-4.1# bmake bin-install
bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@bootstrapped-w_gcc-w_libarchive 

Remove the old prefix and rebuild libtool-base.

bash-4.1# rm -R /opt/local

bash-4.1# cd /pkgbld/pkgsrc/devel/libtool-base
bash-4.1# bmake deinstall
bash-4.1# rm /pkgbld/packages/2013Q2/All/libtool-base-2.4.2nb4.tgz 
bash-4.1# rm /pkgbld/packages/2013Q2/devel/libtool-base-2.4.2nb4.tgz 
bash-4.1# bmake clean-depends
bash-4.1# bmake install
bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@bootstrapped-w_gcc-w_libarchive-w_libtool_base 

At this point, you have a working, self-built prefix to build pkgsrc pacakges from. Some packages require intensive love and care to build, which is a bit disappointing. I suspect some tiny details on setting a different $PREFIX have escaped my notice. Should I find out, I will post here.