Compiling packages from pkgsrc on a SmartOS machine
December 8, 2013. 2203 words.
When the native packaging system does not suffice, it is possible to help oneself. That sort of self-reliance is proven, and yes, it works on OpenSolaris spinn-offs and thus on SmartOS.
Environment
Joyent’s SmartOS is essentially a Solaris-based hypervisor. It employs Solaris Zones as very lightweight OS-level virtualization containers. In addition, Linux KVM has been ported to the (open-sourced, OpenSolaris-derived) Illumos-flavour of Solaris, adding type-I hypervisor virtualization.
SmartOS is canonically booted from USB-sticks and any “work”, i.e., anything requiring persistent state, is done in virtual machines. Irrespective of any discussion of the (non-) merits of such an approach, most of us would be inclined to agree that an operating system on it’s own is in most circumstances not very useful and typically requires further software to fulfil it’s function. An operating system without users and without software has it’s own inherent aesthetics. However, that sort of beauty can only be appreciated by the advanced and most customers have not yet gotten to that point.
For SmartOS, Joyent provides an extensive repository of pre-compiled pkgsrc packagesJoyent pkgsrc. . For many settings, this may well suffice. However, in some situations users or administrators desire to compile their software themselves. A pkgsrc-based system enables them to do precisely that.
Various guides on how to efficiently compile packages using pkgsrc can be found. The main manualThe pkgsrc guide. is the canonical source of information. Regarding Illumos/SmartOS specifically, Jonathan Perkin has written a series of blog posts on this topicPkgsrc Binary Packages for Illumos. .
While all methods have their valid applications, coming from FreeBSD, I myself would like a mechanism which allows me to build software in a pristine environment, i.e. I would like to completely destroy the environment after a compile and compile the next package from a clean slate. I have been nastily bitten by implicit dependencies introduced by autoconf, which happily links against anything to be found on a given system, which can cause quite a headache when things break.
On FreeBSD, Baptiste Daroussin has developed a package called poudriereFreeBSD Handbook, §4.6. Building Packages with Poudriere , which effectively is a testing and compile environment leveraging FreeBSD jails and ZFS. Ignoring many great features of poudriere for the moment, the idea is simple: Populate a chroot directory from a clean base system on a separate dataset. Clone that dataset for n systems, give them ports trees, jail them and compile the packages therein. Afterwards, destroy the clones and start the process anew.
The huge advantage is that any compile system is completely devoid of any packages apart from those specified in the depends-list of said pkgsrc package. So, no implicit autoconf dependencies are introduced, so that I do not get more grey hairs. Errata: Marriage to a not only beautiful, but at times slightly jealous woman has introduced me to the concept that a certain amount of strategically placed grey hair may have it’s merits at times.
While solutions exit which can accomplish that, I wanted to emulate the FreeBSD poudriere solution, as it is clean, high performant and can be accomplished using only system tools. The idea is to create chroot dirs on ZFS datasets and use these to compile pkgsrc packages. Zones cannot, although favourable, be used, as a zone cannot be run from inside a zone.
Infrastructure
I use a Hewlett-Packard ProLiant NL40 and Joyent’s SmartOS. I have violated the “all persistent state in zones” principle a bit and have installed puppet on the hypervisor. I use a self-cooked puppet provider for SmartMachines (Joyent’s Solaris zones), which can be obtained from githubhttp://github.com/cruwe/puppet-smartosvm . Using the 0.0.x branch, I define a SmartMachine thus:
smartosvm {"pkg-master" :
ensure => 'present',
state => 'running',
aliasname => 'pkg-master',
brand => 'joyent',
image_uuid => '17c98640-1fdb-11e3-bf51-3708ce78e75a',
hostname => "pkg-master.cruwe.de",
dns_domain => 'cruwe.de',
resolver => '192.168.178.1',
nics_0_ip => '192.168.178.5',
nics_0_netmask => '255.255.255.0',
nics_0_gateway => '192.168.178.1',
nics_0_nic_tag => 'admin',
nics_0_primary => 'true',
cpu_cap => '100',
cpu_shares => '100',
max_lwps => '2000',
max_locked_memory => '4096',
max_physical_memory => '4096',
max_swap => '8192',
tmpfs => '1024',
quota => '50',
zfs_io_priority => '200',
zfs_root_compression => 'off',
zfs_root_recsize => '128K',
}
On SmartOS you would use vmadm, which yields on my system
[root@ritchie ~]# vmadm list
UUID TYPE RAM STATE ALIAS
bdf1898a-1e29-4a4d-9075-b60a037f2838 KVM 256 stopped davical
b63eb831-1ed2-4c13-8b08-5d8db6c34e91 OS 2048 running wordpress
ca6e9139-217a-49f3-a7b7-d8b589c0cdfa OS 2048 running filesrv
b51ebbff-6587-49d6-8ef6-e51abf702039 OS 4096 running pkg-master
c596758d-d75c-4944-8703-1f9da8125cbb OS 32768 running puppet
Note the UUID of your freshly configured machine. Create a dataset to delegate, delegate and reboot:
[root@ritchie ~]# zfs create \
zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld
[root@ritchie ~]# zonecfg \
-z b51ebbff-6587-49d6-8ef6-e51abf702039
zonecfg:b51ebbff-6587-49d6-8ef6-e51abf702039> add dataset
zonecfg:b51ebbff-6587-49d6-8ef6-e51abf702039:dataset> set name=zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld
zonecfg:b51ebbff-6587-49d6-8ef6-e51abf702039:dataset> end
zonecfg:b51ebbff-6587-49d6-8ef6-e51abf702039> exit
[root@ritchie ~]# vmadm reboot \
b51ebbff-6587-49d6-8ef6-e51abf702039
Note that in order to have working NICs, you need to reboot the VM anyway, my puppet provider is slightly quirky. Use zlogin -z or ssh to work in the zone.
Build
Inside the zone, create further datasets which will later be used by the pkgsrc build environment:
Install the Compiler
[root@pkg-master ~]# $DELEGATE=zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld
[root@pkg-master ~]# zfs set mountpoint=/pkgbld $DELEGATE
[root@pkg-master ~]# for ds in pkgsrc distfiles packages var var/wrkobjdirs chroot chroot/proto
do
zfs create $DELEGATE/$ds
done
Install the necessary software
[root@pkg-master ~]# pkgin in gcc47 scmgit-base
Remember to snapshot the zone itself from the outside.
[root@ritchie ~]# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039@w_sw
Get pkgsrc
I have a local copy of joyent-pkgsrc to reduce outbound network traffic, so I clone from the “inside”. I advise for such a setup, as usually one is bound make to make mistakes and local Ethernet is way faster than outbound DSL.
[root@pkg-master ~]# git clone root@filesrv:/export/mirrors/joyent-pkgsrc /pkgbld/pkgsrc
[root@pkg-master ~]# cd /pkgbld/pkgsrc
[root@pkg-master /pkgbld/pkgsrc]# git checkout -b pkgsrc_2013Q2
[root@pkg-master /pkgbld/pkgsrc]# git status
# On branch pkgsrc_2013Q2
nothing to commit, working directory clean
Build the chroot
Create, populate and start a chroot dir
[root@pkg-master ~]# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/pkgsrc@clean
[root@pkg-master ~]# for dir in bin etc lib opt sbin tmp usr var
> do
> rsync -avh /$dir /pkgbld/chroot/proto
> done
[root@pkg-master ~]# for dir in \
> dev \
> pkgbld \
> pkgbld/pkgsrc \
> pkgbld/distfiles \
> pkgbld/packages \
> pkgbld/var \
> pkgbld/var/wrkobjdirs \
> proc
> do
> mkdir /pkgbld/chroot/proto/$dir
> done
[root@pkg-master ~]# mount -F lofs /dev /pkgbld/chroot/proto/dev
[root@pkg-master ~]# mount -F lofs /proc /pkgbld/chroot/proto/proc
[root@pkg-master ~]# mount -F lofs -o ro /pkgbld/pkgsrc /pkgbld/chroot/proto/pkgbld/pkgsrc
[root@pkg-master ~]# for rw_dir in distfiles packages var/wrkobjdirs
> do
> mount -F lofs -o rw /pkgbld/$rw_dir /pkgbld/chroot/proto/pkgbld/$rw_dir
> done
[root@pkg-master ~]# chroot /pkgbld/chroot/proto /bin/bash
Configure the mk.conf
It is necessary to adapt the configuration file mk.conf, which is to be found in /opt/local/etc/mk.conf. We will later move it when we detach the old prefix and switch to the new.
.ifdef BSD_PKG_MK # begin pkgsrc settings
ABI= 64
PKGSRC_COMPILER= gcc
GCCBASE= /opt/local/gcc47
PKG_TOOLS_BIN= /opt/local/sbin
PKGSRCDIR= /pkgbld/pkgsrc
DISTDIR= /pkgbld/distfiles
PACKAGES= /pkgbld/packages/joyent-2013Q3
WRKOBJDIR= /pkgbld/var/wrkobjdirs
PREFIX= /opt/pkg
LOCALBASE= /opt/pkg
VARBASE= /opt/pkg/var
PKG_DBDIR= /opt/pkg/var/db/pkg
BINPKG_SITES= file:///pkgbld/packages/joyent-2013Q3
DEPENDS_TARGET= bin-install
PKGINFODIR= info
PKGMANDIR= man
PREFER_PKGSRC= yes
PREFER_NATIVE= libexecinfo solaris-pam terminfo
TOOLS_PLATFORM.install?= /opt/pkg/bin/install-sh
TOOLS_PLATFORM.sh?= /opt/pkg/bin/pdksh
TOOLS_PLATFORM.ksh?= /opt/pkg/bin/pdksh
TOOLS_PLATFORM.awk?= /opt/pkg/bin/nawk
TOOLS_PLATFORM.sed?= /opt/pkg/bin/nbsed
TOOLS_PLATFORM.sh?= /usr/bin/bash
ACCEPTABLE_LICENSES+= gnu-gpl-v2 ruby-license
.endif # end pkgsrc settings
Bootstrap
Save your clean state and bootstrap
bash-4.1# zfs snapshot \
zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@clean
bash-4.1# cd /pkgbld/pkgsrc/bootstrap
bash-4.1# ./bootstrap \
--abi=64 \
--full \
--pkgdbdir=/opt/pkg/var/db/pkg \
--pkgmandir=man \
--prefix=/opt/pkg \
--sysconfdir=/opt/pkg/etc \
--varbase=/opt/pkg/var \
--workdir=/pkgbld/var/wrkobjdirs/bootstrap
Wait. Drink some mate. Check your configuration when you are done.
bash-4.1# cd /pkgbld/pkgsrc/lang/gcc47
bash-4.1# bmake show-var VARNAME=CONFIGURE_ENV
ac_cv_path_CAT=/usr/bin/cat
ac_cv_path_ECHO=echo
ac_cv_path_EGREP=/usr/xpg4/bin/grep\ -E
ac_cv_path_ENV=/usr/bin/env
ac_cv_path_FALSE=false
ac_cv_path_GREP=/usr/xpg4/bin/grep
ac_cv_path_TEST=test
ac_cv_path_TRUE=true
ac_given_INSTALL=/opt/pkg/bin/install-sh\ -c\ -o\ root\ -g\ root
AWK=/opt/pkg/bin/nawk
BISON=
CAT=/usr/bin/cat
CC=gcc
CFLAGS=-O
CHMOD=/usr/bin/chmod
CMP=/bin/cmp
COMPILER_RPATH_FLAG=-Wl,-R
CONFIG_SHELL=/opt/pkg/bin/pdksh
CP=/bin/cp
CPPFLAGS=
CXX=g++
CXXCPP=cpp
CXXFLAGS=-O
DIFF=/usr/bin/diff
DL_CFLAGS=
DL_LDFLAGS=
DL_LIBS=
ECHO=echo
EGREP=/usr/xpg4/bin/grep\ -E
ENV_PROG=/usr/bin/env
F77=f77
FALSE=false
FC=f77
FFLAGS=-O
FIND=/usr/bin/find
FLEX=
GREP=/usr/xpg4/bin/grep
HOME=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.home
HOSTNAME=/bin/hostname
INSTALL_INFO=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.tools/bin/install-info
LANG=C
LC_ALL=C
LC_COLLATE=C
LC_CTYPE=C
LC_MESSAGES=C
LC_MONETARY=C
LC_NUMERIC=C
LC_TIME=C
LDFLAGS=
LDFLAGS_FOR_TARGET=
LIBS=
LINKER_RPATH_FLAG=-R
LN=/usr/bin/ln
LS=/usr/bin/ls
MAKE=make
MAKEINFO=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.tools/bin/makeinfo
MKDIR=/usr/bin/mkdir\ -p
MV=/usr/bin/mv
PATH=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.wrapper/bin:/pkgbld/var/wrkobjdirs/lang/gcc47/work/.buildlink/bin:/pkgbld/var/wrkobjdirs/lang/gcc47/work/.gcc/bin:/pkgbld/var/wrkobjdirs/lang/gcc47/work/.tools/bin:/opt/pkg/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/usr/sbin:/opt/pkg/bin:/opt/pkg/bin
PERL=/opt/pkg/bin/perl
PERL_PATH=/opt/pkg/bin/perl
PKG_CONFIG=
PKG_CONFIG_LIBDIR=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.buildlink/lib/pkgconfig:/pkgbld/var/wrkobjdirs/lang/gcc47/work/.buildlink/share/pkgconfig
PKG_CONFIG_LOG=/pkgbld/var/wrkobjdirs/lang/gcc47/work/.pkg-config.log
PKG_CONFIG_PATH=
PKG_SYSCONFDIR=/opt/pkg/etc
PREFIX=/opt/pkg
PTHREAD_CFLAGS=-D_REENTRANT
PTHREAD_LDFLAGS=
PTHREAD_LIBS=-lpthread\ -lrt
PTHREADBASE=/usr
RM=/usr/bin/rm
RMDIR=/usr/bin/rmdir
SED=/opt/pkg/bin/nbsed
SETENV=/usr/bin/env
SORT=/usr/bin/sort
TAR=/usr/bin/gtar
TEST=test
TOUCH=/usr/bin/touch
TR=/usr/xpg4/bin/tr
TRUE=true
It is important here to check whether $PREFIX and other paths are set correctly.
After the bootstrap build, move the old mk.conf to the new prefix (/opt/pkg) and link it to /opt/local. Edit mk.conf to reflect the new $PKG_TOOLS_BIN. Snapshot it.
bash-4.1# mv /opt/local/etc/mk.conf /opt/pkg/etc/
bash-4.1# ln -s /opt/pkg/etc/mk.conf /opt/local/etc/
bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@bootstrapped
Build Your Own Compiler
Build the compiler, libarchive and rebuild libtool-base. Note that perl5 has difficulties building against dtrace. Building gcc47 can take a long time.
bash-4.1# cd /pkgbld/pkgsrc/lang/perl5
bash-4.1# bmake PKG_OPTIONS.perl="-dtrace" bin-install
bash-4.1# cd /pkgbld/pkgsrc/lang/gcc47
bash-4.1# bmake bin-install
bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@bootstrapped-w_gcc
bash-4.1# PATH=/opt/pkg/gcc47/bin:/opt/pkg/gnu/bin:/opt/pkg/bin:/opt/pkg/sbin:/usr/bin:/usr/sbin
bash-4.1# cd /pkgbld/pkgsrc/archivers/libarchive/
bash-4.1# bmake bin-install
bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@bootstrapped-w_gcc-w_libarchive
Remove the old prefix and rebuild libtool-base.
bash-4.1# rm -R /opt/local
bash-4.1# cd /pkgbld/pkgsrc/devel/libtool-base
bash-4.1# bmake deinstall
bash-4.1# rm /pkgbld/packages/2013Q2/All/libtool-base-2.4.2nb4.tgz
bash-4.1# rm /pkgbld/packages/2013Q2/devel/libtool-base-2.4.2nb4.tgz
bash-4.1# bmake clean-depends
bash-4.1# bmake install
bash-4.1# zfs snapshot zones/b51ebbff-6587-49d6-8ef6-e51abf702039/pkgbld/chroot/proto@bootstrapped-w_gcc-w_libarchive-w_libtool_base
At this point, you have a working, self-built prefix to build pkgsrc pacakges from. Some packages require intensive love and care to build, which is a bit disappointing. I suspect some tiny details on setting a different $PREFIX have escaped my notice. Should I find out, I will post here.