Professionally Now Otherwise Engaged - E_FEWTIME

Christopher J. Ruwe

Building ZFS Enabled Live CD for Debian

January 26, 2016. 1641 words.

To safely experiment and work with ZFS root filesystems on Debian-based systems, it is necessary to have fall back options in place. This includes a Live CD with precompiled ZFS modules. As ZFS is not available in the standard Linux kernel and in the standard Linux distributions, I will briefly explain how to build such a Live CD and link to scripts to save the typing.

Rationale

Some use-cases in modern operating system environments, such as boot environment forking or virtual appliance operation and migration are greatly facilitated when using modern copy on write filesystems.

Looking at operating systems of the Unix family, choices basically boil down to btrfs and ZFS. btrfs It’s pronounced better-fs or “b-tree fs”. I know some colleagues who semantically disagree with the first. is a modern filesystem developed at Oracle’s, native to Linux and allegedly rapidly nearing production readiness. Which aformentioned colleagues contest. ZFS is a filesystem developed at Sun’s for then Sun Solaris, now, after the Oracle take-over, Oracle Solaris operating systems.

btrfs is the easy option, as btrfs is available from more recent Linux kernels. In addition increasingly more Linux distributions offer btrfs as an option in their respective installation tools.

While promising, btrfs still lacks some important features and exhibits some strange problems. I have been been bitten trying to boot from a encrypted raid1 related to an open_ctree issueEcrypted btrfs RAID1 array fails to mount at boot. , which apparently has been censored/moderated/withdrawn. Various acquaintances told be about rather crippling performance issues with raid sets in the 1TB scale.

Having moved from Linuxes to OpenSolaris and FreeBSD about 2008, I myself feel more comfortable with ZFS. ZFS support in the Linux Domain is increasing (The withdrawn flocker container persistence orchestratorflocker. for instance offers persistence for Docker containers based on ZFS) and the ZFS On Linux Project ZFS On Linux. gains momentum offering installation packages for Debian and RedHat.

Problem

The drawback of ZFS is that is currently of probably for time to come impossible to ship ZFS modules together with the Linux kernel due to licensing incompatibilities. To have ZFS on Linux, it is necessary to compile the modules for the required kernel locally. dkms packages exist for specific systems, which eases the pain somewhat if one is one one of the supported systems.

Typically, however, ZFS modules are not shipped with the current LiveCD or netinstall images, which requires to include ZFS modules into a customized LiveCD.

Solution

For any Debian-based system, a LiveCD is built using basically n steps:

  1. bootstrap a base chroot,
  2. install necessary packages,
  3. pull and compile ZFS on Linux packages,
  4. install kernel modules and updated modules dependencies and initial ramdisks,
  5. compress the chroot into a live squashfs filesystem and
  6. format an image to be booted from an USB stick or a CDROM.

I will detail these steps with some shell code.

Discussion

Assuming you already have ZFS available, I propose to create a ZFS dataset “chroots” and work from there:

Prepare a Chroot

TRGT="${HOME}/chroots/${DIST}${DIR}"

if [ ! -d $TRGT ]; then
    zfs create rpool${HOME}/chroots/${DIST}${DIR}
        
    debootstrap \
      --arch=amd64 \
      $SUITE \
      $TRGT \
      $MIRROR
fi

if [ ! -d $TRGT/root/bin ]; then
    mkdir -p $TRGT/root/bin
fi

Here, $TRGT denotes a target directory, which is composed of the home directory dataset and a special path for chroots, $DIST signifies the Debian release to be installed and and $DIR a prefix appended to the base dataset. I will continue to work inside the chroot, so I prepare for scripting inside last. You may leave $MIRROR empty, it might be a good idea to point it to some mirror near your location.

Populate the Chroot

Chroot into $TRGT.

sudo chroot $TRGT /bin/bash

There, start installing packages:

apt-get update
apt-get -y install locales
locale-gen en_US.UTF-8

apt-get -y install \
  alien \
  apt-file \
  apt-listchanges \
  apt-transport-https \
  autoconf \
  build-essential \
  cryptsetup \
  dbus \
  deboostrap \
  fakeroot \
  gawk \
  git \
  libtool \
  live-boot \
  lsb-release \
  lsof \
  lsscsi \
  parted \
  ssh \
  sudo \
  wget

apt-get clean

You might want to add a non-root user to the LiveCD.

useradd -s /bin/bash -m live -G sudo
passwd live

Compile Linux from Upstream

Having finished the generic tasks, it is necessary to pull the Linux kernel sources and the sources for ZFS. Compiling the ZFS modules, you will run into error messages concerning the version numbers of the Debian kernel packages. Therefore, it is necessary to use the Linux kernel compiled directly from upstream and have the sources around.

cd $LINUXSRC/..
git clone \
  -b linux-4.1.y \
  --single-branch \
  git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git 
git checkout v${LINUXVERS}
make deb-pkg LOCALVERSION=-amd64 KDEB_PKGVERSION=1   
dpkg -i ../linux*.deb

It might be a good idea to have packages available on the outside, i.e., build packages on your normal system and just copy the packages to the chroot.

Compile the ZFS modules

ZFS kmod modules are compiled against a specific kernel using --with-linux=$LINUX_SOURCES and --with-linux-obj=$LINUX_HEADERS. Having compiled the modules and userland programs, they are installed with dpkg.

for dir in spl zfs
do
  if [ ! -d /usr/src/$dir ]; then
    cd /usr/src &&  git clone git://github.com/zfsonlinux/$dir.git
  else
    cd /usr/src/$dir && git pull
  fi

  cd /usr/src/$dir
  ./autogen.sh
  ./configure \
     --with-linux=${LINUXSRC} \
     --with-linux-obj=${LINUXHDR}
  make deb-utils deb-kmod
  find . -name "*deb" -exec dpkg -i {} \;
done

Likewise, it might good idea to compile kernel kmod modules and package these.

Pack the image

Having sucessfully installed the packages, the custom kernel and the ZFS modules, it remains to pack the resulting directory tree. I use special ZFS datasets to do so, maily to guarantee that I start with a pristine install.

The livefs directory will contain the initramdisk and the kernel as well as the squashfs.

if [ ! -d $LIVE ]; then
    zfs create rpool${HOME}/lives/${DIST}${DIR}
    mkdir -p $LIVE/live
    mkdir -p $LIVE/isolinux
fi

cp $TRGT/boot/vmlinuz-*-amd64 $LIVE/livefs/vmlinuz
cp $TRGT/boot/initrd*amd64 $LIVE/livefs/initrd
mksquashfs $TRGT $LIVE/livefs/filesystem.squashfs -comp xz -e boot

The isolinux directory will contain the boot loader and the bootloader configuration. This last step is not necessary if it is planned to PXE-boot from the network, which I advise.

cp /usr/share/livefs/build/bootloaders/isolinux/isolinux.bin $LIVE/isolinux/.
cp /usr/lib/syslinux/modules/bios/* $LIVE/isolinux/.

cat <<EOF > $LIVE/isolinux/isolinux.cfg
ui menu.c32
prompt 0
menu title Boot Menu
timeout 300

label live-amd64
	menu label ^Live (amd64)
	menu default
	linux /livefs/vmlinuz
 	append initrd=/livefs/initrd boot=live persistence quiet

label live-amd64-failsafe
	menu label ^Live (amd64 failsafe)
	linux /livefs/vmlinuz
	append initrd=/livefs/initrd boot=live persistence config memtest noapic noapm nodma nomce nolapic nomodeset nosmp nosplash vga=normal
EOF

Format the image

Should it be planned to boot from detachable mediums actually, it is necessary to create a burnable image.

xorriso \
  -as mkisofs \
  -r \
	-J \
	-joliet-long \
 	-l \
 	-cache-inodes \
 	-isohybrid-mbr /usr/lib/ISOLINUX/isohdpfx.bin \
 	-partition_offset 16 \
 	-A "Debian Live ${DIST}" \
 	-b isolinux/isolinux.bin \
 	-c isolinux/boot.cat \
 	-no-emul-boot \
 	-boot-load-size 4 \
 	-boot-info-table \
 	-o ${HOME}/tmp/debian-live-${DIST}-$(date +%Y%m%d)-amd64.iso \
 	$LIVE

The paths for -b and -c are relative paths.

The solution I advise places both the initrd and the kernel in the root of a TFTP server and servers the squashfs from a HTTP-server. Care to replace meta-variables in <...> with proper values:

LABEL deblive <distribution> <thedate> (ZFS enabled)
  KERNEL /pub/pxeimg/vmlinuz-deblive-<distribution>-<thedate>-amd64
  APPEND initrd=/pub/pxeimg/initrd-deblive-<distribution>-<thedate>-amd64 dhcp boot=live fetch=http://<HTTPPATH>/squashfs-deblive-<distribution>-<thedate>-amd64.squashfs
Building ZFS Enabled Live CD for Debian - January 26, 2016 - Christopher J. Ruwe