Using Docker on SmartOS Hypervisors

January 27, 2016. 1222 words.

SmartOS is a OpenSolaris based hypervisor consisting of an Illumos environment severely stripped down, QEMU and a port of the Linux KVM module to Solaris. With the no anymore so recent addition of LX-branded zones and docker to SmartOS, it is possible to conveniently provision docker containers on SmartOS.

Problem

The SmartOS documentationSmartOS: The Complete Modern Operating System. is rather sparse in the are on how to efficiently run docker containers in a SmartOS environment. The use case for me is that I run my “lab” using SmartOS zones, a KVM-virtualized firewall appliance and various Linuxes. It is rather efficient to separate services Linuxes using docker, which is why I do it.

Solution

To effectively operate on docker containers in a lab setting, it is necessary to have

1. a provisioned SmartOS host,
2. dockerized zone manifests and
3. a private, insecure docker registry for the use by SmartOS.

Optionally, you might wish to provide for

1. a web interface to the docker registry,
2. an SSL-terminating proxy to the docker registry and possibly
3. a different, secured docker registry for the use from the “outside”.

The last two are deferred to the interested reader as an exercise or possibly to a later post.

Discussion

On SmartOS, LX-branded zones are zones which use a Linux to Solaris system call translation layer. As special zones, they profit having storage from dedicated ZFS datasets, having CPU and I/O limits set by the Solaris resource consumption controls and having a “private” networking stack from the Crossbow framework.

Configuration of SmartOS Hypervisor Host

Assuming a otherwise running and configured SmartOS hypervisor, docker image sources are configured as dataset sources using imgadm(1M)imgadm(1M). . It is necessary to configure additional image sources for docker images and consequently import the corresponding images as datasets.

[root@d0-50-99-46-c1-59 ~]# imgadm sources 
https://images.joyent.com
https://docker.io
[root@d0-50-99-46-c1-59 ~]# imgadm import registry

Creating a Private Docker Registry

Having successfully imported the dataset, a private registry can be provisioned using the following json-manifest:

{
  "alias": "dockerregistry",
  "hostname": "dckrreg1.hb22.cruwe.de",
  "image_uuid": "1236eb25-b04d-c68a-150a-8a0ec3befbc9",
  "nics": [
    {
      "interface": "net0",
      "nic_tag": "admin",
      "gateway": "<your gw>",
      "netmask": "<your netmask>",
      "primary": true,
      "ip": "<a free IP>"
    }
  ],
  "brand": "lx",
  "docker": "true",
  "kernel_version": "3.13.0",
  "cpu_shares": 100,
  "zfs_io_priority": 1000,
  "max_lwps": 2000,
  "max_physical_memory": 1024,
  "max_locked_memory": 1024,
  "max_swap": 16384,
  "cpu_cap": 100,
  "tmpfs": 16384,
  "maintain_resolvers": true,
  "resolvers": [
    "<your resolver>"
  ],
  "quota": 10,
  "internal_metadata": {
    "docker:cmd": "[\"/bin/sh\", \"/docker-registry/entry.sh\"]"
  }
}

I cannot say if the image UUID is deterministically set. It might be best to closely monitor the output of imgadm import <imgname>.

Very possibly and unless you are planning for a large installation, zfs_io_priority, max_lwps, memory, swap and tmpfs sizes can be set considerably smaller.

Observe that in contrast to usual docker operations, we are able to give the docker container an own private IP. This might be a nice feature if you want to give mutually untrusting parties docker containers on the same hypervisor.

The docker container is provisioned using vmadm(1M)vmadm(1M). calling

[root@d0-50-99-46-c1-59 ~]# vmadm create -f <fullpathtojson>

Pulling and Using Images from Private Docker Registry

Image sources such as the newly created private docker registry can be added (docker registries commonly listen on port 5000) and subsequently imported as such:

[root@d0-50-99-46-c1-59 ~]# imgadm sources -a <IP or URL>:5000 -t docker
[root@d0-50-99-46-c1-59 ~]# imgadm import <IP or URL>:5000/<image>

Provisioning a Web Interface

Using curl as an interface to http APIs can be considered inconvenient. Usually, users are spoiled to expect www-interfaces accessible by browser, which Konrad Kleine graciously publishes on githubkwk/docker-registry-frontend. .

{
"alias": "dockerviewer1",
"hostname": "dckrvie1.hb22.cruwe.de",
"image_uuid": "c2600245-08e7-b821-b548-607fedc85a4c",
"nics": [
    {
      "interface": "net0",
      "nic_tag": "admin",
      "gateway": "<your gw>",
      "netmask": "<your netmask>",
      "primary": true,
      "ip": "<another free IP>"
    }
],
"brand": "lx",
"docker": "true",
"kernel_version": "3.13.0",
"cpu_shares": 100,
"zfs_io_priority": 1000,
"max_lwps": 2000,
"max_physical_memory": 1024,
"max_locked_memory": 1024,
"max_swap": 16384,
"cpu_cap": 100,
"tmpfs": 16384,
"maintain_resolvers": true,
"resolvers": [
    "<your resolver>"
],
"quota": 10,
  "internal_metadata": {
    "docker:cmd": "[\"/root/start-apache.sh\"]",
    "docker:tty": true,
    "docker:attach_stdin": true,
    "docker:attach_stdout": true,
    "docker:attach_stderr": true,
    "docker:open_stdin": true,
    "docker:env": "[\"ENV_DOCKER_REGISTRY_HOST=<IP or FQDN of the docker registry you just provisioned>\", \"ENV_DOCKER_REGISTRY_PORT=5000\"]",
    "docker:noipmgmtd": true
  }
}

[root@d0-50-99-46-c1-59 ~]# vmadm create -f <fullpathtojson>

Again, note that resource consumption controls are heavily over-provisioned. ⊕I have not found a solution to have SmartOS pull docker images from a private and possibly secured v2 registry. For internal use, a v1 registry is sufficient. For external use, nothing prevents you to provision a secured v2 registry separately.