Migrating to FreeBSD - Part 5
Migrating to FreeBSD from Solaris 11: Part 5 - Jails
During this transition to FreeBSD for my storage host, some ESXi virtual machines were converted to bhyve and others are being converted to jails.
This part of the series will conceptually cover jails, how they are useful and what they provide differently than a virtualized guest. This article will also briefly touch on the origins, some security implications, what silent observation is, some of the neat things ZFS add, and also some example use cases.
First to give some background on what a jail is, why they were created, and how they are useful today. Jails stemmed from the idea of the chroot. For those not familiar, chroot changes the perspective of the shell to reorient itself to a new root. An example would be chrooting to make some changes to your borked OS installation from a live CD.
root@livecd:~ # mount /dev/da1s2 /mnt/borkedOSroot
root@livecd:~ # chroot /mnt/borkedOSroot
root@livecd:/ # pwd /
This would allow changes to be made to files as if booted into the OS as normal. This could be useful if the kernel needed to be rebuilt or package needed to be installed to repair the OS. With a chroot, only the shell instance and it’s child process have the change in perspective which is strictly limited from a directory perspective, nothing more.
Jails improve upon this idea of the chroot by extending the perspective change to running processes as well. A jail is a chroot of a set of processes relative to a directory. A jail can be anything from a single process to a nearly* full blown instance of a FreeBSD. This is different from a virtualized machine, in that the jailed processes run directly on the host. The host can see those processes but from the perspective of a jail, there is no host*. Running top from inside a jail will only reveal the processes started for that jail, running top from the host will reveal everything including what is running inside jails.
This can lead to some interesting observability of jails. For example, a web server running in a jail can be observed, this include the jail’s files and processes from outside the jail. This can be particularly useful if the web server is compromised and you are investigating the attack vector especially if the attack is actively happening. This one-way-glass observation is without the processes in the jail being aware you are snooping around.
From inside a jail there are telling signs it is a jailed environment. To briefly touch on a couple, there are certain things that are not possible from within a jail (for security reasons) without first enabling it for that jail. For example, access to raw sockets is not enabled by default for new jails, what this means is you can not ping another device from inside the jail. Also uname -a and freebsd-version can have some conflicting output, this is due to the ability to run a different binary releases than what is running on the host. Only a version equal-to or less-than the currently running version is possible. For example, a 9.3-RELEASE can be ran as a jail for legacy reasons on a 11-CURRENT host.
Now that there is a better understanding of what a jail is, and a few things it can do, why else would a person want to use them? Jails have a great quantity of benefits, which not everyone could possibly agree on. To me, the most appealing features of jails are: the ability to access files from the host without using a network protocol, fluid resource usage, and the process separation.
By fluid resource usage, I mean the number of CPUs, amount of RAM, nor disk space need to be defined when creating a jail. The jail uses what it needs and IF desired resource limits can be enforced. Those resource limits are just that, limits, not to be confused with an allocation. This is different from a virtualized guest, where as those elements do need to be defined and are consumed (away from the host) upon starting/creating the guest.
The other great benefit to jails are nullfs mounts into a jail. nullfs are about as old (circa 4.4BSD) as jails and do enhance their use. A nullfs mount allows for a directory to be mounted in another location. “That sounds awfully like a symlink” you say? nullfs mounts are different in that processes treat them differently than symlinks, from the process’ perspective a nullfs directory is real; as opposed to a symlink where the process knows it is a pointer. A symlink can not reference a location outside of a jail, a nullfs mounts can. A nullfs mount can also be defined as read only which can differ from the permissions of the original directory. This does potentially open a vector for privilege escalation where a privileged user inside a jail could potentially modify files on the host for a less-privileged user on the host. That said, using the read-only flag certainly helps mitigate this.
“Okay, so what would be a practical application of a nullfs mount in the context of a jail?” Let’s say you have a web server which logs are generated and for some reason upper management decide they want you to analyze those logs on that host with some new application stack that they read about the night before. So what do you do, after saying “No” and they insist? You create a jail, inside the jail you install that questionable code, then you nullfs mount the log file directory as read-only to /jail/id-001/root/mnt/HostLogFiles/. The jail starts and gets chrooted to /jail/id-001/root, the questionable code is allowed to analyze those log files in a process separated space. You may even resource limit the jail to only use x cores and x RAM in the event the code fork-bombs itself or something asinine like that. In this example, the project goals were met without creating as big of risk to the host as if the questionable stack were allowed to run directly.
The addition of ZFS to FreeBSD was undoubtedly a pretty awesome thing. For jails it greatly enhanced them. All the cool things you can do with ZFS, such as snapshots and clones can be utilized here too. Let’s say you want to upgrade to a newer version of an application but historically the software is released with bugs which brings down the whole system. So what you do is take a snapshot, do the upgrade and see if it works. Better yet, if this is a production machine where down time is unacceptable then: snapshot the production jail, create a clone, then start the clone, run the update and see if it works, if it does work, promote the clone as the production machine, shutdown the old system.
To revisit the webserver use-case again: say you have a production webserver that gets hacked more frequently than never. You create a jail, you setup the webserver exactly as desired, you snapshot it, then you create two clones. One clone serves the content to public, the other is used for internal development. If either the public one is hacked or your developers bork the their copy, you can simply recreate a clone from the production baseline. This makes reverting mistakes or malicious acts instant and very low cost. This is in contrast to running three separate machines in addition to their associated maintenance. While this is possible with virtualized guests on ZFS, there is more overhead. Virtualized guests also leaves resource management another touching point needed from changing a server from production to development purposes. Again, a jail is only going to consume what it needs or up to it’s set limit. Changing a jail for the development team likely only involves changing the IP from the baseline. For a virtualized guest, you might also have to dial down the allocated resources from something high to something more modest for the dev team.
Something I have neglected to explicitly state is jails can only* run FreeBSD. This might seem like a big deal but most of the services I ran on other OSes were converted to virtualized FreeBSD guests and then most recently to jails. I have been very pleased with the results. I even converted a Plex Media Server guest which was easy to install as pkg install plexmediaserver. That said, FreeBSD has become my preferred OS across the board and this is coming from a formerly heavy OSX and Linux user just a few years back. I think it was Allan Jude from BSD Now who said something to the effect ~”FreeBSD is to Linux users, as what Linux is to your grandmother.”
Jails are yet another set of tools that FreeBSD provides which are well developed and incredibly useful. I should have converted to FreeBSD years ago. There are some cases where a jail is not appropriate, mostly relating to tighter security requirements (a root CA for example). However, most of the time if a virtualized guest is going to be running FreeBSD, a jail should be a consideration. Furthermore, even if a current virtualized guest is running a different OS, converting to a FreeBSD jail might be worth the advantages gained with the tight ZFS integration and resource fluidity.