There are many good reasons why you may not know AFS, a old network file system still in use at some Universities. There are also many good reasons on why one should stay as far as possible from it, but here we are: supporting AFS in Kubernetes and Docker.

In my case, the objective was to provide access to AFS to users in a JupyterHub environment, setup at KTH.

Note: this is an experimental setup, not in production, and has not been peer-reviewed nor has received a security audit. Take it with a grain of salt!

The beginning: getting AFS in the kernel

In Linux, AFS access is commonly supplied with a kernel module (added via DKMS), which provides the underlying mechanisms. This module communicates with the user-space utilities, which allows users to obtain Kerberos tickets, and use these to authenticate across the network.

Typically, one would obtain a token with a kinit command (e.g. in the case of KTH):

kinit -f user@KTH.SE

This would create a token, and register it with the user UID, as could be seen with klist.

Then one would issue aklog, and magically the files in /afs/kth.se/home/u/user would become readable.

On a normal shared system, where only few users have root access, this is relatively safe and does has been used in production in many systems.

AFS and containerization

Given that standard containers (in the sense of Docker and Kubernetes) share the system’s kernel and modules, AFS needs to be installed at system level. One can then install AFS utilities in the container (e.g. with openafs-krb5 openafs-client on Ubuntu) and use these to manipulate Kerberos tickets.

To access the files, however, one needs to be able to read/write the AFS file cache, typically mounted in /afs/ and populated by the kernel module above.

Easy: bind mount (-v /afs:/afs for Docker) and we can call the day.

Ok, not so easy

There is a trivia: containers, typically, use a specific UID in the system - which typically is the one of the root user or the user that launched them.

When external users have access to the containers (e.g. if there is any way to ssh in the container, or to run arbitrary commands), processes would live in the same user id space, by default.

This is not a big drama, but it is in our case: we have all users sharing a singe UID, and registering Kerberos tickets under the same UID space!

In other words, everyone would be able to access each-other tickets and files!

A better solution:

Luckily, pagsh exists, and solve (some) of our problems. From Ubuntu:

The pagsh command creates a new command shell (owned by the issuer of the command) and associates a new process authentication group (PAG) with the shell and the user. A PAG is a number guaranteed to identify the issuer of commands in the new shell uniquely to the local Cache Manager. The PAG is used, instead of the issuer’s UNIX UID, to identify the issuer in the credential structure that the Cache Manager creates to track each user.

So, one could run pagsh -c bash, and perform all commands inside this shell, isolated from the other users.

But users are prone to do errors, and you cannot expect all users to never forget to run their commands in pagsh.

If only we had a way to isolate all user commands…

tini enters the chat

On JupyterHub, tini is used to execute the different processes in each user pod. tini is a init-like service, which provides basic functionalities to improve the life of containers.

One key characteristic of tini is that all processes are child of the tini process. In other words, by wrapping tini in pagsh we would have solved most of our problems.

But pagsh is a simple boy

pagsh has a small limitation: the -c argument doesn’t like commands with space.

One could get around this in multiple ways, or could use RFC 1925, point 6a:

(6a) (corollary). It is always possible to add another level of indirection.

The “ugly but working” solution

We will replace tini with a wrapper script tini-afs. This would: - get any argument - wrap it in a bash script - execute it as a shell in pagsh

This boils down to:

!/bin/bash
# (C) 2024 Massimo Girondi massimo@girondi.net - GNU GPL v3

ARGS=$@
echo "ARGS are $@"

cat <<EOF > /tmp/start-tini.sh
#!/bin/bash
tini $@
EOF

chmod +x /tmp/start-tini.sh
pagsh -c /tmp/start-tini.sh

This file needs to be copied to the container during building, and called instead of tini.

In the case of JupyterHub containers:

RUN apt update && \
    apt install -y openafs-client heimdal-clients openafs-krb5 && \
    apt clean -y && rm -rf /var/cache/apt
COPY assets/tini-afs /usr/bin
RUN chmod +x /usr/bin/tini-afs
ENTRYPOINT ["tini-afs", "-g", "--", "start.sh"]

And that should be it. Enjoy!