Lately I've been getting the desired to switch my main headless development box from archlinux to something else.

The main features I currently care about:

I don't want to update my linux kernel every ... single ... week
I already use containers for most of my development
I like the idea of an unadulterated host OS

Given those features I decided to try openSUSE MicroOS since it ticked all the boxes.

Why container-driven development? #

I want better isolation between the tools I use from my host OS. I know I'm not the only one who installs packages and then abandon them only to review my installed packages a year later and am completely confused why something was installed. Further, many package managers aren't great about tracking dependencies of dependencies, which further compounds confusing about what is essential. Theoretically this should make it easier to remove dependencies for my development environment without breaking it. It also makes it easier for me to experiment with different tools without having to properly clean them up when I'm done using them.

Using a host OS directly means that there's a tendency to accumulate tools and packages over time without great mechanisms to cleanup. Now all I have to do is delete a container and rebuild with the new set of tools. This is definitely tedious, but not more tedious than the very popular nixOS, where you have to edit nix files and rebuild.

Installation #

This was a breeze compared to setting up archlinux. The GUI basically did all the heavy lifting of creating the partitions, setting keyboard, languages, timezone, and networking.

I downloaded the ISO from website
I ran cat xxx.iso > /dev/sda
I booted the ISO on my dev machine
I went through the GUI setup process
Done!

Initial setup #

Update sudoers #

1sudo vim /etc/sudoers
2# uncomment section for `wheel` group

Create `wheel` group #

1groupadd wheel

Create user #

1useradd -p <pass> -g wheel erock

Update hostname #

1sudo hostnamectl set-name arc

Update fstab #

1sudo vim /etc/fstab
2# add NAS and extra SSD in my dev box
3# I need cifs-utils for the NAS

Install minimal packages #

Since I'm working with an immutable OS, this is the most dramatic change to something like arch: I have to reboot every time I install a new package. The idea is that this should be a pretty rare event and if it isn't, your workflow needs to change to adapt to this new worldview.

1sudo transactional-update pkg install distrobox mosh cifs-utils

SSH #

I didn't have my pubkey handy during installation of MicroOS, however, sshd was setup automatically and it supported password logins out-of-the-box.

Once I could SSH into the machine, I copied my dotfiles and private keys onto the box.

1cat ~/.ssh/id_xxx.pub > ~/.ssh/authorized_keys

1scp -R ~/dotfiles arc:/home/erock/dotfiles

Setup dev container #

Here's dev.Dockerfile:

 1FROM archlinux:base-devel
 2
 3RUN pacman -Syy && pacman -S --noconfirm \
 4    aspell \
 5    aspell-en \
 6    bat \
 7    curl \
 8    deno \
 9    direnv \
10    expect \
11    fzf \
12    git \
13    gnupg \
14    go \
15    jq \
16    keychain \
17    man-db \
18    man-pages \
19    ncdu \
20    neovim \
21    npm \
22    nodejs \
23    openssh \
24    pass \
25    pass-otp \
26    ripgrep \
27    rsync \
28    scdoc \
29    tree \
30    tmux \
31    unzip \
32    vim \
33    zsh
34
35# place for sources to compile (e.g. aur)
36RUN mkdir /opt/sources
37WORKDIR /opt/sources
38
39# irc chat
40RUN git clone https://git.sr.ht/~taiite/senpai
41WORKDIR /opt/sources/senpai
42RUN make && make install

And here's how I ran it inside distrobox:

1podman build -t dev -f dev.Dockerfile . # archlinux
2distrobox create -i dev dev
3distrobox enter dev
4
5~/dotfiles/dotfiles.sh # copy dotfiles from dotfile repo -- including keys
6~/dotfiles/setup_fresh.sh # setup neovim plugin manager, tmux plugin manager, oh-my-zsh, chsh -s /usr/sbin/zsh

Issues #

I haven't figured out how to run podman inside a docker container. There is documentation for how to do it but it currently isn't working.

For some reason, $SHELL is still set to /bin/bash which means tmux is using the wrong shell. The docs for distrobox instruct us to just run chsh but oh-my-zsh does that automatically upon initial installation. For now, I manually set the shell inside tmux.

On mutating an "immutable" OS #

MicroOS allows the following directories to be modified:

/etc
/var
/home

This is a pretty large hole where files can be modified directly. When comparing it to something like nixOS, it's clear it's an immutable-lite OS. This is good enough for me, but the natural end feels like a fully immutable OS like nixOS.

Conclusion #

That's it! I now have a fully functional development container that works as-if I was on archlinux. So far, I've been very happy with MicroOS+distrobox, it's a really powerful setup that will keep my host machine relatively stable and unchanged.

Update 2024 #

After spending the better part of 6 months using microOS, I ended up ditching it and going back to arch. Unfortunately, doing everything inside of containers turned out to be a ton of work -- to the surprise of no one. The biggest issue I had, more than anything else, was being able to run docker-within-docker. I eventually got it to work but it left a sour taste in my mouth.

Another issue for me was using podman instead of docker. I should have known that this was going to be too many changes, but they trapped me with the "drop in replacement for docker" slogan so I thought it would be okay. Well, it wasn't okay. Basically every build script that uses docker -- I have a lot of Makefile that run docker build --push -- required modifications to support both docker and podman. Further, podman's buildx equivalent was different enough to be annoying.

I also decided to merge my gaming desktop and my dev machine together and at that point I did not feel confident I would be able to play video games on linux using an immutable OS.

Further, I just felt like I was spending so much time tinkering with the workflow that I gave up. I fully accept that people more knowledgeable with containers could make this work.

Having said that, I came back to this article because I'm once again dreaming of container-driven development. I think this time I might start with a host OS that I'm more familiar with -- like arch -- and then just pretend it's immutable.