Fully Encrypted Headless Debian Stretch Setup
Here’s my field notes from the server I recently set up. It’s a rental server at Hetzner, which comes with a very basic rescue system pre-installed.
Starting from there, I followed the steps outlined here. The differences between the Ubuntu version used there and Debian Stretch kept me busy quite a bit so I’m writing my adopted command chain down here to save you some time.
Partition and Encrypt the disks
Hetzner servers usually come with at least 2 disks which are partioned identically with 128MB for /boot (which stays unencrypted), and the rest for the encrypted system.
On top of that two Raid 1 arrays are formed, md0 for /boot and md1 for everything else.
sgdisk -og /dev/sda
sgdisk -n 1:2048:+256M -t 1:fd00 /dev/sda
sgdisk -n 128:-3M:0 -t 128:ef02 /dev/sda
sgdisk -n 2:0:0 -t 2:fd00 /dev/sda
sgdisk -R /dev/sdb /dev/sda
sgdisk -G /dev/sdb
partprobe
mdadm --create /dev/md0 --metadata=0.9 --level=1 --assume-clean --raid-devices=2 /dev/sd[ab]1
mdadm --create /dev/md1 --metadata=1.2 --level=1 --assume-clean --raid-devices=2 /dev/sd[ab]2
Now, md1 will be encrypted, and on top of that encrypted device we set up our root and other partitions using LVM.
I tend to start with reasonable but small partition size and leave most space in the LVM volume group unassigned so I have room to create / enlarge partitions later on as needed.
cryptsetup -c aes-xts-plain -y -s 512 -h sha512 luksFormat /dev/md1
cryptsetup luksOpen /dev/md1 debian
pvcreate /dev/mapper/debian
vgcreate debian-vg /dev/mapper/debian
lvcreate -L 20GB -n root debian-vg
lvcreate -L 16GB -n swap debian-vg
lvcreate -L 40GB -n home debian-vg
lvcreate -L 100GB -n var debian-vg
mkfs.ext2 -L boot -I 128 /dev/md0
mkfs.ext4 -L root /dev/debian-vg/root
mkfs.ext4 -L home /dev/debian-vg/home
mkfs.ext4 -L var /dev/debian-vg/var
mkswap -L swap /dev/debian-vg/swap
mount /dev/debian-vg/root /mnt
mkdir /mnt/{boot,var,home}
mount /dev/md0 /mnt/boot
mount /dev/debian-vg/var /mnt/var
mount /dev/debian-vg/home /mnt/home
Set Up a Basic Debain Stretch With Debootstrap
The most recent version of debootstrap at the time you are doing this may vary, adjust accordingly.
wget http://ftp.de.debian.org/debian/pool/main/d/debootstrap/debootstrap_1.0.89_all.deb
ar -xf debootstrap_1.0.89_all.deb
tar -xf data.tar.gz -C /
debootstrap --arch=amd64 stretch /mnt http://ftp.de.debian.org/debian
Next, edit or create a couple config files.
# make sure resolv.conf is correct (mine was prefilled with Hetzner DNS servers) or do:
echo "nameserver 8.8.8.8" > /mnt/etc/resolv.conf
echo "test" > /mnt/etc/hostname
echo "debian UUID=$(blkid -s UUID -o value /dev/md1) none luks" > /mnt/etc/crypttab
cat > /mnt/etc/fstab << EOF
proc /proc proc defaults 0 0
UUID=$(blkid -s UUID -o value /dev/md0) /boot ext2 defaults 0 0
UUID=$(blkid -s UUID -o value /dev/mapper/debian--vg-root) / ext4 defaults 0 1
UUID=$(blkid -s UUID -o value /dev/mapper/debian--vg-var) /var ext4 defaults 0 2
UUID=$(blkid -s UUID -o value /dev/mapper/debian--vg-home) /home ext4 defaults 0 2
UUID=$(blkid -s UUID -o value /dev/mapper/debian--vg-swap) none swap defaults 0 0
EOF
echo "deb http://security.debian.org/ stretch/updates main" >> /mnt/etc/apt/sources.list
And change into the fresh Debain system with chroot:
mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -t sysfs /sys /mnt/sys
mount -t proc /proc /mnt/proc
chroot /mnt
Set a password, create some symlinks for the Raid devices and create mtab. I’m not sure if the links are actually necessary.
passwd
mkdir /dev/md
ln -s /dev/md0 /dev/md/0
ln -s /dev/md1 /dev/md/1
cp /proc/mounts /etc/mtab
Upgrade the installed packages and install a couple more:
apt update
apt upgrade -y
DEBIAN_FRONTEND=noninteractive apt install -y \
vim \
linux-base \
linux-image-amd64 linux-headers-amd64 \
grub-pc \
mdadm \
cryptsetup \
lvm2 \
initramfs-tools \
openssh-server \
busybox \
dropbear \
locales \
net-tools
dpkg-reconfigure locales # select en_US.UTF-8
Make sure you can actually log into the new system by putting your SSH public key into /root/.ssh/authorized_keys.
echo "YOUR PUBLIC SSHKEY HERE" > /root/.ssh/authorized_keys
Last but not least, set up networking so your system actually goes online after unlocking. I tried to use the modern way, but for some reason it didn’t work, so I went with the classic /etc/network/interfaces config style. Next hurdle was to find out the (not so) predictable network interface name the kernel assigns to my server’s ethernet adaptor. I got it from the syslog after accessing the system through the rescue image (see below for the steps to access the machine in case you cannot get into it after reboot).
It might also be possible to force the Kernel to use the old eth0 style names using the net.ifnames=0 kernel boot parameter (link to serverfault).
cat > /etc/network/interfaces.d/lo << EOF
auto lo
iface lo inet loopback
EOF
# your device name may be different
cat > /etc/network/interfaces.d/enp0s31f6 << EOF
auto enp0s31f6
iface enp0s31f6 inet static
address YOUR.IPv4.ADDRESS
netmask YOUR.IPv4.NETMASK
gateway YOUR.IPv4.GATEWAY
iface enp0s31f6 inet6 static
address YOUR.IPv6.ADDRESS
netmask YOUR.IPv6.NETMASK
gateway YOUR.IPv6.GATEWAY
EOF
Set Up the Initial Ramdisk for Remote Password Entry
Since the whole system (except /boot) is encrypted, you will have to enter the passphrase at boot time.
This is accomplished by embedding the dropbear SSH server into the initial ramdisk image, which allows you to SSH into the server before it even finished booting to enter the passphrase.
Luckily this is all easy and straight forward to set up nowadays:
sed -i "s/NO_START=1/NO_START=0/" /etc/default/dropbear
echo "DEVICE=eth0" >> /etc/initramfs-tools/initramfs.conf
sed -i "s/^#CRYPTSETUP=$/CRYPTSETUP=y/" /etc/cryptsetup-initramfs/conf-hook
For logging in via SSH, install your SSH key, again. You can use the one you just put into /root/.ssh/authorized_keys, as long as it is an RSA key. Other key types (I tried my ed25519 key) will most certainly not work.
The options before the key limit what can be done when logging in with that key, and the command is what will be executed directly after log in.
Since the sole purpose of this is to unlock the disk, the script which does just that is launched directly after log in.
To prevent dropbear from starting when it isn’t needed (that is, outside the initrd in your ‘real’ system), run systemctl disable dropbear. Thanks to Thomas pointing this out in the comments.
echo "no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command=\"/bin/cryptroot-unlock\" YOUR PUBLIC RSA SSHKEY" > /etc/dropbear-initramfs/authorized_keys
In order to avoid SSH complaining about changed host keys every time you unlock the server or connect to it when unlocked there are two ways: run Dropbear on a different port than you run the ‘normal’ SSH server on the unlocked system on, or use a separate known_hosts file for unlocking.
I chose to run Dropbear on another port and set some other options while I’m at it:
sed -i "s/^#DROPBEAR_OPTIONS=$/DROPBEAR_OPTIONS=\"-p 2323 -s -j -k -I 60\"/" /etc/dropbear-initramfs/config
With all that done, it’s time to update the initrd and install the Grub bootloader. The ip kernel boot parameter is essential for setting up the network in the initial ram disk, otherwise dropbear won’t be reachable. Be sure to replace the placeholders with whatever IPv4 config your server has.
update-initramfs -u -k all
echo "GRUB_CMDLINE_LINUX_DEFAULT=\"ip=IPv4IP::GATEWAY:NETMASK\"" >> /etc/default/grub
grub-install /dev/sda
grub-install /dev/sdb
update-grub
# exit the chroot and reboot
exit
umount /mnt/{boot,var,home}
sync
swapoff -L swap
reboot
At this point, you should be able to connect to the server on the configured port and immediately be asked for your passphrase. Enter it, press enter and the connection is closed, while the server continues booting.
It doesn’t work!
You cannot connect to unlock the system, or you unlock it but cannot connect after that? Don’t panic.
With Hetzner, there is always a way to reboot the machine into the rescue system, which can then be used to fix things.
After you logged into the system, follow these steps to unlock disks and get into the Debian again:
cryptsetup luksOpen /dev/md1 debian
vgchange -a y
mount /dev/debian-vg/root /mnt
mount /dev/md0 /mnt/boot
mount /dev/debian-vg/var /mnt/var
mount /dev/debian-vg/home /mnt/home
mount /dev/debian-vg/srv /mnt/srv
mount /dev/debian-vg/data /mnt/opt/data
mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -t sysfs /sys /mnt/sys
mount -t proc /proc /mnt/proc
chroot /mnt
I needed these quite a few times while fixing my network config. Once you’re ready to give it another go, do the same steps as above, but skip the Grub installation:
update-initramfs -u -k all
update-grub
exit
umount /mnt/{boot,var,home}
sync
swapoff -L swap
reboot
That’s it, have fun with your shiny new server!