Jens Krämer

Fully Encrypted Headless Debian Stretch Setup

 |  sysadmin, debian, linux, devops, encryption  |  3 comments

Here’s my field notes from the server I recently set up. It’s a rental server at Hetzner, which comes with a very basic rescue system pre-installed.

Starting from there, I followed the steps outlined here. The differences between the Ubuntu version used there and Debian Stretch kept me busy quite a bit so I’m writing my adopted command chain down here to save you some time.

Partition and Encrypt the disks

Hetzner servers usually come with at least 2 disks which are partioned identically with 128MB for /boot (which stays unencrypted), and the rest for the encrypted system.

On top of that two Raid 1 arrays are formed, md0 for /boot and md1 for everything else.

sgdisk -og /dev/sda
sgdisk -n 1:2048:+256M -t 1:fd00 /dev/sda
sgdisk -n 128:-3M:0 -t 128:ef02 /dev/sda
sgdisk -n 2:0:0 -t 2:fd00 /dev/sda
sgdisk -R /dev/sdb /dev/sda
sgdisk -G /dev/sdb
partprobe

mdadm --create /dev/md0 --metadata=0.9 --level=1 --assume-clean --raid-devices=2 /dev/sd[ab]1
mdadm --create /dev/md1 --metadata=1.2 --level=1 --assume-clean --raid-devices=2 /dev/sd[ab]2

Now, md1 will be encrypted, and on top of that encrypted device we set up our root and other partitions using LVM. I tend to start with reasonable but small partition size and leave most space in the LVM volume group unassigned so I have room to create / enlarge partitions later on as needed.

cryptsetup -c aes-xts-plain -y -s 512 -h sha512 luksFormat /dev/md1

cryptsetup luksOpen /dev/md1 debian
pvcreate /dev/mapper/debian
vgcreate debian-vg /dev/mapper/debian

lvcreate -L 20GB -n root debian-vg
lvcreate -L 16GB -n swap debian-vg
lvcreate -L 40GB -n home debian-vg
lvcreate -L 100GB -n var debian-vg

mkfs.ext2 -L boot -I 128 /dev/md0
mkfs.ext4 -L root /dev/debian-vg/root
mkfs.ext4 -L home /dev/debian-vg/home
mkfs.ext4 -L var /dev/debian-vg/var

mkswap -L swap /dev/debian-vg/swap

mount /dev/debian-vg/root /mnt
mkdir /mnt/{boot,var,home}
mount /dev/md0 /mnt/boot
mount /dev/debian-vg/var /mnt/var
mount /dev/debian-vg/home /mnt/home

Set Up a Basic Debain Stretch With Debootstrap

The most recent version of debootstrap at the time you are doing this may vary, adjust accordingly.

wget http://ftp.de.debian.org/debian/pool/main/d/debootstrap/debootstrap_1.0.89_all.deb
ar -xf debootstrap_1.0.89_all.deb
tar -xf data.tar.gz -C /
debootstrap --arch=amd64  stretch /mnt http://ftp.de.debian.org/debian

Next, edit or create a couple config files.

# make sure resolv.conf is correct (mine was prefilled with Hetzner DNS servers) or do:
echo "nameserver 8.8.8.8" > /mnt/etc/resolv.conf

echo "test" > /mnt/etc/hostname
echo "debian UUID=$(blkid -s UUID -o value /dev/md1) none luks" > /mnt/etc/crypttab

cat > /mnt/etc/fstab << EOF
proc                        /proc   proc    defaults    0 0
UUID=$(blkid -s UUID -o value /dev/md0)                    /boot   ext2    defaults   0 0
UUID=$(blkid -s UUID -o value /dev/mapper/debian--vg-root) /       ext4    defaults   0 1
UUID=$(blkid -s UUID -o value /dev/mapper/debian--vg-var)  /var    ext4    defaults   0 2
UUID=$(blkid -s UUID -o value /dev/mapper/debian--vg-home) /home   ext4    defaults   0 2
UUID=$(blkid -s UUID -o value /dev/mapper/debian--vg-swap)  none    swap defaults     0 0
EOF

echo "deb http://security.debian.org/ stretch/updates main" >> /mnt/etc/apt/sources.list

And change into the fresh Debain system with chroot:

mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -t sysfs /sys /mnt/sys
mount -t proc /proc /mnt/proc
chroot /mnt

Set a password, create some symlinks for the Raid devices and create mtab. I’m not sure if the links are actually necessary.

passwd

mkdir /dev/md
ln -s /dev/md0 /dev/md/0
ln -s /dev/md1 /dev/md/1

cp /proc/mounts /etc/mtab

Upgrade the installed packages and install a couple more:

apt update
apt upgrade -y
DEBIAN_FRONTEND=noninteractive apt install -y \
  vim \
  linux-base \
  linux-image-amd64 linux-headers-amd64 \
  grub-pc \
  mdadm \
  cryptsetup \
  lvm2 \
  initramfs-tools \
  openssh-server \
  busybox \
  dropbear \
  locales \
  net-tools

dpkg-reconfigure locales # select en_US.UTF-8

Make sure you can actually log into the new system by putting your SSH public key into /root/.ssh/authorized_keys.

echo "YOUR PUBLIC SSHKEY HERE" > /root/.ssh/authorized_keys

Last but not least, set up networking so your system actually goes online after unlocking. I tried to use the modern way, but for some reason it didn’t work, so I went with the classic /etc/network/interfaces config style. Next hurdle was to find out the (not so) predictable network interface name the kernel assigns to my server’s ethernet adaptor. I got it from the syslog after accessing the system through the rescue image (see below for the steps to access the machine in case you cannot get into it after reboot).

It might also be possible to force the Kernel to use the old eth0 style names using the net.ifnames=0 kernel boot parameter (link to serverfault).

cat > /etc/network/interfaces.d/lo << EOF
auto lo
iface lo inet loopback
EOF

# your device name may be different
cat > /etc/network/interfaces.d/enp0s31f6 << EOF
auto enp0s31f6
iface enp0s31f6 inet static
  address YOUR.IPv4.ADDRESS
  netmask YOUR.IPv4.NETMASK
  gateway YOUR.IPv4.GATEWAY

iface enp0s31f6 inet6 static
  address YOUR.IPv6.ADDRESS
  netmask YOUR.IPv6.NETMASK
  gateway YOUR.IPv6.GATEWAY
EOF

Set Up the Initial Ramdisk for Remote Password Entry

Since the whole system (except /boot) is encrypted, you will have to enter the passphrase at boot time. This is accomplished by embedding the dropbear SSH server into the initial ramdisk image, which allows you to SSH into the server before it even finished booting to enter the passphrase. Luckily this is all easy and straight forward to set up nowadays:

sed -i "s/NO_START=1/NO_START=0/" /etc/default/dropbear
echo "DEVICE=eth0" >> /etc/initramfs-tools/initramfs.conf
sed -i "s/^#CRYPTSETUP=$/CRYPTSETUP=y/" /etc/cryptsetup-initramfs/conf-hook

For logging in via SSH, install your SSH key, again. You can use the one you just put into /root/.ssh/authorized_keys, as long as it is an RSA key. Other key types (I tried my ed25519 key) will most certainly not work. The options before the key limit what can be done when logging in with that key, and the command is what will be executed directly after log in. Since the sole purpose of this is to unlock the disk, the script which does just that is launched directly after log in.

echo "no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command=\"/bin/cryptroot-unlock\" YOUR PUBLIC RSA SSHKEY" > /etc/dropbear-initramfs/authorized_keys

In order to avoid SSH complaining about changed host keys every time you unlock the server or connect to it when unlocked there are two ways: run Dropbear on a different port than you run the ‘normal’ SSH server on the unlocked system on, or use a separate known_hosts file for unlocking. I chose to run Dropbear on another port and set some other options while I’m at it:

sed -i "s/^#DROPBEAR_OPTIONS=$/DROPBEAR_OPTIONS=\"-p 2323 -s -j -k -I 60\"/" /etc/dropbear-initramfs/config

With all that done, it’s time to update the initrd and install the Grub bootloader. The ip kernel boot parameter is essential for setting up the network in the initial ram disk, otherwise dropbear won’t be reachable. Be sure to replace the placeholders with whatever IPv4 config your server has.

update-initramfs -u -k all

echo "GRUB_CMDLINE_LINUX_DEFAULT=\"ip=IPv4IP::GATEWAY:NETMASK\"" >> /etc/default/grub
grub-install /dev/sda
grub-install /dev/sdb
update-grub

# exit the chroot and reboot
exit
umount /mnt/{boot,var,home}
sync
swapoff -L swap

reboot

At this point, you should be able to connect to the server on the configured port and immediately be asked for your passphrase. Enter it, press enter and the connection is closed, while the server continues booting.

It doesn’t work!

You cannot connect to unlock the system, or you unlock it but cannot connect after that? Don’t panic.

With Hetzner, there is always a way to reboot the machine into the rescue system, which can then be used to fix things.

After you logged into the system, follow these steps to unlock disks and get into the Debian again:

cryptsetup luksOpen /dev/md1 debian
vgchange -a y

mount /dev/debian-vg/root /mnt
mount /dev/md0 /mnt/boot
mount /dev/debian-vg/var /mnt/var
mount /dev/debian-vg/home /mnt/home
mount /dev/debian-vg/srv /mnt/srv
mount /dev/debian-vg/data /mnt/opt/data
mount /dev/debian-vg/btc /mnt/opt/btc

mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -t sysfs /sys /mnt/sys
mount -t proc /proc /mnt/proc

chroot /mnt

I needed these quite a few times while fixing my network config. Once you’re ready to give it another go, do the same steps as above, but skip the Grub installation:

update-initramfs -u -k all
update-grub

exit
umount /mnt/{boot,var,home}
sync
swapoff -L swap

reboot

That’s it, have fun with your shiny new server!

Comments

Michael

Thanks for the good and up-to-date howto. I try to install a hetzner server too but I can´t get working initramfs with network means server is not reachable while dropbear is running. after unlock server is reachable. Is there any caveat? And: Where could I get /bin/cryptroot-unlock? on my debian stretch server I didn´t found it.

Jens

Check the ip= line in /etc/default/grub - the ip address, gateway and netmask given there are what is used to set up the initial networking.

The cryptroot-unlock script is located in /usr/share/cryptsetup/initramfs/bin/cryptroot-unlock and is installed in the initramfs through the hook /usr/share/initramfs-tools/hooks/cryptroot-unlock.

Michael

I tried many different configurations and at the end I put the ip= line in /etc/default/grub and in /etc/initramfs-tools/initramfs.conf.

And thanks for the explanation of cryptroot-unlock. Now I understood.

You can use Markdown here.

For the sake of spam checking any data you submit, including your IP address, will be transferred to the US based Akismet web service (akismet.com). If that's not acceptable for you, you can also reach me by other means.