Let's Encrypt SSL Certificates With HAProxy and Stable Keys

January 08, 2017 | letsencrypt, haproxy, security, devops, linux, debian

I’ve been a (more or less) happy StartSSL customer for years, but since they are going to lose their status as a trusted CA these days for various reasons, I finally got around to switching to Let’s Encrypt. Here’s what I did.

The Situation

I have HAProxy running in front of several nginx instances on different (virtual) machines. HAProxy handles all the TLS stuff. This is Debian Jessie, so HAProxy is a rather old v1.5.x, but that’s fine with the approach outlined here. No need to install backported / self-compiled newer versions.

I’ve also been using HPKP now for a while, which leads to the following requirements:

the keys used for generating certificates should be rather stable, since each change requires changes to the HPKP headers
new keys have to be introduced to clients by inclusion in the HPKP header a long time (at least 2 months, in my case) before they are actually used

So I had to find a way to use my existing keys (which my sites are ‘pinned’ to) and generate new Let’s Encrypt certificates from them. Further I wanted to ensure that future (automatic) certificate renewals also reuse the same private keys. Since Let’s Encrypt only issues certificates with 3 months lifetime, automatic renewal and keeping the same keys both really are a must have.

While the official Let’s Encrypt client, Certbot, goes great lengths to automate the whole certificate generation and renewal process, it unfortunately does not provide this comfort in combination with pre-generated, constant keys and CSRs.

What it does, on the other hand, is integrate with webservers to automatically install certificates and also configure the server for using them. Way too much to put into a single tool for my taste. Support for HAProxy is still in its early stages and requires building stuff yourself, so this is not really an option for now, anyway.

In this article, I’ll show how to setup certbot on Debian under a separate user account and use it in standalone mode, together with HAProxy, to generate and renew certificates from given keys / CSRs, with scripts suitable for unattended operation with any number of certificates / sites.

I won’t cover basic sysadmin tasks, HAProxy SSL setup or HPKP configuration.

OpenSSL Preparations

You can (and should) do all of this on your local machine, we’ll copy the relevant files to the server later.

Generate private keys (if you haven’t already)

openssl genrsa -out yoursite.key 4096

Sidenote: If you’re using HPKP, you really should generate at least two backup keys and include their hashes in your headers as well, to avoid bricking your domain through loss of a private key. Print out your backup keys and/or store them (encrypted) in various different locations. You do not want to leave them lying around on your server because that would render them useless exactly when you need them - if your server got hacked any keys that were stored on it are useless.

I use gpg to encrypt a tar archive which I distribute to various places. It’s also no mistake to make an offsite backup of your primary key of course.

The HPKP hash of an RSA private key to put into the HPKP header can be generated like that:

openssl rsa -in yoursite.key -outform der -pubout| openssl dgst -sha256 -binary | openssl enc -base64

Create an openSSL config file for each certificate

This can be a big timesaver later on if you’re using the same certificate for several (sub)domains and want to regenerate the CSR for this certificate. Use the default file from /etc/ssl/openssl.cnf as a template and copy it to yoursite.cnf for example.

In the [ req ] section, uncomment or add the line

req_extensions = v3_req

You should also change the default_bits value to 4096 here.

Then, in the [ v3_req ] section, add this:

subjectAltName = @alt_names

To save further typing later on you should also change or add the commonName_default and emailAddress_default values in the [ req_distinguished_name ] section:

commonName_default = yoursite.com
emailAddress_default = john.doe@gmail.com

And finally, at the end of the file, add a new section holding your domain names:

[ alt_names ]

DNS.1 = yoursite.com
DNS.2 = www.yoursite.com
DNS.3 = yoursite.net
DNS.4 = www.yoursite.net

Generate a CSR

With the config file you just created this is easy:

openssl req -new -key yoursite.key -out yoursite.csr -config yoursite.cnf

This will ask several questions about country, location and organization. Simply accept the defaults, these values are ignored by Let’s Encrypt anyway. Common name and email address should hold your configured defaults and can thus be accepted by pressing enter as well. Do not enter a passphrase, but just press enter two more times and you’re done.

Whenever you have to regenerate the CSR for this certificate due to a change in the domains it is for, or due to a changed private key, just edit the config file if necessary and re-run the above command.

On the Server

Set Up Certbot and Add a Separate User Account for Running It

Install certbot

The Certbot Site has a nice interactive installation howto. For Debian Jessie the suggestion is to use the certbot package from the jessie-backports repository, which I did.

Create the certbot user

Using a separate account and not having this run as root is just good practice and helps to keep things nicely contained in a dedicated directory. I’ll leave the user creation up to you but in the end you should have a user certbot with /bin/bash as shell and a home of /home/certbot. Note that this user doesn’t need to have a password set at all, for the few times you have to interactively use it (only during setup / while trying things out) you can just use su - certbot as root.

In /home/certbot, create a few directories:

certs which will hold one directory per certificate / site
logs for certbot logs
config for any files certbot creates at runtime

Configure certbot

To simplify certbot usage, it’s a good idea to create a config file in /home/certbot/.config/letsencrypt/cli.ini which holds most of certbots config:

# Use a 4096 bit RSA key instead of 2048
rsa-key-size = 4096

# update to register with the specified e-mail address
email = john.doe@gmail.com

# use a text interface instead of ncurses
text = True
non-interactive = True
agree-tos = True

# use the standalone authenticator
authenticator = standalone

preferred-challenges = http-01
# this is the same port as in the haproxy letsencrypt backend:
http-01-port = 54321

work-dir = /home/certbot
config-dir = /home/certbot/config
logs-dir = /home/certbot/logs

With this, you should be able to successfully run certbot register as the certbot user. This should create some files below /home/certbot/config.

Configure HAProxy

I assume you already have SSL/TLS set up for your sites so I’ll show just the relevant parts here for making the Let’s Encrypt domain validation work.

frontend http
  # more config here...

  # direct all letsencrypt requests to a special backend
  acl letsencrypt-acl path_beg /.well-known/acme-challenge/
  use_backend letsencrypt-backend if letsencrypt-acl

  # more config here...

# define the backend referenced above and point it to the certbot validation server
backend letsencrypt-backend
server letsencrypt 127.0.0.1:54321

This defines an ACL to recognize LetsEncrypt domain validation requests, and points any such requests to a dedicated backend.

The certbot validation server which will be spawned automatically by certbot during the certificate creation / renewal process when it is used in standalone mode listens on port 54321, and that’s exactly where we point our letsencrypt backend to.

Install private key and CSR

Create a /home/certbot/certs/yoursite directory and place yoursite.key and yoursite.csr inside. Ensure proper permissions and file ownerships:

the key is to be kept private must be owned by root and also only be readable by root (chmod 0400 yoursite.key)
the csr can be world readable, it’s not confidential. At the very least it has to be readable by the certbot user.

Run certbot!

Yay! By now we should be all set and ready to try out the validation process. For testing it is advisable to use certbot with the staging argument to prevent using up your daily quota of Let’s Encrypt certificates per domain.

Run this as the certbot user:

certbot certonly --staging --csr yoursite.csr --cert-path yoursite.crt --chain-path chain.pem --fullchain-path fullchain.pem

You should end up with a certificate and the two chain files as specified in the command line options. These are not really usable as they are only signed by the Let’s Encrypt staging environment, but we now know that our HAProxy setup and certbot config work. The fullchain file is what we’ll use later on with HAProxy, since it includes both the Let’s Encrypt intermediate certificate and the certificate for yoursite.com.

Automate All the Things!

With certonly mode and the --csr option it doesn’t make a difference to certbot wether you are creating a new certificate or renewing an existing one. Therefore I also only have one shell script for both tasks. You can have a look at it below. There’s also a Gist

#!/bin/bash

reload_required=false

# attempt to renew when less then 30 days remaining
exp_limit=30;

# returns 0 if no renewal needed, 1 otherwise
check_cert_still_valid () {
  local cert_file=$1

  local exp=$(date -d "`openssl x509 -in $cert_file -text -noout|grep "Not After"|cut -c 25-`" +%s)
  local datenow=$(date -d "now" +%s)
  local days_exp=$(echo \( $exp - $datenow \) / 86400 |bc)

  echo "Checking expiration date for $domain..."

  if [ "$days_exp" -gt "$exp_limit" ] ; then
    echo "The certificate is up to date, no need for renewal ($days_exp days left)."
    return 0
  else
    echo "The certificate $cert_file is about to expire soon. Starting renewal..."
    return 1
  fi
}

get_certificate() {
  local domain=$1
  local cert_file=$2
  local key_file="/home/certbot/certs/${domain}/${domain}.key"
  local csr_file="/home/certbot/certs/${domain}/${domain}.csr"
  local chain_file="/home/certbot/certs/$domain/chain.pem"
  local fullchain_file="/home/certbot/certs/$domain/fullchain.pem"

  # Certbot refuses to overwrite existing files, so remove anything that
  # might get in the way.
  # The certificate used by haproxy is kept separately, so no harm is done by
  # deleting these files:
  rm -f $cert_file $chain_file $fullchain_file
  certbot certonly --csr $csr_file --cert-path $cert_file --chain-path $chain_file --fullchain-path $fullchain_file

  if [ $? -eq 0 ]; then
    echo "Creating $combined_file with latest certs..."
    sudo /usr/local/sbin/le-haproxy-bundle $domain
    echo "Renewal process finished for domain $domain"
    reload_required=true
  else
    echo "certbot failed, not replacing installed certificate for ${domain}"
  fi
}

process_certificate() {
  local domain=$1
  local cert_file="/home/certbot/certs/$domain/${domain}.crt"

  if [ -f $cert_file ]; then
    check_cert_still_valid $cert_file
    if [ ! $? -eq 0 ]; then
      get_certificate $domain $cert_file
    fi
  else
    echo "No certificate for domain $domain exists yet. Creating one..."
    get_certificate $domain $cert_file
  fi
}

domain=$1

# loop over all domains unless a domain is given
if [ "${domain}" = "" ]; then
  for i in $( ls /home/certbot/certs ); do
    process_certificate $i
  done
else
  process_certificate $domain
fi

if [ "$reload_required" = true ]; then
  echo "Reloading haproxy"
  sudo /usr/sbin/service haproxy reload
fi

If called without arguments, this script will loop over the directories in /home/certbot/certs and either create a new certificate or renew an existing one if it’s about to expire in the next 30 days. Since the script only acts on certificates that will expire in the next 30 days, it can be called as often as you want. Once or twice daily through cron should be enough, however. For manual operation you can also call it with one directory name as an argument, i.e. /usr/local/sbin/le-renew-haproxy yoursite.

Before this will work there are two more things to do, however.

Make the certificate usable for HAProxy

The task of taking the generated certificate and combining it with the custom dhparam and private key file to the certificate bundle to be used by HAProxy is carried out by another script, which is called through sudo by le-renew-haproxy:

#!/bin/bash

domain=$1
fullchain_file="/home/certbot/certs/$domain/fullchain.pem"
key="/home/certbot/certs/${domain}/${domain}.key"
combined_file="/etc/haproxy/ssl/${domain}.pem"
dhparam="/etc/haproxy/dhparam"

if [ -f $fullchain_file -a -f $key -a -f $dhparam ]; then
  if [ -f $combined_file ]; then
    cp $combined_file "${combined_file}.old"
  fi
  cat $fullchain_file $key $dhparam > $combined_file
  chmod 0400 $combined_file
fi

Setup sudo permissions for the certbot user

For things to work out non-interactively, you have to give the certbot user permission to use passwordless sudo for these two commands:

/usr/local/sbin/le-haproxy-bundle yoursite to build and install the certificate bundle to /etc/haproxy/ssl
/usr/sbin/service haproxy reload to enable reloading of haproxy

Disclaimer

There might be typos in the scripts / I might have entirely forgotten something. This article is written after the fact, and my server is provisioned with chef so I had to reconstruct / amend / generalize things. Please let me know of any oversights in the comments.

Be careful with HPKP. Use it in report-only mode and/or with short lifetimes until you’re sure you got it right.

Prior Art and Further Reading

The Certbot Documentation

The Let’s Encrypt Website

The idea to use standalone mode through a HAProxy backend and the initial version of my renewal shell script stem from this Gist.

Scott Helmes introductory article on Let’s Encrypt suggests to use a customized OpenSSL config file for each certificate, which eases CSR generation a lot.

Check out the haproxy-ocsp-stapling-updater for setting up OCSP stapling with HAProxy.

Another howto on Using haproxy with Let’s Encrypt

My Gist with all config files and scripts

And finally

Consider donating to the EFF to support their awesome work.