I’ve been a (more or less) happy StartSSL customer for years, but since they are going to lose their status as a trusted CA these days for various reasons, I finally got around to switching to Let’s Encrypt. Here’s what I did.
I have HAProxy running in front of several nginx instances on different (virtual) machines. HAProxy handles all the TLS stuff. This is Debian Jessie, so HAProxy is a rather old v1.5.x, but that’s fine with the approach outlined here. No need to install backported / self-compiled newer versions.
I’ve also been using HPKP now for a while, which leads to the following requirements:
- the keys used for generating certificates should be rather stable, since each change requires changes to the HPKP headers
- new keys have to be introduced to clients by inclusion in the HPKP header a long time (at least 2 months, in my case) before they are actually used
So I had to find a way to use my existing keys (which my sites are ‘pinned’ to) and generate new Let’s Encrypt certificates from them. Further I wanted to ensure that future (automatic) certificate renewals also reuse the same private keys. Since Let’s Encrypt only issues certificates with 3 months lifetime, automatic renewal and keeping the same keys both really are a must have.
While the official Let’s Encrypt client, Certbot, goes great lengths to automate the whole certificate generation and renewal process, it unfortunately does not provide this comfort in combination with pre-generated, constant keys and CSRs.
What it does, on the other hand, is integrate with webservers to automatically install certificates and also configure the server for using them. Way too much to put into a single tool for my taste. Support for HAProxy is still in its early stages and requires building stuff yourself, so this is not really an option for now, anyway.
In this article, I’ll show how to setup certbot on Debian under a separate user account and use it in standalone mode, together with HAProxy, to generate and renew certificates from given keys / CSRs, with scripts suitable for unattended operation with any number of certificates / sites.
I won’t cover basic sysadmin tasks, HAProxy SSL setup or HPKP configuration.
You can (and should) do all of this on your local machine, we’ll copy the relevant files to the server later.
Generate private keys (if you haven’t already)
Sidenote: If you’re using HPKP, you really should generate at least two backup keys and include their hashes in your headers as well, to avoid bricking your domain through loss of a private key. Print out your backup keys and/or store them (encrypted) in various different locations. You do not want to leave them lying around on your server because that would render them useless exactly when you need them - if your server got hacked any keys that were stored on it are useless.
I use gpg to encrypt a tar archive which I distribute to various places. It’s also no mistake to make an offsite backup of your primary key of course.
The HPKP hash of an RSA private key to put into the HPKP header can be generated like that:
Create an openSSL config file for each certificate
This can be a big timesaver later on if you’re using the same certificate for several (sub)domains and want to regenerate the CSR for this certificate.
Use the default file from
/etc/ssl/openssl.cnf as a template and copy it to
yoursite.cnf for example.
[ req ] section, uncomment or add the line
You should also change the
default_bits value to 4096 here.
Then, in the
[ v3_req ] section, add this:
To save further typing later on you should also change or add the
emailAddress_default values in the
[ req_distinguished_name ] section:
And finally, at the end of the file, add a new section holding your domain names:
Generate a CSR
With the config file you just created this is easy:
This will ask several questions about country, location and organization. Simply accept the defaults, these values are ignored by Let’s Encrypt anyway. Common name and email address should hold your configured defaults and can thus be accepted by pressing enter as well. Do not enter a passphrase, but just press enter two more times and you’re done.
Whenever you have to regenerate the CSR for this certificate due to a change in the domains it is for, or due to a changed private key, just edit the config file if necessary and re-run the above command.
On the Server
Set Up Certbot and Add a Separate User Account for Running It
The Certbot Site has a nice interactive installation howto.
For Debian Jessie the suggestion is to use the
certbot package from the
jessie-backports repository, which I did.
Create the certbot user
Using a separate account and not having this run as root is just good practice and helps to keep things nicely contained in a dedicated directory.
I’ll leave the user creation up to you but in the end you should have a user
/bin/bash as shell and a home of
Note that this user doesn’t need to have a password set at all, for the few times you have to interactively use it (only during setup / while trying things out) you can just use
su - certbot as root.
/home/certbot, create a few directories:
certswhich will hold one directory per certificate / site
logsfor certbot logs
configfor any files certbot creates at runtime
To simplify certbot usage, it’s a good idea to create a config file in
/home/certbot/.config/letsencrypt/cli.ini which holds most of certbots config:
With this, you should be able to successfully run
certbot register as the
certbot user. This should create some files below
I assume you already have SSL/TLS set up for your sites so I’ll show just the relevant parts here for making the Let’s Encrypt domain validation work.
The certbot validation server on port 54321 where the letsencrypt-backend points to will be spawned automatically by
certbot during the certificate creation / renewal process when it is used in standalone mode.
Install private key and CSR
/home/certbot/certs/yoursite directory and place
yoursite.csr inside. Ensure proper permissions and file ownerships:
- the key is to be kept private must be owned by root and also only be readable by root (
chmod 0400 yoursite.key)
- the csr can be world readable, it’s not confidential. At the very least it has to be readable by the
Yay! By now we should be all set and ready to try out the validation process.
For testing it is advisable to use certbot with the
staging argument to prevent using up your daily quota of Let’s Encrypt certificates per domain.
Run this as the certbot user:
You should end up with a certificate and the two chain files as specified in the command line options. These are not really usable as they are only signed by the Let’s Encrypt staging environment, but we now know that our HAProxy setup and certbot config work. The fullchain file is what we’ll use later on with HAProxy, since it includes both the Let’s Encrypt intermediate certificate and the certificate for yoursite.com.
Automate All the Things!
certonly mode and the
--csr option it doesn’t make a difference to certbot wether you are creating a new certificate or renewing an existing one.
Therefore I also only have one shell script for both tasks. You can have a look at it below. There’s also a Gist
If called without arguments, this script will loop over the directories in
/home/certbot/certs and either create a new certificate or renew an existing one if it’s about to expire in the next 30 days.
Since the script only acts on certificates that will expire in the next 30 days, it can be called as often as you want. Once or twice daily through cron should be enough, however.
For manual operation you can also call it with one directory name as an argument, i.e.
Before this will work there are two more things to do, however.
Make the certificate usable for HAProxy
The task of taking the generated certificate and combining it with the custom
dhparam and private key file to the certificate bundle to be used by HAProxy is carried out by another script, which is called through sudo by
Setup sudo permissions for the certbot user
For things to work out non-interactively, you have to give the
certbot user permission to use passwordless sudo for these two commands:
/usr/local/sbin/le-haproxy-bundle yoursiteto build and install the certificate bundle to
/usr/sbin/service haproxy reloadto enable reloading of haproxy
There might be typos in the scripts / I might have entirely forgotten something. This article is written after the fact, and my server is provisioned with chef so I had to reconstruct / amend / generalize things. Please let me know of any oversights in the comments.
Be careful with HPKP. Use it in report-only mode and/or with short lifetimes until you’re sure you got it right.
Prior Art and Further Reading
The Certbot Documentation
The Let’s Encrypt Website
The idea to use standalone mode through a HAProxy backend and the initial version of my renewal shell script stem from this Gist.
Scott Helmes introductory article on Let’s Encrypt suggests to use a customized OpenSSL config file for each certificate, which eases CSR generation a lot.
Check out the haproxy-ocsp-stapling-updater for setting up OCSP stapling with HAProxy.
Another howto on Using haproxy with Let’s Encrypt
My Gist with all config files and scripts
Consider donating to the EFF to support their awesome work.