Many folks and organizations are running their own Certificate Authority (CA) internally, to manage certificates for internal services and (infrastructure) devices, which won’t ever be exposed to regular users. Some manage those CAs manually, e.g., via OpenSSL directly, or via tools like easyrsa, cfssl, etc.
In the Freifunk Hochstift infrastructure, we use an internal CA for host certificates (e.g., used for Icinga2), internal services like LDAP, OpenVPN, NetBox, IPMIs, etc. The CA is managed by some helper scripts directly calling the openssl binary. When I created the CA back in 2015 I didn’t think much about the expiry date, set it to 10 years from then, and focused on the infrastructure to be set up. I’ve also set up monitoring for the certificates of the services, we deployed, but did not set up monitoring for the CA expiry.
So here we are, and recently, those 10 years were up, and our OpenVPN used for managed access stopped failing because it couldn’t connect to the Anycasted LDAP servers. All 4 instances failing at once was wild and didn’t make much sense. Some debugging revealed problems to be related to SSL/TLS, yet the service certificates were OK. After a while, I figured out the CA had expired, but now what?
Turns out, you can build a new certificate from an existing key and get yourself unstuck. As this journey wasn’t very pleasant, I stumbled across some mostly-but-not-fully-working articles out there, and had to dive into /usr/lib/ssl/misc/CA.pl, here’s the way I got myself out of this mess.
This article assumes, that you’re using an openssl based CA, and that you know how to handle the openssl command line. If you’re using a tool like easyrsa, cfssl, etc., the same basic steps should likely apply, however, ideally the tooling has knobs or ways to guide you through this endeavor.
In my setup, I have a /etc/ssl/openssl.cnf, which is configured to point to the right CA directory, so the openssl commands used below, know where to find the CA. The relevant setting is the following:
[ CA_default ]
dir = /path/to/ca/ # Where everything is keptCreating a new CA certificate
Luckily, by the design of asymmetric crypto, it’s possible to generate a new certificate from an existing key. To be sure, the new CA certificate to work as a drop-in replacement of the old one, I made sure to have the same CN and other attributes, I’m not 100% sure if that is technically required.
Cleanup
Before we can issue a new CA certificate (with the same common name), we need to revoke the old one, or disable the uniqueness constraint in the index.txt.attr file. I chose to do the first, we the existing certificate wasn’t worth anything any more anyway. To revoke the old CA certificate, we need it within a file.
# Revoke existing CA certificate
openssl ca -revoke cacert.pemCreate new CSR from original key
As with any certificate, this requires a Certificate Signing Request (CSR). However, in contrast to a regular certificate, where we usually, generate a new key and CSR at the same time, we want this CSR to be using the existing CA private key.
# Generate new CSR from existing private key
openssl req -new -key private/cakey.pem -out newCA-2025.csrSign new CSR to get new CA
Once we have the new CSR, we can sign it. We need to make sure, the CA extension is used it to become a usable CA. The following command will sign the CSR in newCA-2025.csr using the private/cakey.pem and will write the new CA certificate to cacert-2025.pem.
openssl ca -create_serial -days 3650 -keyfile private/cakey.pem -extensions v3_ca -in newCA-2025.csr -out cacert-2025.pem
Congrats, you’re the proud owner of a new CA certificate file. Now you just have to have a way to roll out this new certificate to all devices, which require it.
Rollout and services
Luckily for us, we are using SaltStack to managed our devices, which has its own key management and wasn’t affected of the expired certificate. So I could update the CA certificate in the Salt repo and trigger a global rollout. Phew.
Icinga2
For monitoring, we largely rely on Icinga2, which in this situation wasn’t great, as it’s also using the host certificates (and CA) to establish connections between client and server, so a significant part of our monitoring went blind. Luckily, rolling out the new certificate and restarting the Icinga2 service was easily done via Salt, too. Phew, again.
On that note, if you find yourself in the same situation, fix the CA cert on the Icinga2 server(s) first, restart the server, and then fix and restart all nodes. Don’t ask me how I know, and how much time this realization had cost me.