Manually Updating the TLS Certificates of the checkmk Agent Receiver

Recently, we faced a significant challenge that required us to rename a checkmk site. This process, while seemingly straightforward, involved complex steps due to the new TLS encrypted request system checkmk uses for secure communication between the agent and the agent receiver.

The Challenge

In the latest version, checkmk introduced an agent controller that communicates with the checkmk server via TLS encrypted requests. This new feature enhances security by ensuring monitoring data is transmitted securely. However, it also introduced a new layer of complexity when we had to rename a site.

When renaming a checkmk site, the site ID is inherently changed. Since the TLS certificates used by checkmk's agent-receiver are tied to this site ID, the certificates must be updated to reflect the new site name. Unfortunately this happened not automatically by using omd cp. Without the certificate updates, the agent controller cannot register itself at the checkmk agent receiver.

Our Solution: Step-by-Step Guide

Here’s a detailed guide on how we navigated this challenge and successfully updated the TLS certificates to match the new site name.

Step 1: Locate the CA Certificate Configuration

The first step is to identify where the CA certificate configurations are stored. For checkmk, these are located in the SITE_ID/etc/ssl directory.

Step 2: Create a New Site Local CA Certificate

Using the existing private key (ca.pem), we created a new local CA certificate for the site. This certificate is crucial as it acts as the root of trust for other certificates. Copy the output to file ca.pem while replacing the old certificate.

openssl req -new -key ca.pem -days 3650 -nodes -x509 -subj "/CN=Site '<<site ID>>' local CA" -extensions ca -config <(echo '[req]'; echo 'distinguished_name=req'; echo '[ca]'; echo 'subjectKeyIdentifier=hash'; echo 'basicConstraints=critical,CA:true,pathlen:0'; echo 'keyUsage=critical,keyCertSign,cRLSign')

Step 3: Create a New Site Agent Signing CA Certificate

Next, we generated a new agent signing CA certificate using the private key located in agents/ca.pem. This certificate ensures that the agents' communications are securely signed and verified. Copy the output to file agents/ca.pem while replacing the old certificate.

openssl req -new -key agents/ca.pem -days 3650 -nodes -x509 -subj "/CN=Site '<<site ID>>' agent signing CA" -extensions ca -config <(echo '[req]'; echo 'distinguished_name=req'; echo '[ca]'; echo 'subjectKeyIdentifier=hash'; echo 'basicConstraints=critical,CA:true,pathlen:0'; echo 'keyUsage=critical,keyCertSign,cRLSign')

Step 4: Create a New Certificate for the Agent Receiver Service

For the agent receiver service, we created a new certificate using the existing key (agent_receiver_cert.pem). This step is critical as the agent receiver service is responsible for handling the incoming monitoring data securely. Copy the output to file agent_receiver_cert.pem while replacing the old certificate.

openssl req -new -key agent_receiver_cert.pem -nodes -subj "/CN=<<site ID>>" | openssl x509 -req -CA ca.pem -CAkey ca.pem -days 365 -CAcreateserial -sha256 -extfile <(echo 'subjectAltName=DNS:<<site ID>>'; echo 'basicConstraints=critical,CA:false')

Step 5: Create a New Certificate for the Site

Finally, we generated a new site certificate using the key from sites/<<site ID>>.pem. This certificate is essential for the site’s identity in the TLS communication. Copy the output to file sites/<<site ID>>.pem while replacing the old certificate.

openssl req -new -key sites/hauptsystem.pem -nodes -subj "/CN=<<site ID>>" | openssl x509 -req -CA ca.pem -CAkey ca.pem -days 365 -CAcreateserial -sha256 -extfile <(echo 'subjectAltName=DNS:<<site ID>>'; echo 'basicConstraints=critical,CA:false')

Step 6: Restart checkmk

The final step was to restart checkmk to apply the changes. This ensured that all services recognized the new certificates and resumed secure communication.

Conclusion

Renaming a checkmk site and updating its TLS certificates can be a daunting task due to the intricate steps involved in regenerating and applying new certificates. However, with a clear understanding of the process and the correct commands, we successfully navigated this challenge.

We hope this detailed guide helps fellow IT professionals facing similar challenges. If you have any questions or need further assistance, feel free to reach out!