Open-source contribution as an architectural decision and product direction, lessons learned from OpenBao

The difficulty in software architecture lies not only in finding a solution to a problem, but also in determining where that solution should live within the system. This is precisely the question that my colleagues and I had to consider, and which led to this issue on the OpenBao project: https://github.com/openbao/openbao/issues/3533

The problem of a renewed certificate that is never reloaded

As part of bootstrapping a new platform, the OpenBao instance must start before the public key infrastructure (PKI) with an intermediate certificate authority (CA) has been initialised. To allow the service to start, it was therefore created with a self-signed certificate. Once OpenBao had been initialised, the intermediate PKI could be configured. The temporary certificate then had to be replaced with one issued by this PKI and subsequently renewed automatically by cert-manager. This is where we encountered a problem: the TLS certificate presented by OpenBao needed to be reloaded after rotation without requiring the service to be restarted.

OpenBao can use a certificate and private key stored in files to secure its TCP listener:

listener "tcp" {
  tls_cert_file = "/openbao/tls/tls.crt"
  tls_key_file  = "/openbao/tls/tls.key"
}

At startup, the server reads these files, constructs the TLS key pair, and then keeps it in memory. The problem arises when the certificate is renewed while the process continues to run. The files on disk contain the new certificate, but OpenBao continues to present the one it loaded at startup.

Inconsistent states

On Kubernetes using cert-manager, we may end up in a situation where:

• cert-manager has successfully renewed the certificate

• the Kubernetes Secret contains the new version

• the volume mounted in the pod exposes the new files

• OpenBao retains the old certificate in memory

  From the platform’s perspective, the rotation has succeeded. However, for the client establishing the TLS connection, nothing has changed.

This lack of change may remain invisible for several days, until the certificate still held in memory expires. The service then suddenly starts rejecting connections, even though the new certificate is available on disk.

The invariant the system must guarantee

The technical problem is simple to state: the process does not automatically reread the files. The invariant is that the listener must continue to present the latest valid certificate pair, then automatically converge towards the new one as soon as it is complete.

How do we determine which component should guarantee the invariant?

In the absence of a native reloading mechanism, the rotation operation therefore depends on an external trigger: sending a SIGHUP signal, adding a sidecar to monitor the files, periodically running a CronJob, using a hook, or restarting the pod.

These strategies shift responsibility for the certificate lifecycle to components external to the service. They increase the coupling between the application and its runtime environment and, in my view, make operations more complex. TLS certificate rotations are becoming increasingly frequent and must therefore remain transparent operations. However, in the current setup with an external trigger, a service interruption becomes more likely and could require intervention from an administrator, for example when restarting the pod with Shamir sealing enabled and no auto-unseal mechanism.

To address our bootstrap problem, several options were on the table:

• add a sidecar responsible for monitoring the files and then sending a signal to the process (for example configmap-reload, k8s-sidecar or prometheus-config-reloader)
• trigger a restart
• or modify the application directly

All these approaches can work. However, they do not place responsibility, complexity, or maintenance costs in the same location.

In Kubernetes, the sidecar must be able to access the OpenBao process in order to send it a signal. This requires enabling the shareProcessNamespace: true option, which allows containers within the same pod to share their process namespace. This workaround increases the operational complexity of the deployment and also changes the pod’s isolation properties.

By externalising the emission of a signal to update the certificate, the behaviour no longer depends directly on the actual state of the files, but assumes that an external event will be correctly generated, transmitted, and processed. This raises new questions:

• How can the event be replayed if it fails?

• How can we verify that the signal has actually been processed?

• Who is responsible for consistency between the state on disk and the state of the listener?

• Does the mechanism automatically converge towards the expected state after a transient error?

• What happens when the files are only partially modified? The rotation of the certificate and key is not necessarily atomic. The certificate may be written before the key, or vice versa. For a few moments, the two files on disk may not form a valid pair. This highlights an important rule: detecting a change must not be confused with accepting a new state (detecting a new version, loading and validating it outside the active state, and replacing the current state only when the new version is valid).

  In my view, this raises an issue that requires distinguishing the data flow from the control flow. The problem is not limited to reloading the files; it also concerns their validation, which belongs to the control flow and ensures that the system converges towards the expected state.

Other solutions exist, such as Stakater Reloader, which can monitor Secrets and automatically trigger a rolling restart of workloads. However, as stated previously, a restart is not always neutral.

None of these solutions is inherently wrong. However, each shifts responsibility to the runtime environment and requires every integrator to rebuild a similar capability around OpenBao. As always in architecture, it is a matter of trade-offs. Sometimes, the right solution is to adapt the infrastructure. Sometimes, a process and an operating procedure must be defined. And sometimes, the most sustainable decision is to change the software itself. This is what we were able to do thanks to open source.

The initial requirement seemed closely tied to Kubernetes and cert-manager. However, taking a step back revealed a more general limitation in the software.

Not every limitation encountered in open-source software necessarily justifies contributing to the product. Some are specific to an organisation or a platform. Others would be too costly to maintain relative to the expected benefit.

Before modifying a project, several questions can help determine where responsibility should lie:

• Is the problem genuinely generic?

• Are multiple users likely to resort to the same workarounds?

• Does the product already include part of the mechanism that will be involved in the solution? In our case, for example, the server already had a reloading mechanism triggered by a SIGHUP signal on which we could build.

• Is the scope of the change sufficiently well defined? The application is the only layer capable of deciding what to do when the new pair is invalid.

  My position is to follow the Tell, Don’t Ask principle, according to which behaviour should be placed alongside the data and state on which it operates. Modifying an open-source project, whatever it may be, is an architectural decision in its own right. It can eliminate local complexity, delegate responsibility by sharing maintenance with the community, and provide consistent behaviour across multiple environments.

The final solution is probably more robust than the workaround we would have implemented. The full discussion can be found here: https://github.com/openbao/openbao/pull/3530

What I find interesting is that a concrete problem encountered in the field within a local context could be generalised and lead to the addition of a new feature.