Skip to content

ExternalSecrets with Kubernetes Backend

Date: 2026-03-23 Status: Accepted Context: Distributing secrets (database credentials, TLS certs) across namespaces without committing them to git

Decision

Use the ExternalSecrets Operator (ESO) with a Kubernetes provider (in-cluster) as the secret backend. Secrets are stored as native Kubernetes secrets in a source namespace and distributed to target namespaces via ClusterSecretStore.

How it works

graph LR
    source["<b>Source namespace</b><br/>e.g. prj2-system<br/>Secret: postgres-password"] -->|"ESO sync"| target["<b>Target namespaces</b><br/>e.g. prj2-2026-*<br/>Secret: postgres-password"]

Used for two purposes:

  1. TLS certificate replication — wildcard cert from cert-manager namespace → all workload namespaces
  2. Database credential sharing — PostgreSQL password from prj2-system → team namespaces

Rationale

  • Zero extra infrastructure — no Vault server to deploy, configure, unseal, and maintain. The Kubernetes API is the secret store.
  • Sufficient for our threat model — students have no kubectl access, so in-cluster secrets are not exposed to them. The risk is primarily around git hygiene (not committing secrets), not runtime secret exfiltration.
  • Stepping stone — ESO supports multiple backends. If we later need Vault (e.g., for Tier 2 self-service or cross-cluster secrets), we can swap the backend without changing the ExternalSecret resources.
  • GitOps-compatible — ExternalSecret manifests (which reference secrets by name, not value) can be committed to git safely.

Alternatives Considered

HashiCorp Vault

  • ✅ Industry-standard secret management with audit logging, dynamic secrets, rotation
  • ❌ Significant operational overhead (deployment, unsealing, backup, HA)
  • ❌ Overkill for our current scale and threat model
  • Rejected: Too much infrastructure for the current use case. May revisit for Tier 2.

SealedSecrets

  • ✅ Encrypt secrets in git, decrypt in-cluster
  • ❌ One-way: secrets are sealed per-cluster, no cross-namespace distribution
  • ❌ Doesn't solve the TLS replication or credential sharing use case well
  • Rejected: Limited distribution model

Plaintext in git

  • ❌ Security risk — credentials visible in repository history
  • Current state for some secrets (Grafana admin, Harbor admin) — migration to ESO is a known TODO

Consequences

Positive

  • No additional services to operate
  • Secret distribution is declarative and auditable via ExternalSecret manifests
  • Backend is swappable without changing consumer resources

Negative

  • Kubernetes secrets are base64-encoded, not encrypted at rest (unless etcd encryption is configured)
  • Some secrets still committed as plaintext in the IaC repo — needs cleanup
  • No audit trail for secret access (Vault would provide this)