A secret is any value that, if leaked, immediately grants an attacker a capability: API tokens, database password, TLS private key, signing key, OAuth client secret. Configuration is everything else — values you would be comfortable committing to a public repo. The two are regularly conflated, and the operationally correct practice of treating only the secret half as a secret is the foundation of every tool below.

This topic covers the lifecycle of secrets in production — not “use a password manager”, which is the consumer version.

The Lifecycle

A secret is born, used, rotated, and dies — each stage has a discipline.

A secret that never rotates is a slow leak waiting to happen. A secret that cannot be revoked is a permanent liability. The industry shift — from static files of secrets to dynamic, short-lived credentials — is exactly this lifecycle made executable.

Static vs Dynamic Secrets

The architectural rule for any new system: prefer dynamic secrets for service-to-service credentials. Static secrets remain acceptable for long-lived platform credentials (the root KMS key, the initial Vault unseal key) — and even those should be individually held and audited.

Vault and the Lease Model

HashiCorp Vault is the canonical dynamic-secrets system. The model:

1. A workload authenticates to Vault (via Kubernetes service account, AWS IAM, GCP Workload Identity, AppRole, …).
2. Vault issues a policy-bounded lease: “you may mint a Postgres credential for database X valid for TTL = 1h, renewable until max_ttl = 24h”.
3. The lease is revocable: a breach quarantines the workload by removing its policy, immediately invalidating every credential it has minted.
4. The workload reads the credential, uses it, lets the lease expire (or renews it on every heartbeat).

Crucially, Vault is not a place to put old secrets — it is a minting authority that issues short-lived ones. Treating Vault as a static-secrets store misses the dynamic model entirely.

Cloud KMS and Envelope Encryption

Cloud KMS (AWS KMS, GCP KMS, Azure Key Vault) is the primitives layer: it holds one master key (called a Customer Managed Key, CMK) that never leaves the KMS. To encrypt a secret:

1. Generate a one-time data-encryption key (DEK) locally.
2. Encrypt your data with the DEK.
3. Ask KMS to encrypt the DEK under the CMK — get back a wrapped blob.
4. Store the ciphertext + wrapped-DEK together; store neither key.

Decrypt reverses: ask KMS to unwrap the DEK under the CMK, then use the DEK locally. The CMK appears in audit logs only, never in the application or storage. The CMK can be disabled in one place to revoke every secret encrypted under it.

This is envelope encryption, the standard pattern for storing secrets in object storage or databases without ever writing a raw key to a persistent location.

Git-Ops Secrets: SOPS

For configuration-as-code, where the secret must be committed alongside the YAML, SOPS (Secrets OPerationS, Mozilla) is the standard. SOPS encrypts individual values in a YAML/JSON file — the file structure remains visible, the values are unreadable without the decryption key.

• KMS / age backend: each value is encrypted under a public key (an age X25519 key or an AWS KMS CMK). The private key never touches the repo.
• Per-key access: rotating the decrypter revokes access to every secret encrypted under the old key.
• In-tree visibility: database_host: prod-db.example.com readable; database_password: ENC[AES256_GCM,...] encrypted. Reviewers see what changed without seeing the plaintext.

The pattern: commit SOPS-encrypted files to the git repo; the deploy pipeline decrypts at apply time using a CI-held decryption key; no plaintext is ever written to disk in the repo.

The .env Fail State

The industry’s legacy pattern — .env files of secrets — fails in three characteristic ways at scale:

1. Plaintext at rest on disk — every developer’s laptop, every cron container, every CI runner has a copy. One stolen laptop is a full secret leak.
2. Rotation requires resending — to rotate a value, every consumer must receive the new file. Tracking who received the new one is impossible; stale copies live forever.
3. No revocation — kubectl delete secret does not errase the value from disk on the nodes that already mounted it. A revoked Kubernetes Secret is still cached in the local kubelet memory until the pod restarts.

.env is acceptable for local development and for non-secret configuration (log levels, feature flags). For secrets in production, the lifecycle tools (Vault, KMS, SOPS) exist precisely because .env cannot.

Practice Trajectory

1. Spin up Vault in dev mode; configure a KV secret and a database secret engine; mint and revoke credentials. Trace the audit log.
2. Configure SOPS with age and a YAML file. Commit the encrypted file. Decrypt in a CI pipeline using a key stored in the CI secret store. Verify the plaintext never reaches the repo.
3. Pick one secret in your project currently in a .env file. Migrate it through the lifecycle — provision via Vault or KMS, document the TTL, schedule the rotation, write the revocation runbook.
4. Implement envelope encryption in 50 lines of code (your language’s crypto library + a KMS call). Note where the DEK lives and where it never does.
5. Audit a production deployment’s secret flow: from commit-time to runtime. Where is the secret in plaintext, for how long, and what revokes it if compromised?

When It’s the Right Tool