Secrets Management in Production: Vault, AWS Secrets Manager, SOPS

In 2023, I was reviewing a pull request for a junior developer's first feature. Everything looked good — clean code, proper error handling, solid test coverage. Then I scrolled to the bottom and saw it: const STRIPE_SECRET_KEY = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"; hardcoded in the source file. The key was live. In production. Committed to a public GitHub repository.

We rotated the key within minutes, but the damage was already done. GitHub's cache, Google's crawlers, and who knows how many automated scanners had already seen it. According to GitGuardian's 2024 State of Secrets Sprawl Report, 12.8 million new secrets were leaked in public GitHub repositories in 2023 alone — a 28% increase from the previous year. That's 35,000 secrets per day.

Secrets management isn't a "nice to have" security practice. It's the foundation that every other security measure depends on. Your database encryption means nothing if the encryption key is in an environment variable on a developer's laptop. Your network firewall is useless if the VPN credentials are in a Slack message. This guide covers the three most popular secrets management solutions and gives you a concrete implementation plan.

What Counts as a "Secret" (It's More Than You Think)

Before diving into tools, let's define scope. A "secret" is any piece of information that, if exposed, could grant unauthorized access to systems, data, or services. This includes:

• API keys: Stripe, Twilio, SendGrid, third-party service keys
• Database credentials: Connection strings, passwords, certificates
• Encryption keys: AES keys, RSA private keys, JWT signing keys
• OAuth tokens: Client secrets, refresh tokens, service account tokens
• SSH keys: Private keys for server access, deploy keys
• TLS certificates: Private keys for HTTPS termination
• Cloud credentials: AWS access keys, GCP service account JSON, Azure client secrets
• Internal service passwords: Redis, RabbitMQ, Elasticsearch authentication

A 2024 IBM Cost of a Data Breach Report found that compromised credentials were the most common initial attack vector, responsible for 16% of all breaches with an average cost of $4.81 million per incident. The report also found that breaches involving compromised credentials took the longest to identify — an average of 292 days.

The Anti-Patterns: What NOT to Do

Before we discuss solutions, let's catalog the common mistakes. If your team does any of these, you have a secrets management problem:

The Three Approaches Compared

1. HashiCorp Vault: The Enterprise Standard

HashiCorp Vault is the most feature-rich secrets management platform. It's not just a key-value store — it's a complete secrets lifecycle manager with dynamic credentials, encryption as a service, and fine-grained access control.

Key Features:

• Dynamic secrets: Generate short-lived database credentials on demand. No shared passwords, no rotation needed — credentials expire automatically.
• Encryption as a Service (Transit): Encrypt/decrypt data without exposing encryption keys to applications.
• PKI management: Issue and manage TLS certificates.
• Identity-based access: Authenticate using Kubernetes service accounts, AWS IAM roles, or OIDC tokens.
• Audit logging: Every secret access is logged with who, what, and when.

# Initialize Vault (first time setup)
vault operator init -key-shares=5 -key-threshold=3

# Unseal Vault (requires 3 of 5 key holders)
vault operator unseal $KEY_1
vault operator unseal $KEY_2
vault operator unseal $KEY_3

# Store a secret
vault kv put secret/myapp/production \
  database_url="postgres://user:pass@host:5432/db" \
  stripe_key="sk_live_xxx" \
  jwt_secret="super-secret-key"

# Read a secret
vault kv get -field=database_url secret/myapp/production

# Enable dynamic database credentials
vault secrets enable database

vault write database/config/myapp-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="myapp-readonly,myapp-readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/myapp" \
  username="vault_admin" \
  password="admin_password"

vault write database/roles/myapp-readonly \
  db_name=myapp-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Application requests short-lived credentials
vault read database/creds/myapp-readonly
# Returns: username=v-token-readonly-abc123, password=xyz789, ttl=1h

Application Integration (Node.js):

const vault = require('node-vault')({
  apiVersion: 'v1',
  endpoint: process.env.VAULT_ADDR,
  token: process.env.VAULT_TOKEN // Or use Kubernetes auth
});

async function getSecrets() {
  const result = await vault.read('secret/data/myapp/production');
  return result.data.data; // { database_url, stripe_key, jwt_secret }
}

// With Kubernetes authentication (no static token needed)
async function authenticateWithK8s() {
  const jwt = fs.readFileSync('/var/run/secrets/kubernetes.io/serviceaccount/token', 'utf8');
  const result = await vault.kubernetesLogin({
    role: 'myapp',
    jwt: jwt
  });
  vault.token = result.auth.client_token;
}

2. AWS Secrets Manager: The Managed Solution

AWS Secrets Manager is a fully managed service for storing and rotating secrets. If you're on AWS, it's the path of least resistance — no infrastructure to manage, native integration with RDS, Lambda, ECS, and EKS.

# Store a secret
aws secretsmanager create-secret \
  --name myapp/production/database \
  --secret-string '{"username":"admin","password":"MySecureP@ss","host":"db.internal","port":5432}'

# Retrieve a secret
aws secretsmanager get-secret-value --secret-id myapp/production/database

# Enable automatic rotation (every 30 days)
aws secretsmanager rotate-secret \
  --secret-id myapp/production/database \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456:function:secret-rotator \
  --rotation-rules AutomaticallyAfterDays=30

Application Integration (Node.js):

const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');

const client = new SecretsManagerClient({ region: 'us-east-1' });

async function getSecret(secretName) {
  const command = new GetSecretValueCommand({ SecretId: secretName });
  const response = await client.send(command);
  return JSON.parse(response.SecretString);
}

// Cache secrets to avoid API calls on every request
let cachedSecrets = null;
let cacheExpiry = 0;

async function getCachedSecret(secretName) {
  if (cachedSecrets && Date.now() < cacheExpiry) {
    return cachedSecrets;
  }
  cachedSecrets = await getSecret(secretName);
  cacheExpiry = Date.now() + 300000; // Cache for 5 minutes
  return cachedSecrets;
}

3. SOPS: Encrypted Files in Git

SOPS (Secrets OPerationS) by Mozilla takes a different approach: it encrypts secrets within files that you commit to git. The encrypted files are version-controlled alongside your code, but only authorized users/services can decrypt them.

# Encrypt a YAML file using AWS KMS
sops --encrypt --kms arn:aws:kms:us-east-1:123456:key/abc-123 secrets.yaml > secrets.enc.yaml

# The encrypted file looks like this:
# database_url: ENC[AES256_GCM,data:abc123xyz...,tag:def456]
# stripe_key: ENC[AES256_GCM,data:ghi789...,tag:jkl012]
# Keys are visible, values are encrypted

# Decrypt and edit
sops secrets.enc.yaml  # Opens in $EDITOR with decrypted values

# Decrypt in application
sops --decrypt secrets.enc.yaml

# Use with age (simpler key management than KMS)
age-keygen -o keys.txt  # Generate a key pair
sops --encrypt --age age1abc123... secrets.yaml > secrets.enc.yaml

Application Integration:

# In a Docker entrypoint script
#!/bin/bash
# Decrypt secrets and export as environment variables
eval $(sops --decrypt --output-type dotenv secrets.enc.yaml)
exec "$@"  # Run the actual application

Head-to-Head Comparison

Pre-Commit Secret Scanning

Prevention is better than remediation. These tools scan your code for secrets before they're committed to git:

# Install gitleaks (the best open-source option)
brew install gitleaks

# Scan current repository
gitleaks detect --source . --verbose

# Set up as a pre-commit hook
# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.1
    hooks:
      - id: gitleaks

# GitHub-native: enable push protection
# Settings -> Code security -> Secret scanning -> Push protection -> Enable

GitHub's push protection (free for public repos, available for private repos on Enterprise) blocks pushes that contain known secret patterns. According to GitHub, it prevented over 1 million secret leaks in its first year.

My Opinionated Recommendations

1. Start with SOPS, graduate to Vault. SOPS is the fastest path from "secrets in .env files" to "secrets encrypted and version-controlled." You can implement it in an afternoon. When you outgrow it (need dynamic credentials, complex access policies, or compliance auditing), migrate to Vault.

2. Never store secrets in Kubernetes Secrets without encryption. Kubernetes Secrets are base64-encoded, not encrypted. Enable etcd encryption at rest, or better yet, use the Vault CSI driver or External Secrets Operator to inject secrets from a proper secrets manager.

3. Rotate secrets on a schedule, not just after incidents. Every secret should have a maximum TTL. API keys: 90 days. Database passwords: 30 days. JWT signing keys: 7 days. Vault's dynamic secrets solve this automatically — credentials expire and are regenerated without human intervention.

4. The .env file is not a secrets manager. I see teams with a .env.production file shared via Slack or stored in a private GitHub repo. This is barely better than hardcoding. Every person who has ever had access to that Slack channel has the production database password forever.

5. Audit access, not just storage. Knowing that your secrets are encrypted is good. Knowing who accessed which secret at what time is better. Vault's audit log and AWS CloudTrail provide this. SOPS via git history provides a partial view.

Action Plan: From Zero to Secure in 3 Weeks

Week 1: Assessment and Quick Wins

• Audit your codebase for hardcoded secrets (run gitleaks on full history)
• Install gitleaks as a pre-commit hook across all repositories
• Rotate any secrets found in git history (they're already compromised)
• Enable GitHub secret scanning and push protection

Week 2: Implementation

• Choose your tool: SOPS (small team), AWS Secrets Manager (AWS-native), or Vault (enterprise)
• Migrate all production secrets from .env files to your chosen tool
• Update application code to fetch secrets from the secrets manager at startup
• Implement secret caching to avoid API rate limits and latency

Week 3: Hardening

• Set up secret rotation for database credentials and API keys
• Enable audit logging and set up alerts for unusual access patterns
• Document the secrets management workflow for the team
• Create a runbook for secret rotation and incident response

Sources and Further Reading

• GitGuardian — 2024 State of Secrets Sprawl Report
• IBM — 2024 Cost of a Data Breach Report
• HashiCorp Vault Documentation
• AWS Secrets Manager Documentation
• Mozilla SOPS (GitHub)
• Gitleaks — Secret Detection Tool
• GitHub — Secret Scanning Push Protection
• OWASP — Secrets Management Cheat Sheet

I'm Ismat, and I build BirJob — Azerbaijan's job aggregator scraping 80+ sources daily.