Chapter 13

Secret Manager

Scope. This chapter covers Secret Manager as the resource holding a versioned, access-controlled credential: the secret and version model, IAM, rotation notification, replication, the four consumption paths, audit logging, and getting a secret's plaintext out of Terraform state and out of Git. Cloud KMS and every cryptographic control — key rings, keys, CMEK, rotation mechanics, destruction — is Chapter 14; this chapter shows only the wiring where a secret is encrypted with a customer-managed key. Prerequisites. §2.21 Security Projects, §3.9 IAM Roles, §3.17 Deny Policies, §3.24 Service Account Impersonation, §3.25 Service Account Keys. Verified against. Google Cloud console and API surface as of 2026-09, Cloud SDK 583.0.0, hashicorp/google provider 8.x; see sources at end.

Every application has credentials it must never hold in its own source: a database password, a third-party API key, a signing key, a webhook secret. Secret Manager is the service that stores that value, versions it, controls who can read it, and hands it to a workload at run time instead of at commit time. Getting this chapter wrong looks like success for months — the secret works, the pipeline is green — and then fails all at once, either because a credential leaked and nobody could tell which version was live, or because a "rotation" that was configured turned out never to have rotated anything.

The design questions this chapter answers are narrow but recur constantly: where does a secret live relative to the project that uses it, who may read it versus who may create a new version of it, what happens when it is time to change the value, and how does each of Compute Engine, GKE, Cloud Run, and a CI/CD pipeline actually obtain it without a long-lived key ever touching disk. The two costliest misconceptions in production are that --rotation-period rotates a secret, and that reading latest on every call is safe. Neither is true, and both are addressed directly below.

13.1 Secret Manager Architecture §

A Secret Manager secret is a named container for versions: immutable payloads holding the actual credential bytes. Creating a secret does not create a version — gcloud secrets create with no --data-file produces an empty container; adding a version is a separate call. Every version has a numeric identifier, a lifecycle state (ENABLED, DISABLED, DESTROYED), and a fixed replication policy chosen when the secret is created and never changed afterward: automatic (Google chooses regions) or user-managed (you list --locations). A regional secret, created with --location, is a distinct resource type addressed under a location rather than replicated.

Two access surfaces sit on top of this model. The control plane — creating a secret, adding a version, changing IAM, setting rotation — is audited in Admin Activity logs by default. The data plane — reading a version's payload with AccessSecretVersion — is Data Access logging, off by default; §13.12 covers turning it on.

Placement follows §2.21: a secret used by a single application in a single project is project-local, next to the workload that reads it; a secret shared across projects — a shared upstream API credential, a cross-tenant signing key — is created in rc-saas-shared-sec-01 and granted out per-consumer, the same boundary §2.21 sets for keys and SCC configuration.

Console. Security → Secret Manager → Create secret sets the name, initial value, replication, and optional CMEK key, topics, and rotation in one form; every later change is a new version, not an edit of the first.

API. secretmanager.googleapis.com, resource projects/PROJECT_ID/secrets/SECRET_ID, versions at projects/PROJECT_ID/secrets/SECRET_ID/versions/N. Regional secrets add a locations/LOCATION segment.

Naming. This book names secrets <app>-<env>-<purpose>, e.g. billing-prod-db-password, billing-prod-stripe-api-key — enough to identify the consumer and blast radius from the name alone, without a description lookup.

Pitfall. The maximum payload is 64 KiB. A secret is a credential, not a configuration bundle; a TLS certificate chain with a large intermediate set or a JSON blob of many credentials can exceed it — split into one secret per credential instead of concatenating.

13.2 Secrets §

A secret is a container of versions plus metadata — it holds no payload itself. The metadata carries labels for cost and inventory, annotations, and three lifecycle controls worth setting deliberately:

• --expire-time or --ttl deletes the entire secret and every version in it once reached. This is a destructive timer on a production credential, so set it only where a secret is genuinely temporary.
• --version-destroy-ttl delays payload deletion after a destroy call, leaving a window in which a mistaken destruction can be caught and investigated in Data Access logs (§13.12) before the payload is unrecoverable.
• --replication-policy, which is fixed at creation and can never be changed (§13.11).

gcloud secrets create billing-prod-db-password \
  --replication-policy=user-managed \
  --locations=us-central1,europe-west1 \
  --labels=app=billing,env=prod \
  --version-destroy-ttl=86400s

--secret-type documents, it does not enforce. The annotation tells tooling what a secret holds; it restricts neither the payload format nor who may read it. Do not present it as a control in a design review.

Pitfall. --ttl on a secret deletes the secret itself, not just old versions — a detail that reads as harmless housekeeping and takes a live credential with it. Use lifecycle management on versions for hygiene, and reserve secret-level expiry for genuinely ephemeral secrets.

13.3 Secret Versions §

A version's payload is immutable: there is no update operation, only gcloud secrets versions add to create a new one. disable makes a version unreadable without destroying it — the reversible way to take a credential out of service — while destroy deletes the payload irreversibly once any --version-destroy-ttl elapses.

The alias latest always resolves to the highest-numbered ENABLED version at access time. It is not a pointer you set: it moves the instant a new version is added and enabled. A workload that reads latest on every request picks up a new credential immediately; one that reads it only at startup picks it up on the next restart; one that caches the value indefinitely never picks it up at all. Production code should pin a specific version number and roll forward deliberately — this is what makes rotation (§13.5) a non-breaking, two-version overlap rather than a race.

gcloud secrets versions list billing-prod-db-password
gcloud secrets versions disable 3 --secret=billing-prod-db-password
gcloud secrets versions destroy 1 --secret=billing-prod-db-password

13.4 IAM §

Secret Manager IAM binds at the secret, the project, or higher. The most common misconfiguration in the product is a single line:

roles/secretmanager.secretAccessor granted at the project level grants read access to every current and future secret in that project.

It is the binding that "just works" for whoever asked, it is invisible in a per-secret review because it is not on any secret, and it silently extends to secrets created years later. The least-privilege position is a binding on each individual secret.

gcloud secrets add-iam-policy-binding billing-prod-db-password \
  --member='serviceAccount:sa-run-billing@rc-saas-prod-app-01.iam.gserviceaccount.com' \
  --role='roles/secretmanager.secretAccessor'

Creation and consumption are different principals, the same separation §2.21 draws for keys and §14.13 applies to Cloud KMS: the rotation handler gets secretVersionAdder and can write a new credential without ever reading the current one, while the workload gets secretAccessor alone and can never change what it reads.

Pitfall. roles/secretmanager.secretVersionAdder looks harmless because it cannot read. It can still add a version — so a compromised rotation handler can overwrite a credential the application will then use, redirecting the application to an attacker-controlled upstream without ever reading a secret. Treat write access to a secret as equivalent in blast radius to read access.

13.5 Rotation §

Secret Manager does not rotate secrets. This is the single most misunderstood thing about the product, so state it plainly:

"Secret Manager does not rotate secrets itself. Instead, it sends notifications to Pub/Sub topics."

--rotation-period and --next-rotation-time schedule a message, nothing more. Minting a new credential in the upstream system and calling versions add with it is work you implement and deploy. A secret with rotation "configured" and no handler subscribed will notify forever and never change — while every dashboard reports rotation as enabled.

gcloud pubsub topics create billing-prod-db-password-rotate

gcloud secrets update billing-prod-db-password \
  --add-topics=projects/rc-saas-prod-app-01/topics/billing-prod-db-password-rotate \
  --next-rotation-time="2026-10-01T04:00:00Z" \
  --rotation-period=2592000s

The Secret Manager service agent publishes the notification, so it needs publish rights:

SA="service-PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com"
gcloud pubsub topics add-iam-policy-binding billing-prod-db-password-rotate \
  --member="serviceAccount:${SA}" \
  --role='roles/pubsub.publisher'

The handler's sequence matters, and it is the part that makes rotation non-breaking:

1. Authenticate to the upstream system and generate a new credential there.
2. Add it as a new secret version.
3. Only then disable the version it replaced — never destroy it immediately, so callers pinned to the old version number keep working through the overlap window (§13.3).

Make the handler idempotent: Secret Manager retries a failed delivery for up to seven days, so a handler that mints a credential per delivery rather than per rotation will churn the upstream account.

Pitfall. Rotation that adds a version but never disables the old one is not rotation — it is accumulation. The compromised credential you rotated away from stays valid indefinitely, which defeats the entire exercise. The disable step is the one that actually revokes.

13.6 Application Access §

A workload reads a secret through a client library or a direct AccessSecretVersion call, authenticated as its attached identity — never by embedding the value in an environment variable set at build time, which bakes a version's plaintext into an image or a deploy manifest and defeats rotation completely.

DB_PASSWORD=$(gcloud secrets versions access 7 \
  --secret=billing-prod-db-password)

Resolve the version into a shell variable rather than nesting the substitution inside a larger command; it is easier to audit and easier to log safely.

Pin the version number in configuration, not latest (§13.3), and treat a version bump as a deployment: update the pinned number, redeploy, confirm health, then disable the prior version. That sequence is what turns rotation into an ordinary, reversible release rather than a simultaneous change across every running instance.

Cache deliberately. Calling AccessSecretVersion on every request is a latency and quota cost for a value that changes monthly; caching it forever means rotation never takes effect. Cache for a bounded interval, and re-read on authentication failure.

Pitfall. A secret read into an environment variable is visible to every process in the container, to anything that can read /proc, and to most crash handlers and error reporters, which serialize the environment into a stack trace. Read secrets into process memory and keep them out of the environment where the threat model warrants it.

13.7 GKE Integration §

GKE reaches Secret Manager through the Secret Manager add-on (secrets-store-gke.csi.k8s.io), a Google-managed CSI driver you enable on the cluster or node pool rather than install yourself:

gcloud container clusters update rc-saas-prod-gke-01 \
  --location=us-central1 \
  --enable-secret-manager

A SecretProviderClass maps secret resource paths to files mounted into the pod's filesystem, read by the workload's own attached Kubernetes service account under Workload Identity Federation (§4.12, §9.11) rather than a mounted key. --enable-secret-manager-rotation re-syncs the mounted file on an interval so a workload reading the file (not caching it in memory) picks up a new version without a restart — a different mechanism from §13.5's Pub/Sub notification, and it still does not mint the credential itself. GKE and its identity model are Chapter 9; this section covers only how the secret reaches the pod filesystem.

13.8 Cloud Run Integration §

Cloud Run mounts a secret two ways, and they have opposite freshness behavior — which is the whole of what a reviewer needs to know here:

gcloud run services update billing-api-prod \
  --project=rc-saas-prod-app-01 --region=us-central1 \
  --update-secrets=/secrets/db/password=billing-prod-db-password:7

Pin a version in either mode (§13.3). A volume mount configured to latest picks up a new version mid-process, which sounds desirable and means the application can observe the credential changing underneath an open connection pool — the failure is intermittent and looks like a database problem. Pin the version, and roll it forward as a deployment.

Volume mounts do not support regional secrets (§13.11), so a service in a residency-constrained deployment is pushed toward the environment-variable form and its redeploy-to-rotate behavior. Decide that pairing deliberately rather than discovering it when the mount fails.

Grant roles/secretmanager.secretAccessor on the specific secret to the service's attached identity (sa-run-<service>), never to a shared account. Chapter 10 owns the Cloud Run revision and identity model; this section covers only the secret path.

13.9 Compute Engine Integration §

A VM has no dedicated secret-mounting mechanism; it reaches Secret Manager the same way it reaches any Google API — through its attached user-managed service account (Chapter 8) obtaining a short-lived token from the metadata server, with no key on disk. Grant that service account roles/secretmanager.secretAccessor on the specific secrets it needs, and call the API from a client library or a startup script:

DB_PASSWORD=$(gcloud secrets versions access 7 \
  --secret=billing-prod-db-password)

A startup script that writes the plaintext to a file on local disk recreates the problem Secret Manager solves — hold the value in memory for the process that needs it, or write it to tmpfs, never to persistent disk. §8.16 owns the service account attachment and hardening this depends on.

13.10 CI/CD Integration §

A pipeline should never hold a long-lived credential for Secret Manager itself; it authenticates through Workload Identity Federation (§4.3, §26.4) and impersonates a deploy service account (§3.24) scoped to roles/secretmanager.secretAccessor on exactly the secrets that build needs — never a downloaded service account key, which is itself the secret this chapter exists to eliminate.

Cloud Build declares secrets in cloudbuild.yaml rather than fetching them imperatively:

availableSecrets:
  secretManager:
    - versionName: projects/rc-saas-shared-cicd-01/secrets/\
billing-prod-stripe-api-key/versions/latest
      env: STRIPE_KEY
steps:
  - name: bash
    entrypoint: bash
    args: ["-c", "curl -H \"Authorization: Bearer $STRIPE_KEY\" ..."]
    secretEnv: ["STRIPE_KEY"]

The secret is referenced only inside args, escaped with $, and never written to build logs. Grant the Cloud Build service account secretAccessor on that one secret, not project-wide.

13.11 Secret Replication §

Replication is fixed at creation and cannot be changed afterward. automatic lets Google choose storage locations and is the right default for a secret with no data-residency requirement. user-managed pins storage to the --locations you list, which is what a residency or sovereignty constraint (§15.8, §15.9) requires — and each location can carry its own customer-managed key. A CMEK'd secret requires a user-managed replication policy; automatic replication cannot be paired with CMEK. Wiring is --kms-key-name at creation (global) or --regional-kms-key-name on a regional secret, plus the service agent grant:

gcloud secrets add-iam-policy-binding billing-prod-db-password \
  --member="serviceAccount:service-PROJECT_NUMBER@gcp-sa-secretmanager\
.iam.gserviceaccount.com" \
  --role='roles/cloudkms.cryptoKeyEncrypterDecrypter'

The key ring itself, its region, and rotation are Chapter 14 (§14.8); this is the whole of the wiring on the Secret Manager side.

13.12 Auditing Secret Access §

Data Access audit logs are not enabled by default, so every AccessSecretVersion call — every read of a secret's plaintext — goes unrecorded until you turn the log type on. This is the single most important operational fact in this chapter for incident response: without it, "who read this credential before it leaked" has no answer.

cat > /tmp/audit-policy.yaml <<'EOF'
auditConfigs:
  - service: secretmanager.googleapis.com
    auditLogConfigs:
      - logType: DATA_READ
EOF
gcloud projects set-iam-policy rc-saas-prod-app-01 /tmp/audit-policy.yaml

Query recorded access with the exact method name:

gcloud logging read \
  'protoPayload.methodName="google.cloud.secretmanager.v1.SecretManagerSer\
vice.AccessSecretVersion"' \
  --project=rc-saas-prod-app-01 --freshness=7d

Route this alongside the control-plane events already covered by Admin Activity logging — CreateSecret, AddSecretVersion, SetIamPolicy — to the logging project (§2.22) so both planes land in one retained, tamper-evident sink.

13.13 Eliminating Secrets from Terraform State §

google_secret_manager_secret_version with secret_data stores the plaintext value in the Terraform state file, in the clear, for as long as that state exists — including every prior version if state is versioned in Cloud Storage per §26.14. This is the same exposure §2.21 treats state as: a secret store in its own right, requiring the same access controls as the secrets it might contain.

The provider's write-only alternative avoids this entirely:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 8.0"
    }
  }
}

resource "google_secret_manager_secret" "db_password" {
  secret_id = "billing-prod-db-password"

  replication {
    user_managed {
      replicas {
        location = "us-central1"
      }
      replicas {
        location = "europe-west1"
      }
    }
  }
}

resource "google_secret_manager_secret_version" "db_password" {
  secret                 = google_secret_manager_secret.db_password.id
  secret_data_wo         = var.db_password
  secret_data_wo_version = 1
}

secret_data_wo is declared write-only and is never persisted to state; secret_data_wo_version is a trigger you increment to force a new version when the value changes. Write-only arguments require Terraform 1.11 or later — confirm the runner's version before adopting this pattern. Exactly one of secret_data or secret_data_wo may be set on a version resource; mixing them is a plan-time error. Prefer supplying var.db_password from an ephemeral source (a rotation handler's own output, or a value piped in at apply time) rather than a .tfvars file, which reintroduces the exact problem on disk that write-only state avoids in state. §26.17 owns the general practice of protecting sensitive state; this section is the Secret-Manager-specific instance of it.

13.14 Eliminating Secrets from Git §

The only mechanism worth trusting here is that no credential is ever typed into a file that gets committed: every value that would be a secret is created in Secret Manager first and referenced by resource path, never copied into source. That discipline needs enforcement on both ends. A pre-commit hook (detect-secrets, gitleaks, or equivalent, run locally) catches an accidental paste before it reaches a remote; a CI-side scan of every pushed commit is the backstop for the hook someone skipped. Chapter 15 owns content inspection at scale — Sensitive Data Protection's credential infoTypes can scan repository exports and build artifacts for exactly this class of finding; Chapter 37 owns the broader supply-chain controls, including source integrity, that this fits inside.

Security Command Center has no native repository secret-scanning detector; its finding catalog covers cloud resources rather than source history, so Sensitive Data Protection is the first-party mechanism for a committed credential and Chapter 37 owns everything around it.

Treat a secret that reaches Git as compromised, not merely exposed: rotate it in the upstream system and add a new Secret Manager version before doing anything about the Git history. Rewriting history removes the text but not the fact that the value was public; skip straight to rotation, then clean up history as a secondary step so the credential the old commits reference is already worthless.

Chapter Summary §

• A secret is a container for immutable versions; a version's lifecycle is ENABLED, DISABLED, or irreversibly DESTROYED.
• Project-local secrets live with their consuming project; secrets shared across projects belong in rc-saas-shared-sec-01 per §2.21.
• Maximum payload is 64 KiB; store one credential per secret.
• roles/secretmanager.secretAccessor at the project level reaches every secret in the project — bind at the secret instead.
• Secret Manager does not rotate anything; --rotation-period only publishes a Pub/Sub notification, and a handler you build and deploy must mint the new credential and add the version.
• The latest alias resolves at access time; production pins a version number and rolls forward deliberately, which is what makes the two-version overlap non-breaking.
• Replication policy (automatic or user-managed) is fixed at creation and cannot change afterward; CMEK requires user-managed replication.
• CMEK wiring is --kms-key-name/--regional-kms-key-name plus a roles/cloudkms.cryptoKeyEncrypterDecrypter grant to the Secret Manager service agent; the key itself is Chapter 14.
• GKE consumes secrets through the Secret Manager CSI add-on (--enable-secret-manager), mounted as files under the pod's own Workload Identity Federation identity.
• Cloud Run environment variables resolve once at deploy time; volume mounts re-fetch on every read, including latest.
• Compute Engine and CI/CD both reach Secret Manager through an attached or federated identity — never a downloaded service account key.
• Data Access audit logs for secretmanager.googleapis.com are off by default; enabling DATA_READ is what makes AccessSecretVersion visible.
• secret_data in google_secret_manager_secret_version persists plaintext to Terraform state; the write-only secret_data_wo pair (Terraform 1.11+) avoids it.
• A secret that reaches Git is compromised, not merely exposed: rotate it before rewriting history, and use pre-commit plus CI scanning to prevent the next one.

Security Checklist §

Sources §

• Secret Manager overview — https://cloud.google.com/secret-manager/docs/overview (last validated 2026-09-03)
• Create and access a secret — https://cloud.google.com/secret-manager/docs/create-secret (last validated 2026-09-03)
• Secret rotation — https://cloud.google.com/secret-manager/docs/secret-rotation (last validated 2026-09-03)
• Configuring secrets for Cloud Run — https://cloud.google.com/run/docs/configuring/secrets (last validated 2026-09-03)
• Using secrets in Cloud Build — https://cloud.google.com/build/docs/securing-builds/use-secrets (last validated 2026-09-03)
• Understanding audit logs (Data Access logging) — https://cloud.google.com/logging/docs/audit/understanding-audit-logs (last validated 2026-09-03)
• IAM service agents — https://cloud.google.com/iam/docs/service-agents (last validated 2026-09-03)
• Sensitive Data Protection infoTypes reference — https://cloud.google.com/sensitive-data-protection/docs/infotypes-reference (last validated 2026-09-03)
• SecretPayload proto (64 KiB maximum) — https://googleapis.github.io/googleapis/master/google/cloud/secretmanager/v1/resources.proto (last validated 2026-09-03)
• Secret Manager add-on for GKE — https://cloud.google.com/secret-manager/docs/secret-manager-managed-csi-component (last validated 2026-09-03)
• Attaching a service account to a VM — https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances (last validated 2026-09-03)
• google_secret_manager_secret_version resource — https://raw.githubusercontent.com/hashicorp/terraform-provider-google/main/website/docs/r/secret_manager_secret_version.html.markdown (last validated 2026-09-03)
• Terraform write-only arguments — https://developer.hashicorp.com/terraform/language/resources/ephemeral/write-only (last validated 2026-09-03)