Back matter

Complete GCP Production Checklist

This checklist is intended as a final architecture and production-readiness review.

It is deliberately broad.

Not every control applies to every environment. A small internal service and a regulated multi-region platform do not need identical architectures.

For every item marked not applicable, somebody should be able to explain why.

1. Organization and Resource Hierarchy §

Organization §

• The GCP organization is under an identity domain controlled by the organization.
• Organization administrators are explicitly identified.
• Organization-level privileges are limited to people who genuinely require them.
• Administrative access uses strong authentication.
• Break-glass access is documented.
• Break-glass credentials are protected and tested.
• Emergency access generates an audit trail.
• Organization policies are managed intentionally rather than left entirely at defaults.
• Resource-location requirements are documented.
• Domain-sharing restrictions have been reviewed.

Folder hierarchy §

• Folder structure represents meaningful administrative or policy boundaries.
• Production and nonproduction are separated.
• Shared infrastructure has an explicit home.
• Security resources have an explicit home.
• Logging resources have an explicit home.
• Networking resources have an explicit home.
• Business-unit folders do not create accidental privilege inheritance.
• Folder-level IAM has been reviewed.

Projects §

• Every production project has a documented owner.
• Every project has a documented purpose.
• Project naming is standardized.
• Labels or tags required for ownership and cost allocation are present.
• Project creation is controlled.
• Default networks are removed or deliberately retained.
• Required APIs are enabled intentionally.
• Unused APIs are reviewed.
• Project deletion permissions are tightly controlled.
• Critical projects use deletion protection or an equivalent process where appropriate.
• Sandbox projects are separated from production trust boundaries.

2. Billing and Cost Controls §

• Billing accounts have limited administrative access.
• Production projects are attached to the correct billing account.
• Budgets exist.
• Budget alerts exist.
• Alerts reach somebody who will act on them.
• Cost allocation by project is possible.
• Cost allocation by environment is possible.
• Cost allocation by team or workload is possible where required.
• Expensive services have expected baseline costs documented.
• Logging volume has been reviewed for cost.
• Network egress has been reviewed for cost.
• Cross-region traffic has been reviewed for cost.
• Autoscaling has maximum limits where runaway scale would be dangerous.
• BigQuery or other usage-based platforms have appropriate cost controls.
• Nonproduction resources shut down or scale down when appropriate.
• Orphaned disks, addresses, snapshots, and other idle resources are reviewed regularly.

3. Identity and Access Management §

Human identity §

• Shared human accounts are prohibited.
• Individual administrators use named identities.
• MFA is enforced according to organizational policy.
• Groups are used instead of repeated direct user grants where practical.
• Group ownership is known.
• Joiner/mover/leaver processes update access promptly.
• Former employees and contractors lose access promptly.
• Production access is narrower than development access.
• Privileged roles are separated from routine user roles.
• Just-in-time privilege is used where practical.
• Privileged access can be reviewed after the fact.

IAM policy §

• Routine use of Owner is avoided.
• Routine use of Editor is avoided.
• Basic roles have been reviewed.
• Predefined roles are preferred when they fit.
• Custom roles have a documented reason.
• Custom roles are reviewed as APIs evolve.
• IAM Conditions are used where they reduce unnecessary privilege.
• IAM Deny policies are considered for high-value guardrails.
• Cross-project grants are documented.
• Cross-organization grants are documented and justified.
• Public principals such as allUsers and allAuthenticatedUsers are reviewed explicitly.
• Permission changes are captured in audit logs.

Service accounts and machine identity §

• Service accounts are named by workload or purpose.
• Service accounts have documented owners.
• One service account is not reused across unrelated workloads without justification.
• Service accounts have least privilege.
• Default service accounts are not granted broad roles casually.
• User-managed service account keys are avoided.
• Existing long-lived keys are inventoried.
• Unnecessary keys are revoked.
• Key creation is restricted where practical.
• Workload Identity Federation is used for external workloads where practical.
• GKE workload identity is used where practical.
• Service account impersonation is used instead of downloadable keys where appropriate.
• CI/CD systems use short-lived credentials where possible.
• Machine identity permissions are reviewed separately from human permissions.

4. Networking §

VPC architecture §

• VPC design is documented.
• Custom-mode networks are used where appropriate.
• Production and nonproduction network trust is separated.
• Shared VPC ownership is clear where Shared VPC is used.
• Host and service projects have documented responsibilities.
• IP address allocation is planned.
• Subnet ranges do not overlap with connected networks.
• GKE secondary ranges have adequate capacity.
• Future growth is considered before address space is exhausted.
• IPv6 requirements have been reviewed.

Routing and egress §

• Routes are understood.
• Custom routes have documented purpose.
• Dynamic routing requirements are understood.
• Cloud Router configuration is documented.
• Internet egress is deliberate.
• Cloud NAT is used where appropriate.
• NAT capacity is sufficient.
• Egress IP requirements are documented.
• Third-party allowlists do not rely on unstable source addresses.
• Egress filtering requirements have been evaluated.
• Cross-region routing assumptions are understood.

Firewall policy §

• Firewall rules follow least access.
• Broad 0.0.0.0/0 ingress rules are explicitly justified.
• Administrative ports are not exposed publicly without a documented reason.
• Firewall rule ownership is known.
• Hierarchical firewall policies are used where they simplify enterprise enforcement.
• Firewall logging is enabled where useful.
• Obsolete firewall rules are removed.
• Network tags and service-account targets are used intentionally.
• Emergency firewall changes are reviewed after the incident.

Private service access §

• Private Google Access requirements are understood.
• Private Service Access ranges are planned where used.
• Private Service Connect is considered where appropriate.
• Private DNS resolution is tested.
• Internal APIs are not accidentally reachable from the internet.

Hybrid connectivity §

• VPN or Interconnect topology is documented.
• Redundancy exists where the business requires it.
• BGP behavior is understood.
• Failover paths are tested.
• On-premises route advertisement is controlled.
• Cloud route advertisement is controlled.
• DNS works during connectivity failover.
• Monitoring exists for tunnels, circuits, and routing sessions.

5. DNS and Certificates §

• Public DNS ownership is known.
• Private DNS ownership is known.
• Zone delegation is documented.
• Administrative access to DNS is restricted.
• DNS changes are versioned or auditable.
• Critical records have appropriate TTLs.
• Split-horizon behavior is documented where used.
• Certificate issuance method is documented.
• Certificate renewal is automated.
• Certificate expiration is monitored.
• Private CA requirements have been reviewed.
• TLS policy matches organizational requirements.
• Obsolete protocols and ciphers are not enabled without justification.

6. Load Balancing and Edge Security §

• The chosen load balancer matches the protocol and scope required.
• Health checks test meaningful application behavior.
• Backend services are not exposed through unintended paths.
• TLS terminates at an intentional boundary.
• HTTP-to-HTTPS behavior is explicit.
• Cloud Armor is evaluated for internet-facing applications.
• WAF rules are tuned rather than blindly enabled.
• Rate limiting exists where abusive traffic can create operational or cost risk.
• DDoS assumptions are documented.
• Origin bypass is prevented where required.
• CDN caching rules do not cache sensitive responses.
• Cache invalidation behavior is understood.

7. Compute Engine §

• VM purpose and owner are documented.
• Instance service accounts use least privilege.
• External IP addresses are avoided when unnecessary.
• Administrative access is controlled.
• OS Login is evaluated.
• IAP-based administration is evaluated.
• Shielded VM features are enabled where compatible.
• Secure Boot requirements are evaluated.
• Confidential VM requirements are evaluated for sensitive workloads.
• Base images are controlled.
• Image provenance is known.
• Startup scripts do not contain secrets.
• Patch process exists.
• Reboot process exists.
• Instance templates are versioned.
• Managed instance groups use health checks.
• Autoscaling limits are appropriate.
• Persistent data is not accidentally stored only on ephemeral disks.
• Snapshot policy exists where needed.
• Restore has been tested.
• Serial console access policy is reviewed.
• Unused instances are removed.

8. Google Kubernetes Engine §

Cluster design §

• Standard versus Autopilot was chosen intentionally.
• Cluster region/zone topology matches availability requirements.
• Private cluster configuration is evaluated.
• Control-plane access is restricted.
• Authorized networks are configured where appropriate.
• Release channel is selected intentionally.
• Upgrade process is documented.
• Maintenance windows/exclusions are configured where needed.
• Node pools have documented purpose.
• Node pool scaling limits are appropriate.

Identity and authorization §

• Workload identity is enabled where practical.
• Kubernetes service accounts map to only the required cloud permissions.
• Human access to the Kubernetes API is group-based where practical.
• Kubernetes RBAC follows least privilege.
• Cluster-admin access is rare and reviewed.
• Break-glass cluster access is documented.
• Namespace boundaries are not mistaken for complete security boundaries.

Workload security §

• Pod security controls are defined.
• Privileged containers are prohibited unless justified.
• Host namespace access is controlled.
• HostPath use is controlled.
• Capabilities are restricted.
• Containers run as non-root where practical.
• Read-only root filesystems are used where practical.
• Resource requests are defined.
• Resource limits are defined where appropriate.
• Network policies are used where application segmentation matters.
• Admission policy is used for important invariants.
• Image source is controlled.
• Vulnerability scanning is part of the image lifecycle.
• Binary Authorization or equivalent provenance controls are evaluated.

Operations §

• Cluster logs are centralized.
• Workload logs are centralized.
• Metrics are collected.
• Control-plane events are observable.
• Node pressure alerts exist.
• Pod crash loops are detected.
• Pending pods are detected.
• Autoscaler failures are detected.
• Persistent volumes are backed up where needed.
• Cluster recovery has been considered.
• Critical manifests exist outside the cluster.
• GitOps state can recreate intended configuration where used.

9. Cloud Run and Serverless §

• Public versus authenticated access is intentional.
• The runtime service account has least privilege.
• Ingress configuration is explicit.
• Egress configuration is explicit.
• VPC connectivity is used only where required.
• Secrets are loaded from a managed secret store.
• Secrets are not embedded in images.
• Minimum instance count matches latency requirements.
• Maximum instance count limits runaway scaling.
• Concurrency is tested for the application.
• CPU and memory are sized from measurements.
• Request timeout is appropriate.
• Application startup behavior is understood.
• Database connection behavior under scale is understood.
• Revision traffic splitting is controlled.
• Rollback procedure is known.
• Logs and metrics are collected.
• Custom domain and TLS behavior are tested.
• Cloud Armor/load balancer integration is used where the threat model requires it.

10. Storage §

Cloud Storage §

• Uniform bucket-level access is enabled where appropriate.
• Public Access Prevention is enabled unless public access is required.
• Public buckets are explicitly approved.
• Bucket location matches residency and availability requirements.
• Object versioning requirements are evaluated.
• Soft-delete behavior is understood.
• Lifecycle rules are configured.
• Retention policy requirements are configured.
• Bucket Lock is evaluated for immutable retention requirements.
• CMEK requirements are reviewed.
• Signed URL use is time-limited and documented.
• Sensitive data classification is known.
• Logging and access analysis are sufficient.
• Cross-project access is documented.

11. Databases §

General §

• Database owner is documented.
• Data classification is documented.
• Public accessibility is avoided unless explicitly required.
• Private connectivity is used where practical.
• Administrative access is limited.
• Application identities have only required database privilege.
• Password authentication is minimized where IAM database authentication is available and suitable.
• Secrets are rotated where passwords still exist.
• Encryption requirements are documented.
• CMEK requirements are evaluated.
• Maintenance strategy is documented.
• Capacity monitoring exists.
• Connection usage is monitored.
• Slow queries can be investigated.
• Backup policy exists.
• Point-in-time recovery requirements are configured.
• Restore has been tested.
• Retention matches business requirements.
• Cross-region recovery requirements are understood.
• Schema migrations have rollback or forward-recovery plans.

Cloud SQL / AlloyDB §

• High availability is enabled where required.
• Read replicas are used for a measured requirement, not by default.
• Failover behavior has been tested.
• Connection pooling is appropriate.
• Serverless scaling cannot exhaust database connections.
• Database flags are reviewed.
• Maintenance windows match operating needs.

12. Messaging and Event-Driven Systems §

• Pub/Sub topic ownership is documented.
• Publisher identities have minimal rights.
• Subscriber identities have minimal rights.
• Dead-letter behavior is configured where appropriate.
• Retry behavior is understood.
• Duplicate delivery is handled safely.
• Ordering requirements are explicit.
• Message retention is appropriate.
• Sensitive payloads are handled according to policy.
• Backlog growth is monitored.
• Poison messages can be diagnosed.
• Consumer scaling cannot overwhelm downstream systems.
• Event schemas are versioned where necessary.

13. Secrets and Key Management §

Secret Manager §

• Secrets are stored in Secret Manager or an approved equivalent.
• Application code contains no production secrets.
• CI/CD files contain no production secrets.
• Container images contain no production secrets.
• Terraform state has been reviewed for sensitive values.
• Secret access uses least privilege.
• Secret versions are used intentionally.
• Rotation requirements are documented.
• Rotation can occur without an extended outage.
• Secret access is auditable.

Cloud KMS §

• Key ownership is documented.
• Key rings follow a deliberate organization.
• Key permissions are separated from data permissions where required.
• Rotation policy is configured.
• Key destruction process is documented.
• Accidental key destruction is protected against.
• CMEK dependencies are inventoried.
• Cross-region recovery implications are understood.
• HSM requirements are evaluated.
• External key-management requirements are evaluated.
• Loss of KMS access is included in recovery planning.

14. Logging, Monitoring, and Observability §

Logging §

• Admin Activity audit logs are retained appropriately.
• Data Access logging is enabled where the threat model or compliance requirements justify it.
• Policy Denied logs are available.
• Critical application logs are centralized.
• Security logs are centralized.
• Log sinks are protected.
• Log storage has limited administrative access.
• Retention periods are documented.
• Sensitive data is not unnecessarily written to logs.
• Credentials and tokens are redacted.
• High-volume logging has cost controls.
• Logs can be exported to a SIEM where required.
• Incident responders know where to find historical evidence.

Monitoring §

• Service health metrics exist.
• Infrastructure metrics exist.
• Application metrics exist.
• User-visible failure is measured where practical.
• SLOs exist for important services.
• Alerting is based on actionable conditions.
• Alerts have owners.
• Alerts have runbooks.
• Alert noise is reviewed.
• Uptime or synthetic checks exist for critical endpoints.
• Prometheus/OpenTelemetry integration is standardized where used.
• Dashboards show useful operating state rather than decorative metrics.

15. Security Operations §

• Security Command Center is configured at the appropriate scope.
• Findings have owners.
• High-severity findings generate an operational response.
• Vulnerability findings are triaged.
• Misconfiguration findings are triaged.
• Threat detections are routed to responders.
• False positives are tuned without blindly suppressing categories.
• Security findings are retained or exported as required.
• SIEM integration is tested where used.
• Security teams can identify asset ownership.
• Security teams can identify the deployment source for a workload.
• Security teams can identify the service account used by a workload.
• Automated remediation is limited to actions that are safe to automate.

16. Application and API Security §

• Authentication method is documented.
• Authorization is enforced server-side.
• API keys are not used as a substitute for user identity.
• OAuth/OIDC configuration is reviewed.
• Token audience and issuer are validated.
• Service-to-service identity is explicit.
• Administrative APIs are more restricted than public APIs.
• Rate limiting is configured where appropriate.
• Input validation exists.
• Security headers are configured where relevant.
• CORS policy is explicit.
• Error responses do not expose unnecessary internal detail.
• API logs do not expose secrets.
• Abuse monitoring exists for public APIs.
• Cloud Armor or an API gateway layer is used where it provides meaningful control.

17. CI/CD and Software Supply Chain §

Source §

• Source repositories require authenticated access appropriate to their sensitivity.
• Protected branches are configured.
• Production changes require review.
• Direct pushes to protected production branches are restricted.
• Commit-signing policy is defined if required.
• Secret scanning runs.
• Dependency scanning runs.
• Static analysis runs where useful.
• IaC scanning runs.

Builds §

• Builds use dedicated identities.
• Build identities use least privilege.
• Build environments do not receive production administrative credentials.
• Private build pools are used where the network model requires them.
• Build dependencies come from controlled sources where required.
• Build provenance is retained.
• Build logs are retained.
• Failed builds cannot promote artifacts.
• Artifact hashes or immutable identifiers are used.

Artifact Registry §

• Repository IAM is limited.
• Production artifacts are separated where appropriate.
• Images are scanned.
• Cleanup policies exist.
• Mutable tags are not the only production identity of an artifact.
• Deployments can identify the exact image digest.
• Promotion copies or references an already-built artifact instead of silently rebuilding different code.

Deployment §

• Deployment identity is separate from build identity where practical.
• Production deploy rights are restricted.
• Approval requirements are documented.
• The approved artifact is the artifact actually deployed.
• Canary or staged rollout is used where appropriate.
• Rollback is tested.
• Deployment health is validated automatically.
• Failed health validation stops promotion.
• Emergency deployment path is documented and auditable.

18. Terraform and OpenTofu §

• Infrastructure definitions live in version control.
• Provider versions are constrained.
• Module versions are constrained.
• Remote state is protected.
• State is not committed to Git.
• State access uses least privilege.
• State recovery/versioning exists.
• Sensitive values in state are understood.
• Production applies do not run from arbitrary developer laptops without an explicit policy decision.
• Pull requests run formatting and validation.
• Pull requests run a plan.
• Plans are reviewed.
• Policy checks run where required.
• Applies correspond to reviewed source.
• Destructive changes receive special attention.
• prevent_destroy or other safeguards are used where useful, not as a substitute for process.
• Imports are documented.
• Drift is detected.
• Manual Console changes are discouraged or reconciled.
• Modules have clear ownership.
• Modules have documentation.
• Modules do not hide dangerous defaults.

19. Ansible and Configuration Management §

• Inventories are generated or maintained from an authoritative source.
• Production inventory is protected.
• SSH host-key verification policy is explicit.
• Privilege escalation is controlled.
• Secrets use an approved secret system.
• Plaintext secrets are not committed to Git.
• Roles are versioned.
• Collections are versioned.
• Playbooks are idempotent where practical.
• Check mode is used where useful.
• Production changes are tested elsewhere first.
• Configuration drift can be detected.
• Configuration rollback or rebuild is possible.
• Ad hoc administrative commands are minimized.
• Emergency ad hoc changes are reconciled into managed configuration.

20. Policy and Governance §

• Organization policies implement important guardrails.
• Policies are documented.
• Policies are tested before broad enforcement.
• Dry-run modes are used where supported.
• Exception process exists.
• Exceptions have owners.
• Exceptions have expiration/review dates.
• Policy changes are reviewed.
• Policy changes are auditable.
• Mandatory regions are enforced where required.
• Public resource creation is restricted where required.
• Service account key creation is restricted where required.
• External IP usage is restricted where required.
• Domain sharing is restricted where required.
• Encryption policy is enforced where required.
• Compliance mappings distinguish technical evidence from actual legal compliance.

21. Backups and Disaster Recovery §

• Every stateful production system has an explicit recovery requirement.
• RPO is documented.
• RTO is documented.
• Backup frequency supports RPO.
• Restore process supports RTO.
• Backups exist in a failure domain appropriate to the threat model.
• Backup IAM is narrower than normal production administration where practical.
• Backups are encrypted.
• Backup retention is documented.
• Backup deletion is protected where appropriate.
• Database point-in-time recovery is configured where required.
• Object versioning/retention is configured where required.
• Kubernetes state recovery is defined.
• Terraform/OpenTofu state is recoverable.
• KMS dependencies are included in recovery planning.
• DNS dependencies are included in recovery planning.
• Secrets are included in recovery planning.
• Recovery procedures are documented.
• Recovery procedures are tested.
• Restore tests verify application usability, not merely that files exist.
• Disaster-recovery exercises produce follow-up work.
• Recovery ownership is known during an incident.

22. Reliability and Availability §

• Availability requirement is documented.
• Single points of failure are known.
• Regional dependencies are known.
• Zonal dependencies are known.
• Multi-region design is used only where the business requires its complexity.
• Health checks are meaningful.
• Autoscaling is tested.
• Capacity limits are understood.
• Quotas are monitored before exhaustion.
• Dependency failure behavior is defined.
• Timeouts are configured.
• Retries are bounded.
• Retry storms are considered.
• Circuit breaking is used where appropriate.
• Graceful degradation is designed where appropriate.
• Maintenance behavior is understood.
• Failover is tested.
• Failback is tested where relevant.

23. Incident Response §

• Incident severity levels are defined.
• On-call ownership is defined.
• Escalation paths exist.
• Security escalation paths exist.
• Incident communication channels are defined.
• Cloud audit evidence can be preserved.
• Application logs can be preserved.
• VM/container evidence procedures are documented where necessary.
• Compromised service accounts can be disabled quickly.
• Compromised keys can be revoked quickly.
• Workloads can be isolated from the network.
• Traffic can be redirected or blocked.
• Malicious deployments can be rolled back.
• Break-glass access works.
• Emergency changes are logged.
• Post-incident reviews are blameless and technically useful.
• Corrective actions receive owners.
• Corrective actions are tracked to completion.

24. Data Protection and Privacy §

• Data classes are defined.
• Sensitive data locations are known.
• Unnecessary sensitive data is not collected.
• Data residency requirements are documented.
• Retention requirements are documented.
• Deletion requirements are documented.
• Encryption requirements are documented.
• Access to sensitive data is auditable.
• Sensitive data is not unnecessarily copied into logs.
• Sensitive data is not unnecessarily copied into lower environments.
• Production data used in testing is masked or otherwise controlled.
• Data export paths are understood.
• Service perimeter requirements are evaluated.
• Sensitive Data Protection/DLP capabilities are evaluated where relevant.

25. Documentation and Ownership §

• Every production service has an owner.
• Every production project has an owner.
• Every critical alert has an owner.
• Every critical runbook has an owner.
• Architecture is documented.
• Major trust boundaries are documented.
• Data flows are documented.
• External dependencies are documented.
• Known technical debt is documented.
• Recovery procedures are documented.
• Deployment procedures are documented.
• Emergency procedures are documented.
• Documentation is versioned where practical.
• Documentation is reviewed after major architectural change.
• The system can be understood by somebody other than the person who built it.

26. Final Go-Live Review §

Before declaring the environment production-ready:

• Architecture review is complete.
• Security review is complete.
• IAM review is complete.
• Network review is complete.
• Cost review is complete.
• Production data handling is approved.
• Monitoring is live.
• Alerts have been tested.
• Logs are reaching the expected destination.
• CI/CD production path has been tested.
• Rollback has been tested.
• Backup has completed successfully.
• Restore has been tested.
• DNS has been tested.
• TLS has been tested.
• Failure of critical dependencies has been considered.
• Quotas have been reviewed.
• Support/escalation contacts are known.
• Runbooks exist for likely incidents.
• Break-glass access has been tested.
• Somebody other than the primary implementer has reviewed the environment.
• The team can explain what happens if the primary region disappears.
• The team can explain what happens if the deployment identity is compromised.
• The team can explain what happens if the database is accidentally deleted.
• The team can explain how to rebuild the infrastructure from source.
• The team can explain where the authoritative configuration lives.

The Last Question §

After every checkbox, architecture diagram, approval, test, and deployment, one question remains:

If this fails tonight, do we know enough about the system to understand what happened, contain the damage, and recover it?

If the answer is no, the work is not finished.