Chapter 34
Incident Response in GCP
Scope. This chapter is a procedure. Chapters 16, 17, and 18 own detection and are not re-explained; Chapter 29 owns compliance evidence, while §34.12 owns evidence gathered under time pressure. Each section is an ordered set of steps with the command at each step, and — the part that matters most — which steps are irreversible and which destroy evidence if taken too early. Where this book's ordering departs from Google's published guidance, it says so and gives the reason. Prerequisites. Chapter 16 (Security Command Center), Chapter 17 (logging and audit), Chapter 3 (IAM), Chapter 8 (Compute Engine), Chapter 5 (networking). Verified against. Google Cloud console and API surface as of 2026-09, Cloud SDK 583.0.0; every command resolved with
--help; see sources at end.
Incident response in a cloud environment differs from the on-premises version in one decisive way: the control plane is an API, so containment is a few commands rather than a walk to a rack — and so is destruction of the evidence you needed. The commands that stop an attack and the commands that erase the record of it are adjacent in the documentation and often in the same runbook.
So the ordering is the content of this chapter. Nearly every section below is a sequence, and its value is in which step comes first. Snapshot before stop; isolate before investigate; preserve logs before the retention window closes; and never take the one action that looks like containment and destroys the most volatile evidence you have.
One finding underlies §34.5 and it changes what containment means. Service account access tokens cannot be revoked. The SDK states it plainly: gcloud auth revoke on a service account "does not revoke the service account token on the server because service account tokens are not revocable". Disabling the account, deleting its keys, and removing its roles all leave already-issued tokens working until they expire. Containment therefore has a floor measured in time, not in commands, and every plan in this chapter is built around that.
Assume the responder is tired and under pressure. That is why irreversibility is flagged in every table rather than discussed once.
34.1 Incident Preparation §
Everything in this chapter is faster if four things exist beforehand, and none of them can be created during an incident.
The four:
| Prerequisite | Why it cannot wait |
|---|---|
| A forensics project | Snapshots need somewhere isolated to go, with its own IAM |
| Break-glass access | The responder may not hold the roles the response needs |
| Log export already running | Exports are forward-looking (§29.10) |
| A tested runbook | The first execution of a procedure should not be during an incident |
The forensics project holds nothing but evidence. It is separate from the security project (§2.21) because its access population is different: responders read it during an incident, and nobody writes to it routinely.
Break-glass is the interesting design problem. A standing account with broad permissions is a permanent risk; an account with no permissions is useless when needed. Privileged Access Manager resolves this by making elevation a time-bounded, approved, logged grant rather than a standing binding:
gcloud pam entitlements create ENTITLEMENT_ID \
--location=global --organization=123456789012 --entitlement-file=ent.yaml
The entitlement is defined in a file, not in flags — there is no --max-duration or --approver flag, and looking for one wastes time. Privileged Access Manager's core is generally available; multi-party approval is Preview and must not be the only thing standing between an attacker and production.
The gcp-breakglass@rickcollette.domain group is the emergency path and its use must page someone. An emergency access mechanism nobody is alerted about is an unmonitored back door.
There is no gcloud command to open a support case. The command group does not exist in either the GA or beta tree. A P1 to Google Cloud Customer Care is raised through the console or the API, and the responder needs to know that before they need it.
Pitfall. A runbook that references a project, group, or entitlement that does not exist is discovered at 3 a.m. Test the response path on a schedule — including the break-glass grant — or the preparation is documentation rather than capability.
34.2 Detection §
Chapter 16 owns Security Command Center and Chapters 17 and 18 own logs and alerting. This section owns only the handoff: the moment a finding becomes an incident.
A finding is not an incident and most findings never become one. The transition is a decision, and it needs a defined trigger rather than someone's judgment at the time.
Three trigger classes, and they behave differently:
| Source | What it indicates | Latency |
|---|---|---|
| Threat findings (Event, Container, VM Threat Detection) | Something is happening now (§16.6) | Minutes |
| Log-based alerts on control-plane events | A specific action was taken (§18.15) | Near real-time |
| Misconfiguration findings | A weakness exists, not that it is being used | Hours |
Only the first two should page. A misconfiguration finding is a ticket; treating it as an incident trains responders to ignore the channel, which is how a real threat finding gets missed.
The finding carries what triage needs, and §34.3 uses it: the affected resource, the principal, the source IP, and the severity. Export findings continuously (§16.15) so the response process reads from a stream rather than from the console.
Pitfall. Detection depends on the detectors being enabled, and disabling one produces no finding and no alert — the absence of threat findings looks identical to safety (§16.6). Alert on enablement changes to the detection services themselves, or the first sign of a sophisticated attacker is silence.
34.3 Triage §
Triage answers three questions in order, and the order matters because the answer to each changes the effort spent on the next.
1. Is it real? A large fraction of findings are legitimate activity — a penetration test, a new deployment tool, an administrator working from a new location. Check the principal against known automation before anything else.
2. What is the blast radius? What can the compromised identity or workload actually reach:
gcloud asset analyze-iam-policy-longrunning \
--organization=123456789012 \
--gcs-output-path=gs://rc-saas-forensics-01/triage/identity.json
3. Is it still happening? An active intrusion is contained first and investigated second; a historical one is investigated first, because containment of something that already stopped destroys evidence for no benefit.
Severity drives the pace, not the procedure. The steps in §34.4 onward are the same at every severity; what changes is whether they run now or during business hours.
Record the decision and the time. §29.10 needs the join between a finding and an incident record, and the only moment anyone will reliably capture it is when it is made.
Pitfall. Question two is the one that gets skipped, because the affected resource is visible and the identity's reach is not. An incident scoped to the compromised VM, when the VM's service account had project-wide permissions, is scoped wrongly — and the scope decides everything that follows.
34.4 Containment §
Containment stops the damage spreading. Its whole difficulty is that the fastest options are the most destructive, and the most complete option is not available at all.
The containment options, ranked by speed and by what they cost:
| Action | Speed | Evidence cost | Reversible |
|---|---|---|---|
| Network isolation (§34.11) | Seconds | None | Yes |
| IAM deny policy on the principal | ~2 min, up to 7 | None | Yes |
| Disable a service account | Minutes | None | Yes |
| Remove a role binding | Minutes | None | Yes |
| Remove from a group | Minutes to hours | None | Yes |
| Stop an instance | Immediate | Destroys memory and serial output | Yes |
| Delete a resource | Immediate | Destroys everything | No |
Prefer network isolation first. It stops exfiltration and lateral movement in seconds, costs no evidence, and is trivially reversible — which makes it the right first action even when you are not yet sure.
IAM deny policies are the fastest identity-side control because they are evaluated before allow policies (§3.17), so a deny takes effect without unpicking every grant. Google documents roughly two minutes typical propagation and up to seven in the worst case.
Group removal is the slowest lever, documented as taking several minutes and potentially hours. Never rely on it for containment; use a deny policy and clean up group membership afterward.
None of these invalidate an existing token (§34.5), so every one of them has a residual window.
Pitfall. Stopping an instance feels like containment and is mostly evidence destruction: it loses guest memory and the serial port buffer. Isolate the network instead — the instance is then just as harmless and still holds everything you need.
34.5 Credential Revocation §
This section exists because the intuitive model is wrong, and acting on the intuitive model leaves an attacker with working access for up to an hour after you believe you have locked them out.
Service account access tokens are not revocable. The SDK is unambiguous: gcloud auth revoke on a service account "does not revoke the service account token on the server because service account tokens are not revocable", and when used with a service account "this command has only a local effect".
What each action actually does:
| Action | Stops new tokens | Invalidates existing tokens |
|---|---|---|
gcloud iam service-accounts keys delete | Yes, for that key | No |
gcloud iam service-accounts disable | Yes | No |
| Removing role bindings | No — token still issued | Authorization fails on next call |
| IAM deny policy | No | Authorization fails on next call |
| Deleting the service account | Yes | Eventually |
The two rightmost columns are the important distinction. Disabling an account stops it obtaining new tokens and does nothing about the one an attacker already holds. Removing authorization does not invalidate the token either — but it makes the token useless, because each API call is authorized at call time. That is the actual containment mechanism: remove the permissions, not the credential.
Plan for the residual window. An access token's default lifetime is one hour. If constraints/iam.allowServiceAccountCredentialLifetimeExtension is set in your organization, it can be up to twelve. Check which applies to you before an incident, not during one:
gcloud org-policies describe iam.allowServiceAccountCredentialLifetimeExtension \
--organization=123456789012 --effective
Do all of it, in this order: deny policy or role removal first, because it takes effect on the next call; then disable the account; then delete keys; then rotate whatever the credential protected.
Pitfall. A responder who deletes the key, sees the deletion succeed, and closes the incident has done the one thing that provably does not stop the attacker. Google's own documentation states that deleting a key does not revoke short-lived credentials issued from it — and the credential in an attacker's hands is a short-lived credential.
34.6 Service Account Compromise §
The procedure, in order, with the reason each step is where it is.
- Scope the identity's reach before touching it — analysis is harder once permissions are gone (§34.3).
- Remove authorization, which is what actually works (§34.5).
- Disable the account, stopping new tokens.
- List and delete user-managed keys, so it cannot be re-enabled into a working state:
gcloud iam service-accounts keys list \
--iam-account=sa-app-prod@rc-saas-prod-app-01.iam.gserviceaccount.com \
--managed-by=user
- Find what it did, filtering the audit log by principal:
SA=sa-app-prod@rc-saas-prod-app-01.iam.gserviceaccount.com
gcloud logging read \
"protoPayload.authenticationInfo.principalEmail=\"$SA\"" \
--project=rc-saas-prod-app-01 --order=asc --limit=1000 --format=json
- Rotate everything it could read — secrets, keys, and any credential it had access to (Chapter 13, Chapter 14).
The audit log fields that matter are protoPayload.authenticationInfo.principalEmail for who, protoPayload.requestMetadata.callerIp for where from, protoPayload.methodName for what, and protoPayload.authorizationInfo for whether it was permitted.
serviceAccountDelegationInfo is the field that reveals impersonation chains. If the account was reached by impersonation rather than by a key, that field names the human or service that did it — and the incident is larger than one account.
Pitfall. Step 6 is the one that gets deferred, because rotation is disruptive and the account is already disabled. But an attacker with an hour of valid token had an hour to read secrets, and every one of them is compromised whether or not it was accessed. Rotate on the assumption of access, not on evidence of it.
34.7 VM Compromise §
The order here is the sharpest example in the chapter, and this book departs from Google's published sequence deliberately. Security Command Center's own remediation guidance lists stopping the instance before backing it up for forensic analysis; that loses volatile evidence, and the reason to reorder is that a network-isolated instance is already harmless.
The procedure:
| # | Step | Command | Irreversible | Destroys evidence if done earlier |
|---|---|---|---|---|
| 1 | Record current state | gcloud compute instances describe | No | — |
| 2 | Capture serial output | gcloud compute instances get-serial-port-output | No | Lost at stop |
| 3 | Isolate the network | §34.11 | No | — |
| 4 | Snapshot every disk, while running | gcloud compute disks snapshot | No | — |
| 5 | Protect the disks | gcloud compute instances set-disk-auto-delete --no-auto-delete | No | — |
| 6 | Suspend or stop | gcloud compute instances suspend | No | Memory lost at stop |
| 7 | Delete, keeping disks | gcloud compute instances delete --keep-disks=all | Yes | — |
Snapshots can be taken from disks attached to a running instance, which is what makes step 4 possible before step 6 and is the single most useful fact in this section.
The serial port buffer is a rolling one megabyte and it is lost when the instance stops. Capture it before anything else changes the machine's state.
Never use gcloud compute instances reset. It performs a hard reset and does not cleanly shut down the guest — it is the equivalent of pulling the power, and it destroys memory without even the courtesy of a clean stop.
gcloud compute instances suspend saves guest state to storage, which sounds like memory acquisition and is not: there is no documented customer path to read or export that saved state. Compute Engine has no first-party volatile memory acquisition, and a plan that assumes one is a plan with a gap.
Step 5 matters more than it looks. A boot disk with auto-delete set is destroyed with the instance, so deleting a compromised VM without clearing that bit destroys the disk image the investigation needs.
Pitfall. Snapshots inherit the source disk's encryption. A snapshot of a CMEK-encrypted disk is unreadable without the key, so the forensics project needs access to that key (§14.8) — which is a grant to arrange in §34.1, not during the incident.
34.8 Container Compromise §
A compromised container is two problems: the workload and the node it ran on. Solving only the first is the common mistake.
Contain the workload first, because it is fast and reversible:
- Cordon the node so nothing new schedules there:
kubectl cordon NODE. - Apply a deny-all NetworkPolicy to the pod's namespace, which isolates it without deleting it.
- Do not delete the pod. A deployment recreates it, and you have destroyed the running state and gained nothing.
Then treat the node as §34.7. A container escape means the node's operating system is compromised, and the node is a Compute Engine instance with a service account — which in a well-built cluster is sa-gke-node-<cluster> holding four roles (§9.31), and in a badly-built one is the default compute account with everything.
The node's Workload Identity configuration decides how far it goes. With Workload Identity enabled (§4.12, §9.11) a compromised pod gets only its own Kubernetes service account's mapped identity. Without it, the pod can reach the node's service account through the metadata server, and the blast radius is the node's permissions.
kubectl drain is for maintenance, not incidents. It evicts pods to other nodes, which is exactly wrong when the question is whether the workload is malicious.
Preserve the container image by digest. The running image is evidence, and a tag will be repointed (§24.5). Record the digest from the pod spec before anything is rescheduled.
Pitfall. Cordoning a node stops new scheduling and leaves the compromised pod running and networked. The NetworkPolicy is the containment; the cordon only stops the problem spreading to new workloads on the same node.
34.9 Data Exposure §
The first question is what was accessed, and it usually has no answer — which is the finding.
Data Access audit logs are the only record of a read, and they are off by default for every service except BigQuery (§33.11). An exposure investigation in an estate without them can establish what was reachable and never what was taken.
So the procedure is:
- Establish reachability from the IAM policy and the network path — what the compromised identity could have read.
- Establish access, if logs allow it, from Data Access logs filtered to the principal and the period.
- Classify what was in scope using Sensitive Data Protection's discovery output (Chapter 15), which tells you whether the exposed data was sensitive without reading it again.
- Close the path, then decide about notification.
Cloud Storage does not log object reads without Data Access logging enabled. Object-level access is invisible; only the bucket's configuration history is available. That is the gap most exposure investigations run into.
Notification obligations have legal deadlines (§30.4, §30.3), and they generally start when the determination is made rather than when the incident began. Record the determination time (§29.10).
Pitfall. Enabling Data Access logs during the investigation gives you a record from that moment forward and creates the impression of coverage. It answers nothing about the period under investigation, and a report that presents post-incident logs without saying so is misleading.
34.10 IAM Compromise §
An attacker who reaches IAM does not need to persist in any resource — the policy is the persistence.
Find the change:
gcloud logging read \
'protoPayload.methodName="SetIamPolicy"' \
--project=rc-saas-prod-app-01 --order=asc --limit=100 --format=json
Recover the prior state from Cloud Asset Inventory, which is the only place the previous policy exists:
gcloud asset get-history \
--organization=123456789012 \
--asset-names=//cloudresourcemanager.googleapis.com/projects/rc-saas-prod-app-01 \
--content-type=iam-policy \
--start-time=2026-08-20T00:00:00Z
The 35-day limit applies (§29.5). A policy change older than that has no recoverable prior state unless §29.2's scheduled export captured it — which is the concrete reason that export exists.
Restore with an etag, not a blind write. gcloud projects set-iam-policy is authoritative and will discard any legitimate change made since the compromise. Using the etag from the policy you are replacing makes the write fail rather than silently clobber a concurrent change.
Look for what else the attacker added, because an IAM compromise is rarely a single binding: a new service account, a key on an existing account, a workload identity pool provider with a permissive attribute condition (§26.4), or an added auditConfig exemption hiding subsequent activity (§31.10).
Pitfall. Restoring the policy removes the attacker's binding and does not remove anything they created with it. A service account created during the window survives the policy restore, holds its own bindings, and is the persistence mechanism the restore was supposed to remove.
34.11 Network Isolation §
Network isolation is the best first containment action in this chapter: seconds to apply, no evidence cost, fully reversible.
Use a network firewall policy rule at a high priority, targeting the instance's service account rather than a network tag — the book's standing rule (§5.13):
gcloud compute network-firewall-policies rules create 100 \
--firewall-policy=fp-prod-global --global-firewall-policy \
--direction=EGRESS --action=deny --layer4-configs=tcp,udp,icmp \
--dest-ip-ranges=0.0.0.0/0 \
--target-service-accounts=sa-app-prod@rc-saas-prod-app-01.iam.gserviceaccount.com
--layer4-configs takes a protocol name alone to mean every port — tcp covers tcp:1-65535 — so tcp,udp,icmp is the practical all-protocols deny. There is no documented all value, and a rule that omits a protocol leaves it permitted.
Deny egress before ingress. Egress stops exfiltration and command-and-control; ingress stops nothing the attacker is currently doing if they already have a shell.
Targeting by service account isolates every instance running as that identity, which is usually right during an incident — if one instance in a managed instance group is compromised, the others are running the same image with the same credentials.
Removing the external IP is not isolation. gcloud compute instances delete-access-config removes inbound reachability and leaves egress working through Cloud NAT, which is the direction that matters.
Keep the responder's own path open. A deny-all egress rule breaks the IAP tunnel and the logging agent along with the attacker's connection. Either scope the rule to leave 35.235.240.0/20 reachable, or accept that the instance is now investigable only through its disks.
Pitfall. Rules in a network firewall policy are ordered by priority, and a permissive rule at a lower number wins. An isolation rule at priority 100 does nothing if an allow-all rule sits at priority 50 — which §33.9 says is more common than it should be. Check the effective rules, not the rule you just added.
34.12 Evidence Collection §
§29.2 owns compliance evidence with a schedule and a retention lock. This section owns evidence gathered in an hour, and the difference is that here you cannot go back for what you did not take.
Collect in this order, most volatile first:
| # | Evidence | Volatility |
|---|---|---|
| 1 | Serial port output | Lost at instance stop |
| 2 | Running process and network state, if reachable | Lost at stop |
| 3 | Disk snapshots | Lost only on disk deletion |
| 4 | Audit logs for the period | Lost at retention expiry (§34.13) |
| 5 | IAM policy and asset state | Lost after 35 days (§29.5) |
| 6 | Container image by digest | Lost if the registry is cleaned up |
Write snapshots into the forensics project, not the compromised one. An attacker with the permissions that produced this incident can plausibly delete evidence stored where they still have access.
Establish a chain of custody as you go. Record the command, the operator, the time, and the resulting object's hash. §29.2's create-only IAM pattern applies here too: a responder should be able to add evidence and not to alter it.
The forensics project needs the CMEK key for any encrypted snapshot it receives (§34.7). This is the single most common reason a snapshot taken correctly turns out to be unreadable.
Pitfall. Evidence collection competes with containment for the same minutes, and containment usually wins the argument. The resolution is that steps 1 and 3 above take seconds and go before containment; everything else can follow it. Making that split explicit in the runbook prevents the argument happening during the incident.
34.13 Log Preservation §
Logs age out on a schedule that does not care about your investigation, and the preservation action has to be taken before the window closes.
Know the three retention floors (§17.13):
| Bucket | Retention |
|---|---|
_Required | 400 days, fixed, cannot be changed |
_Default | 30 days |
| Custom | 1 to 3,650 days |
Admin Activity logs go to _Required and are safe for 400 days. Data Access logs go to _Default unless routed elsewhere, which means thirty days — shorter than most investigations take to begin.
The emergency action is a sink to a long-retention destination, created now:
gcloud logging sinks create sink-incident-2026-08 \
storage.googleapis.com/projects/rc-saas-forensics-01/buckets/rc-saas-forensics-logs-01 \
--log-filter='timestamp>="2026-08-01T00:00:00Z"' \
--project=rc-saas-prod-app-01
A sink is forward-looking and does not export logs already written. For those, read them out within the retention window and store the result — which is why §34.12 lists log extraction as a collection step rather than assuming the logs will be there later.
Lock the destination. A locked log bucket cannot have its retention shortened, and a Cloud Storage bucket with a retention lock (§29.2) cannot have objects deleted early. Locking is irreversible, which is the point.
Cloud Storage object holds are the finer instrument. gcloud storage objects update --event-based-hold places a hold on a specific object without changing bucket-wide policy — the legal-hold mechanism for the specific evidence in an investigation.
Pitfall. Locking a log bucket does not prevent the sink that fills it from being deleted (§17.20). A privileged insider stops the flow rather than altering the record, and the locked bucket faithfully preserves everything up to the moment the sink was removed. Alert on sink deletion.
34.14 Recovery §
Recovery restores service. Its risk is restoring the vulnerability along with the data.
The sequence:
- Establish when the compromise began, from §34.6's audit log analysis. Everything after that time is suspect.
- Restore from a backup that predates it, which requires backups with enough retention to reach back that far (§29.9).
- Patch the entry path before restoring, or the restored system is compromised by the same route within hours.
- Rebuild rather than clean where you can. A restored image from a known-good pipeline (§8.33) is trustworthy in a way a cleaned host is not.
- Rotate every credential the compromised system held, on the assumption of access (§34.6).
Immutable infrastructure makes this dramatically easier, which is the security argument for it: a VM built from a golden image and configured by a playbook (§28.12) can be replaced rather than repaired, and the replacement is provably the intended configuration.
Backups in the same failure domain may be compromised too (§33.19). A backup in the same project, reachable by the same credentials, is as suspect as the source — which is the argument for backup vaults with enforced retention that a compromised project cannot shorten.
Pitfall. Recovery is the point at which the incident stops being urgent, so the remaining steps — rotation, the postmortem, the control that would have prevented it — compete with normal work and lose. Schedule them before the incident is declared closed, because the declaration is what removes the priority.
34.15 Postmortems §
A postmortem's purpose is to change the system, and the format that achieves that is well established.
Blameless, in the specific sense that the analysis assumes people acted reasonably given what they knew, and asks why the reasonable action produced a bad outcome. This is not politeness — a process that assigns blame gets slower and less accurate reporting, which makes the next incident worse.
The sections that matter:
| Section | The question it answers |
|---|---|
| Timeline | What happened, with times, from the evidence |
| Detection | How it was found, and how it could have been found sooner |
| Response | What was done, and which steps helped |
| Root cause | Why the condition existed, not who created it |
| Contributing factors | What made it worse or slower |
| Actions | Specific, owned, dated changes |
Detection latency is the most useful single metric. The gap between when an attacker acted and when anyone noticed is the number that most directly predicts the next incident's severity, and it is improvable in ways "prevent the vulnerability" is not.
Every action item should be a control, not a reminder. "Be careful with firewall rules" is not an action; a constraint (§31.7), a plan validation rule (§26.25), or an alert (§18.15) is. Chapter 33's closing argument applies: the fix for a recurring failure is a mechanism, because the incentive that produced it has not changed.
Pitfall. A postmortem that produces twenty action items produces none, because nothing with twenty owners gets done. Three items with named owners and dates beat a complete list, and the complete list belongs in the risk register rather than in the action plan.
34.16 Automation §
Automated response is worth building for the narrow set of incidents where the correct action is unambiguous and the cost of being wrong is low.
The mechanism is Security Command Center to Pub/Sub to a responder service. §16.15 owns the export configuration; this section owns the responder:
- A continuous export publishes findings to a Pub/Sub topic.
- A Cloud Run service subscribes, authenticated by OIDC (§32.7).
- It matches the finding category against a small allowlist of automatable responses.
- It acts, and it records what it did.
Automate containment, never remediation. Isolating a network, disabling a service account, and revoking a session are reversible and cheap to get wrong. Deleting a resource, changing an IAM policy, or restarting a workload are not.
The responder's own identity is a target. It needs permission to contain, which means permission to disrupt — so it should hold narrowly scoped roles, run with its own service account, and be as carefully reviewed as anything in Chapter 32. An automated responder with broad permissions is a self-inflicted lateral movement path.
Google's published automated-remediation samples are an archived repository, not a product. Treat them as reference code to read rather than as something to deploy, and do not present them to a stakeholder as a supported Google capability. Security Command Center's own playbooks are an Enterprise-tier feature, and that tier is scheduled to shut down (§16.x) — so an automation strategy built on it has an expiry date.
Every automated action needs a human notification. Automation that contains silently produces an outage nobody can explain, and the explanation is the most valuable thing the automation can emit.
Pitfall. An automated responder that acts on a false positive causes an outage, and the response is to disable it. It is then off when a real incident arrives. Rate-limit it, allowlist narrowly, run it in report-only mode for a full cycle first — and give it an exit date from that mode, or §33.20 applies.
Chapter Summary §
- Service account access tokens are not revocable — the SDK says so plainly. Disabling the account or deleting its keys does not invalidate a token an attacker already holds.
- The actual containment mechanism is removing authorization, because each API call is authorized at call time; the token keeps existing and stops working.
- Plan for a residual window of one hour, or up to twelve if
constraints/iam.allowServiceAccountCredentialLifetimeExtensionis set. Check which applies before an incident. - IAM deny policies are the fastest identity-side lever (~2 minutes, up to 7); group removal is the slowest (minutes to hours) and must never be relied on for containment.
- Network isolation is the best first action: seconds, no evidence cost, fully reversible.
- Snapshots can be taken from disks attached to a running instance, which is why snapshot comes before stop.
- This book departs from Security Command Center's published order, which lists stopping the instance before backing it up — a network-isolated instance is already harmless, and stopping it destroys memory and the serial buffer.
- The serial port buffer is a rolling one megabyte, lost at stop; capture it first.
- Never use
gcloud compute instances reset— it is a hard reset with no clean shutdown. suspendsaves guest state but there is no documented customer path to read it; Compute Engine has no first-party memory acquisition.- Clear boot disk auto-delete before deleting a compromised instance, or the disk goes with it.
- Snapshots inherit the source's encryption, so the forensics project needs the CMEK key — arranged in advance, not during the incident.
- For a compromised container, apply a NetworkPolicy; cordoning only stops new scheduling, and draining is exactly wrong.
- Cloud Storage does not log object reads without Data Access logging, so most exposure investigations can establish reachability and never access.
- Enabling Data Access logs during an investigation answers nothing about the period under investigation.
- Cloud Asset Inventory is the only source of a prior IAM policy, and it reaches back 35 days.
- Restore an IAM policy with an etag; an authoritative write discards concurrent legitimate changes.
- An IAM restore does not remove what the attacker created with the access — a new service account survives it.
- Deny egress before ingress, target by service account, and check effective rules: a lower-priority allow-all beats your isolation rule.
- Data Access logs go to
_Defaultand last thirty days unless routed; a sink is forward-looking and will not capture what is already written. - Locking a log bucket does not prevent the sink that fills it from being deleted — alert on sink deletion.
- There is no
gcloudcommand to open a support case, in the GA or beta tree. - Privileged Access Manager entitlements are defined in a file; multi-party approval is Preview.
- Automate containment, never remediation, and notify a human on every automated action.
- Google's automated-remediation samples are an archived repository, not a product.
Security Checklist §
| Control | Why it matters | How to verify (CLI + Console) |
|---|---|---|
| Forensics project exists with its own IAM | Evidence must not live where the attacker has access | gcloud projects describe rc-saas-forensics-01 |
| Forensics project granted the CMEK keys | An encrypted snapshot is unreadable without them | Key IAM policy (§14.8) |
| Break-glass path exists and is tested | The first execution should not be during an incident | gcloud pam entitlements list --location=global --organization=ORG_ID |
| Break-glass use pages someone | An unmonitored emergency path is a back door | Alert policy on the entitlement's grants |
| Credential lifetime constraint checked | It decides whether the residual window is 1 hour or 12 | gcloud org-policies describe iam.allowServiceAccountCredentialLifetimeExtension --organization=ORG_ID --effective |
| Runbook removes authorization before disabling | Disabling does not invalidate an existing token | Runbook step order |
| Runbook snapshots before stopping | Stopping destroys memory and serial output | Runbook step order |
Runbook never contains instances reset | It is a hard reset with no clean shutdown | grep -r reset in the runbook |
| Detection service enablement is alerted on | Disabling a detector produces no finding | Alert on gcloud scc manage services update in Admin Activity |
| Data Access logs enabled before an incident | They cannot establish access retroactively | gcloud projects get-iam-policy PROJECT_ID --format='yaml(auditConfigs)' |
| Log sink deletion alerted on | Locking the bucket does not protect the sink | Alert policy over Admin Activity (§17.20) |
| Isolation rule priority beats existing allow rules | A lower-priority allow-all wins | gcloud compute network-firewall-policies get-effective-firewalls |
| Isolation preserves the responder's IAP path | Otherwise the instance becomes uninvestigable live | Rule scope versus 35.235.240.0/20 |
| Responders can add evidence and not alter it | Chain of custody as IAM (§29.2) | Forensics bucket IAM |
| Backups reach back further than detection latency | You must restore from before the compromise began | Backup retention versus measured detection latency |
| Automated responder holds narrow, reviewed roles | It has permission to disrupt by design | The responder's service account bindings |
| Automated responder notifies a human every time | Silent containment is an unexplained outage | Notification channel configuration (§18.4) |
| Report-only mode for automation has an exit date | Otherwise §33.20 applies | The exception register |
Sources §
- https://cloud.google.com/iam/docs/keys-create-delete — that deleting a key does not revoke short-lived credentials issued from it (last validated 2026-09-04)
- https://cloud.google.com/iam/docs/service-account-creds — access token lifetime and the credential types (last validated 2026-09-04)
- https://cloud.google.com/iam/docs/deny-overview — deny policy evaluation order and propagation time (last validated 2026-09-04)
- https://cloud.google.com/compute/docs/instances/suspend-stop-reset-instances-overview — what stop, suspend, and reset each do to instance state (last validated 2026-09-04)
- https://cloud.google.com/compute/docs/troubleshooting/viewing-serial-port-output — the serial port buffer and its volatility (last validated 2026-09-04)
- https://cloud.google.com/compute/docs/instances/deleting-instance — boot disk auto-delete behavior on deletion (last validated 2026-09-04)
- https://cloud.google.com/compute/docs/instances/suspend-resume-instance — what suspend preserves and its restrictions (last validated 2026-09-04)
- https://cloud.google.com/storage/docs/object-holds — event-based and temporary holds as the legal-hold mechanism (last validated 2026-09-04)
- https://cloud.google.com/storage/docs/audit-logging — that object reads are not logged without Data Access logging (last validated 2026-09-04)
- https://cloud.google.com/iam/docs/access-change-propagation — how long an IAM change takes to take effect (last validated 2026-09-04)
- https://cloud.google.com/docs/authentication/token-types — access token lifetime and revocability (last validated 2026-09-04)
- https://cloud.google.com/logging/docs/buckets —
_Requiredand_Defaultretention and bucket locking (last validated 2026-09-04) - https://cloud.google.com/asset-inventory/docs/overview — asset history and its 35-day window (last validated 2026-09-04)
- https://cloud.google.com/iam/docs/pam-overview — Privileged Access Manager entitlements and grants (last validated 2026-09-04)
- https://cloud.google.com/kubernetes-engine/docs/how-to/security-mitigations — container escape mitigation and the node boundary (last validated 2026-09-04)
- https://cloud.google.com/security-command-center/docs/respond-compute-engine-threats — Google's published Compute Engine response guidance, whose ordering this chapter departs from (last validated 2026-09-04)