Chapter 18

Monitoring, Reliability, and Incident Detection

Scope. This chapter covers Cloud Monitoring and the reliability practice built on it: the metric model, dashboards, alert policies, uptime and synthetic checks, Managed Service for Prometheus, OpenTelemetry, tracing, SLOs and error budgets, and the routing of alerts to humans. It owns notification channels and alert policies for the whole book. Log storage and routing are Chapter 17; security findings are Chapter 16; incident response procedure is Chapter 34. Prerequisites. Chapter 17 (§17.2 Admin Activity logs, §17.9 sinks), Chapter 3 (§3.9 roles), Chapter 9 and Chapter 10 for the workloads being monitored. Verified against. Google Cloud console and API surface as of 2026-09, Cloud SDK 583.0.0, hashicorp/google provider 8.x; see sources at end.

Monitoring appears in a security book for a specific reason: the detection half of security operations runs on monitoring infrastructure, not on security infrastructure. Security Command Center tells you a detector fired. Cloud Monitoring is what wakes a human up, and it is the only mechanism in Google Cloud that does. Every alert this book has promised — a firewall rule opened to the internet, a sink deleted, a key destroyed, a service account granted an administrative role — is a Cloud Monitoring alert policy over a log-based metric, and the plumbing is identical to the plumbing that pages someone when latency rises.

That shared plumbing is an advantage and a trap. The advantage is that you build one notification topology, one on-call rotation, and one set of runbooks. The trap is that the two workloads have opposite tolerances. An operational alert that fires spuriously twice a week is annoying; a security alert that fires spuriously twice a week is disabled within a month, and nobody records that it was. Security alerts must be rarer and more specific than operational ones, which means they are written against different signals: control-plane events that should essentially never happen, rather than thresholds on continuous metrics.

The reliability material here — SLIs, SLOs, error budgets — earns its place for the same reason. An error budget is the mechanism that decides whether a team ships or stabilizes, and a security control that degrades reliability without a budget conversation gets removed by the team it slowed down. Reliability engineering is how a security control survives contact with a delivery organization.

18.1 Cloud Monitoring §

Cloud Monitoring ingests, stores, and queries time series, and evaluates alert policies against them. It is the only Google Cloud service that can notify a human.

A metrics scope defines what a project can see. A scoping project holds the scope; monitored projects are added to it, and dashboards and alert policies in the scoping project see metrics from all of them. That is how a single pane covers an estate without copying data.

This estate uses one scoping project per environment tier, not one for the whole organization: rc-saas-shared-sec-01 scopes production and shared projects for security alerting, and each environment's operations project scopes its own. A single organization-wide scope makes every dashboard slow and every alert policy ambiguous about which project it refers to.

The GA command surface is narrower than the API. gcloud monitoring provides dashboards, policies (with a conditions subgroup), snoozes, and uptime. Notification channel descriptors are listable only through the beta component, which is not installed here — the channel types themselves are documented and are listed in §18.17.

IAM. roles/monitoring.viewer to read, roles/monitoring.editor to build dashboards and policies, roles/monitoring.alertPolicyEditor for the narrower job of managing alerts, and roles/monitoring.metricWriter for workloads that write custom metrics — which is what the GKE node service account holds (§9.11) and nothing more.

Pitfall. Adding a project to a metrics scope grants visibility to every principal who can read the scoping project's metrics. A scoping project that spans production and development lets a developer read production metrics, including any label containing a customer identifier.

18.2 Metrics §

A metric is a named, typed measurement collected from a monitored resource over time. Google describes roughly 6,500 built-in metric types across about 270 monitored resource types.

Three properties define a time series, and getting them wrong is the source of most confusing charts:

Cardinality is the cost and the failure mode. Cardinality is "the number of unique combinations of values for the set of labels", and a label whose values are unbounded — a request ID, a user email, a full URL path — multiplies the series count without limit. This is the single most common way a team makes their own monitoring unusable.

Log-based metrics are the bridge from Chapter 17 to this one. They are "Cloud Monitoring metrics that are derived from the content of log entries", in two shapes: counter metrics "tally how many log entries match specific criteria", and distribution metrics "accumulate numeric data from matching log entries". Counters are what security alerting is built from (§18.15).

gcloud logging metrics create iam-policy-changes \
  --project=rc-saas-shared-sec-01 \
  --description="Count of SetIamPolicy calls across the estate." \
  --log-filter='protoPayload.methodName:"SetIamPolicy"'

Pitfall. A log-based metric counts entries in the project where the metric is defined, from that project's logs. Defining it in the security project counts only that project's own events unless the aggregated sink routes other projects' logs into a bucket there and the metric is scoped to that bucket with --bucket-name.

18.3 Dashboards §

A dashboard is a saved arrangement of charts over a metrics scope. Its value in a security context is not the charts — it is that a dashboard is a versioned artifact you can review.

Dashboards are JSON and belong in Git. The gcloud surface takes a file, and Terraform takes the same JSON as a string, which makes a dashboard reviewable in the same pull request as the control it visualizes:

gcloud monitoring dashboards create \
  --config-from-file=dashboards/security-control-plane.json \
  --project=rc-saas-shared-sec-01

resource "google_monitoring_dashboard" "security_control_plane" {
  project        = "rc-saas-shared-sec-01"
  dashboard_json = file("${path.module}/dashboards/security-control-plane.json")
}

Build two, not twenty. A service dashboard answers "is this application healthy" with its SLIs (§18.13) and its error budget burn. A security control-plane dashboard answers "did anything change that should not have", charting the log-based metrics behind your alert policies. Everything else is a chart someone builds during an incident and then abandons.

A dashboard is not an alert. Nobody watches a dashboard at three in the morning. The rule this book applies: if a condition matters enough to appear on a dashboard as a threshold line, it matters enough to be an alert policy, and if it does not deserve an alert, it does not deserve a threshold line either.

Pitfall. Dashboard JSON embeds project IDs in its filters. Copying a production dashboard into a development scoping project produces charts that silently show production data or no data at all, depending on scope membership. Parameterize project IDs through Terraform rather than committing the exported JSON verbatim.

18.4 Alert Policies §

An alert policy is a set of conditions, a combiner, and a set of notification channels. When the conditions are met an incident opens and notifications are sent.

Four condition types cover everything this book alerts on:

A PromQL condition must be alone. Google is explicit: "A PromQL condition must be the sole condition in a policy, while multiple metric-threshold or metric-absence conditions can coexist within a single policy."

LogMatch versus a log-based metric is the security-alerting decision. A LogMatch condition fires on the appearance of a single matching entry, with minimal delay and no metric to configure — right for an event that should never happen. A threshold over a log-based counter tolerates a baseline and alerts on a rate — right for failed authentications, where zero is not the expected value.

gcloud monitoring policies create \
  --project=rc-saas-shared-sec-01 \
  --policy-from-file=policies/alert-sink-deleted.json

alertStrategy controls the noise. notificationRateLimit restricts how frequently notifications are sent, notificationPrompts selects whether opening and closing both notify, and autoClose closes incidents that stop receiving data after a stated duration. Set autoClose on every metric-based policy; without it, incidents from resources that were deleted stay open forever.

Pitfall. The combiner applies across conditions within one policy, and it is easy to write OR when the intent was "both must be true". A policy that pages on either high latency or high error rate is usually correct; one meant to page only on both needs AND and, more often, a single better-chosen condition.

18.5 Uptime Checks §

An uptime check issues requests from Google-operated locations to a target and records whether it responded. Google's own description: they "issue requests from multiple locations throughout the world to publicly available URLs or Google Cloud resources to see whether the resource responds."

Public checks support three protocols — "HTTP (standard web requests), HTTPS (secure web requests with optional SSL certificate validation), TCP (direct connection checks to specified ports)" — and Google notes that "All protocols follow redirects, and uptime checks don't load page assets or run JavaScript."

Regions are coarse and deliberate. "The uptime-check region USA includes the USA_OREGON, USA_IOWA, and USA_VIRGINIA regions. Other regions include EUROPE, SOUTH_AMERICA, and ASIA_PACIFIC. Selecting Global deploys from all available regions."

gcloud monitoring uptime create "billing-prod-https" \
  --project=rc-saas-shared-sec-01 \
  --resource-type=uptime-url \
  --resource-labels=host=api.rickcollette.domain,project_id=rc-saas-prod-app-01 \
  --protocol=HTTPS \
  --path=/healthz \
  --port=443 \
  --validate-ssl=true \
  --period=5 \
  --regions=usa,europe

Private uptime checks reach inside the VPC. They "enable HTTP or TCP requests into a customer Virtual Private Cloud (VPC) network while enforcing Identity and Access Management (IAM) restrictions and VPC Service Controls perimeters", targeting "the internal IP address configured in the Service Directory endpoint" — and, importantly, "When your service contains multiple endpoints, one endpoint is chosen at random."

The security-relevant use is certificate expiry. --validate-ssl=true on an HTTPS check produces an SSL-certificate metric that alerts before a certificate expires, which is the cheapest available guard against the most common self-inflicted outage. Certificate Manager (§7.x) automates renewal; the uptime check proves it worked.

Pitfall. A public uptime check requires the target to accept traffic from Google's uptime servers, whose ranges are listed by gcloud monitoring uptime list-ips. Adding those ranges to a Cloud Armor allow rule or a firewall policy without also constraining the path creates a small unauthenticated surface — scope the check to a dedicated health endpoint that reveals nothing.

18.6 Synthetic Monitoring §

A synthetic monitor runs your own code on a schedule against your service and reports pass or fail, where an uptime check only observes a response.

It is generally available and it is a Cloud Function underneath. Google's statement: "Synthetic monitors are now GA (General Availability). You can create synthetic monitors by using Terraform, the Cloud console, and the Monitoring API. When you configure a synthetic monitor, you create a Google Cloud Function V2 that executes code using NodeJS or Typescript using an open source framework that is distributed and managed by Google Cloud Monitoring."

Because it is your code, it can assert on semantics — that a login flow completes, that an authorization check actually denies, that a signed URL expires when it should. That last category is the one this book cares about: a synthetic monitor is the only mechanism here that continuously tests a security control rather than observing its configuration.

It is created through the uptime command group, using --synthetic-target to name the deployed function:

gcloud monitoring uptime create "authz-denial-probe" \
  --project=rc-saas-shared-sec-01 \
  --synthetic-target=projects/rc-saas-shared-sec-01/locations/us-central1/functions/authz-probe \
  --period=15

Give the monitor its own identity and its own credentials. The function runs as a service account (sa-run-authz-probe, per §10.x naming) that must hold exactly the access the probe exercises and nothing else. A probe that authenticates as a privileged principal to prove that a low-privilege one is blocked has proved nothing.

Pitfall. A synthetic monitor is a scheduled deployment of code with credentials, running on a fifteen-minute cycle forever. It is a supply-chain surface like any other, so it goes through the same build and provenance path as production code (Chapter 23), not a console paste.

18.7 Managed Service for Prometheus §

Google Cloud Managed Service for Prometheus is "Google Cloud's fully managed, multi-cloud, cross-project solution for Prometheus and OpenTelemetry metrics" — Prometheus-compatible ingestion and PromQL query, with Monitoring's storage and retention behind it.

Two collection models, and Google names its preference:

Google also states the support difference plainly — managed collection carries "Full Google Cloud technical support"; self-deployed carries "Limited Google Cloud technical support".

Managed collection is configured with a custom resource, not a config file. Scrape targets are declared by a PodMonitoring resource in the workload's namespace, which means the scrape configuration is reviewed in the same manifest as the workload and is subject to the same admission control (§9.x).

The security value is that PromQL alerting stays available. Teams arriving from a self-hosted Prometheus have alerting rules written in PromQL, and PrometheusQueryLanguageCondition (§18.4) evaluates them directly — with the caveat that such a condition must be the only one in its policy.

Pitfall. Prometheus label conventions produce high cardinality easily, and Managed Service for Prometheus bills by samples ingested. A single pod label on a deployment that restarts frequently is fine; a request_id or user label is a bill nobody predicted.

18.8 OpenTelemetry §

OpenTelemetry is the vendor-neutral instrumentation standard Google Cloud has adopted as its primary path for metrics and traces from application code.

On Compute Engine the answer is the Ops Agent. Google states that "The Ops Agent is Google Cloud's primary telemetry collection tool for Compute Engine instances. It consolidates logging, metrics, and traces into a single unified process, leveraging Fluent Bit for logs… and the OpenTelemetry Collector for metrics and traces", and that "The Ops Agent can gather OpenTelemetry Protocol (OTLP) metrics and traces from applications instrumented with OpenTelemetry SDKs." The recommendation is explicit: "Google recommends using the Ops Agent for new workloads and eventually transitioning existing VMs to use the Ops Agent."

OTLP data is mapped into Monitoring's model by the Telemetry API: an "OTLP Gauge → Cloud Monitoring gauge, OTLP Sum (monotonic, cumulative) → Cloud Monitoring cumulative, OTLP Histogram → Cloud Monitoring distribution", with the note that "All INT64 metrics are translated to DOUBLE value types in Cloud Monitoring to prevent value-type collisions."

Instrument once, export anywhere. The practical argument for OpenTelemetry in a security context is portability of evidence: the same instrumentation feeds Cloud Monitoring, a SIEM, and whatever the organization adopts next, so telemetry requirements written into a service contract do not become a lock-in decision.

Pitfall. Trace and metric payloads carry whatever attributes the developer attached. An OpenTelemetry span attribute containing a bearer token, an email address, or a full request body is exported to Monitoring and retained there under Monitoring's IAM, not under the log views of §17.8. Review span attributes the way you review log statements.

18.9 Application Performance Monitoring §

Application performance monitoring in Google Cloud is not a single product; it is the combination of metrics (§18.2), traces (§18.10), and error aggregation (§18.11) over the same resource labels.

The join key is the monitored resource. A Cloud Run revision, a GKE pod, or a Compute Engine instance appears with the same resource labels in metrics, in traces, and in logs, which is what lets a latency spike be followed to a trace and then to the log entries of that request. Nothing about this works if services write logs without the resource context — which is the practical reason to log through the platform's own agent rather than to a file.

Three signals, three questions:

• Metrics — is it slow, and how slow, aggregated over time?
• Traces — where in the call graph is the latency, for one request?
• Errors — which exception is responsible, and how often?

The security angle is anomaly context, not performance. A step change in latency or error rate at a moment that also has an unusual Admin Activity entry (§17.2) is the correlation that turns two ambiguous signals into an investigation. Making that correlation cheap is the argument for one metrics scope, one log destination, and consistent resource labeling across the estate.

Pitfall. Sampling is on by default for traces, so the interesting request is frequently not in the trace store. Do not design an investigation procedure that assumes a specific request is traceable; design it to assume the aggregate is.

18.10 Trace §

Cloud Trace is "a distributed tracing system for Google Cloud that collects latency data from applications and displays it in near real-time in the Google Cloud console."

Traces arrive from instrumentation, not from the platform. Google Cloud load balancers and some managed services propagate and generate spans, but a useful trace requires the application to propagate context — the W3C traceparent header, or Google's X-Cloud-Trace-Context — across every hop. A service that terminates context produces a trace that stops at its front door.

The GA gcloud surface is scope management, not trace management. gcloud observability trace-scopes creates and manages trace scopes; there is no GA gcloud trace command group. Trace data itself is read through the console and the API.

Access is separate from log access, which teams routinely miss: roles/cloudtrace.user reads traces, and someone restricted from Data Access logs by a log view (§17.8) may still read request URLs and parameters in trace span names.

Judgment. Set span names to route templates, never to concrete paths. /orders/{id} is a low-cardinality span name and reveals nothing; /orders/8831/card/4111... is a high-cardinality span name that has just put data in a store with different access controls than your logs.

Pitfall. Trace context propagation is also a trust boundary. Accepting an inbound traceparent from the public internet lets a caller choose your trace IDs, which is harmless for correlation and unhelpful for anything else — strip and regenerate it at the edge for external traffic.

18.11 Error Reporting §

Error Reporting "aggregates and displays errors produced in your running cloud services", grouping stack traces into recurring issues rather than listing individual exceptions.

Its input is your logs. Errors are recognized from log entries whose payload contains a stack trace in a recognized format, or that are written through a client library's error reporting path. That means Error Reporting inherits whatever your logging does — including any sensitive data in an exception message.

The security use is the change in shape, not the errors themselves. A new error group appearing at high volume immediately after a deployment is an operational signal. A new authentication or authorization error group appearing without a deployment is a security signal, and it is often the earliest one available — credential stuffing and permission probing both show up here before they show up anywhere else.

Its GA gcloud surface is limited; gcloud error-reporting is available only through the beta component, which is not installed in this environment. Automation against Error Reporting uses the REST API and the client libraries.

Pitfall. Stack traces routinely contain query parameters, request bodies, and connection strings. Error Reporting stores them and groups them, which means a single exception with a credential in its message becomes a persistent, searchable record of that credential. Sanitize exception messages at the source, and treat this as a reason to keep secrets out of anything that can appear in a trace (§13.x).

18.12 SLOs §

A service level objective is a target for a service level indicator over a stated period. In Cloud Monitoring it is a first-class resource attached to a service.

The API models it in three parts, which is what makes it configurable as code:

resource "google_monitoring_custom_service" "billing" {
  project      = "rc-saas-shared-sec-01"
  service_id   = "billing-prod"
  display_name = "Billing API (production)"
}

resource "google_monitoring_slo" "billing_availability" {
  project             = "rc-saas-shared-sec-01"
  service             = google_monitoring_custom_service.billing.service_id
  slo_id              = "billing-availability-30d"
  display_name        = "99.9% of requests succeed, 30-day rolling"
  goal                = 0.999
  rolling_period_days = 30

  request_based_sli {
    good_total_ratio {
      good_service_filter  = join(" AND ", local.slo_good_filter)
      total_service_filter = join(" AND ", local.slo_total_filter)
    }
  }
}

The provider caps goal at 0.999. An estate that wants to express four nines does so with a shorter period or a differently defined SLI, not by raising the number — which is a useful constraint, because a 99.99% target measured over 30 days permits about four minutes of failure and is rarely a target anyone can actually defend.

Judgment. Write one availability SLO and one latency SLO per user-facing service, and nothing else initially. SLOs on internal components produce a number nobody negotiates over, which is the whole point of having one.

Pitfall. An SLO whose SLI is defined over load balancer metrics measures the load balancer. If the failure mode you care about is the backend returning a 200 with an error body, the SLI has to come from the application, not the platform.

18.13 SLIs §

A service level indicator is the measurement an SLO is a target for. Cloud Monitoring supports three shapes, and choosing among them is the actual design work.

Request-based is the default and the honest one. A good/total ratio over the requests users actually made says what fraction of user interactions succeeded. A windows-based SLI can report 99.9% good minutes while every request in the bad minutes failed, which is a materially different user experience described by the same number.

Define "good" narrowly. A good request is one that returned a successful status and did so within the latency threshold users notice. Splitting availability and latency into two SLOs is defensible; defining availability as "did not return 5xx" and then being surprised by complaints is not.

Security controls belong in the SLI conversation. An authorization check that adds 40 milliseconds to every request consumes latency budget. Making that visible as an SLI cost, negotiated openly, is how the control stays deployed — an invisible cost gets removed during the next performance push without a security review.

Pitfall. SLI filters written against metric labels silently return no data when a label is renamed, and an SLI with no data does not fail — it reports perfect compliance. Pair every SLO with a MetricAbsence alert (§18.4) on its own input series.

18.14 Error Budgets §

An error budget is the failure the SLO permits: one minus the goal, over the period. A 99.9% target over 30 days permits roughly 43 minutes of failure.

It is a decision rule, not a report. The budget's purpose is to answer one question — do we ship, or do we stabilize? — with a number both engineering and product agreed to in advance rather than an argument during an outage.

Three states, three behaviors:

• Budget healthy — ship. Risk-taking is affordable, including migrations and security control rollouts that might cause errors.
• Budget burning fast — alert on the burn rate, not on the raw SLI. A fast-burn alert (a large fraction of the budget consumed in a short window) is the highest-signal operational page available.
• Budget exhausted — feature work stops until reliability work restores it.

Burn-rate alerting is where the budget becomes operational. Alerting on "the SLI is below target right now" pages on every transient blip. Alerting on "we are consuming budget fast enough to exhaust it in two hours" pages on things that matter, and Cloud Monitoring's SLO burn-rate condition is built for exactly this.

Security's stake is the exhausted state. A team with an exhausted budget will not accept a risky security migration that week, and it should not — but the same rule means the migration is scheduled rather than refused. Without a budget, "we cannot take that risk right now" is unfalsifiable and permanent.

Pitfall. Error budgets only function when someone is empowered to enforce the stop. A budget that is exhausted every month and never stops anything trains everyone to ignore it, and it then also devalues the burn-rate alerts built on it.

18.15 Security Alerting §

Security alerting is the mechanism that turns a Chapter 17 log entry into a page. This section owns that mechanism for the whole book; Security Command Center's finding export is §16.14.

The pattern is one shape, repeated: a log-based counter metric over an Admin Activity filter (§17.2), plus an alert policy with a threshold of zero and a notification channel that reaches a human.

gcloud logging metrics create sink-config-changes \
  --project=rc-saas-shared-sec-01 \
  --description="Deletion or modification of a log sink, organization-wide." \
  --log-filter='protoPayload.serviceName="logging.googleapis.com"
    AND (protoPayload.methodName:"DeleteSink"
    OR protoPayload.methodName:"UpdateSink")'

Alert on Admin Activity, because it cannot be turned off. An alert built on Data Access logs (§17.3) can be silenced by editing an audit config; an alert built on Admin Activity survives every attempt to suppress it, and the suppression attempt itself is an Admin Activity event.

The alert set this book has promised across earlier chapters, in one place:

Route these separately from operational alerts. A distinct notification channel, a distinct rotation, and a distinct runbook per row. Merging them into the operational stream is the reliable way to have them ignored.

Pitfall. A threshold-of-zero alert on a log-based counter has a latency floor set by the metric's aggregation window, typically a minute or more. For events where seconds matter, use a LogMatch condition (§18.4) instead, which fires on the entry rather than on a derived metric.

18.16 Operational Alerting §

Operational alerting is everything that pages for availability, latency, saturation, or capacity, and its governing rule is different from §18.15's: alert on symptoms, not on causes.

Symptom-based alerting means alerting on what a user experiences. High CPU is a cause and often harmless; elevated request latency at the edge is a symptom and always matters. A page for every cause produces an alert catalogue nobody can maintain and an on-call rotation nobody wants.

Four signals cover most services — latency, traffic, errors, and saturation — and in this book they are expressed as SLO burn-rate alerts (§18.14) rather than as raw thresholds wherever an SLO exists.

Snoozes are the right tool for planned work. A snooze suppresses matching alerts for a bounded window, which is what a maintenance window needs; disabling a policy is what a team does instead and then forgets to undo:

gcloud monitoring snoozes create \
  --project=rc-saas-shared-sec-01 \
  --display-name="Billing DB maintenance window" \
  --criteria-policies=projects/rc-saas-shared-sec-01/alertPolicies/POLICY_ID \
  --start-time="2026-10-04T02:00:00Z" \
  --end-time="2026-10-04T04:00:00Z"

Never snooze a security policy. The maintenance window that justifies suppressing a latency alert is exactly the window in which a change to a security control would be least noticed. If a planned change will fire a security alert, notify the on-call in advance and let it fire.

Pitfall. Every alert policy needs autoClose in its alertStrategy. Without it, incidents for resources that have been deleted — a scaled-down instance, a removed job — remain open indefinitely and mask new incidents on the same policy.

18.17 Pager Integration §

Pager integration is the delivery of a notification to a human who is awake, and Cloud Monitoring's notification channels are the only supported path.

Google documents these channel types: email; the Google Cloud console mobile app; PagerDuty, with "two-way synchronization enabling incident creation in PagerDuty from Monitoring alerts"; SMS to verified phone numbers, which Google notes is "not fully reliable in certain regions"; Slack; webhooks to a public endpoint "supporting basic and token authentication"; Pub/Sub, for custom automation; and Google Chat, which Google labels a Preview feature and which must therefore not be a production dependency.

One warning in Google's own documentation reshapes the design: "Cloud Mobile App, PagerDuty, Webhooks, and Slack notifications are all powered by the same Google Cloud internal service and therefore share a single point of failure."

Which means: two independent channels for anything that matters. For a page that must arrive, pair a channel from that shared-fate group with one outside it — email or SMS — rather than pairing PagerDuty with Slack and believing you have redundancy.

resource "google_monitoring_notification_channel" "security_pagerduty" {
  project      = "rc-saas-shared-sec-01"
  display_name = "Security on-call (PagerDuty)"
  type         = "pagerduty"

  sensitive_labels {
    service_key = var.pagerduty_service_key
  }
}

The PagerDuty service key is a secret and never a literal. It is a variable fed from Secret Manager (§13.7), and sensitive_labels keeps it out of the readable resource body — but not out of state, so the write-only pattern of §13.13 applies.

Pitfall. A webhook channel posts to a public endpoint. Anything that can reach that URL can inject a fake incident notification into whatever consumes it, so the receiver must validate the token and must not treat the payload as authenticated on arrival.

18.18 Incident Management §

Incident management is the practice around the notification: who acknowledges, who commands, how state is tracked, and what happens afterward. Cloud Monitoring supplies the incident object; the rest is yours.

Monitoring's incident is a lifecycle, not a message. A policy's conditions being met opens an incident; it closes when the conditions clear or when autoClose expires. Acknowledgement, assignment, and escalation live in the paging system (§18.17), which is why the PagerDuty channel's two-way synchronization is worth the integration effort.

Google Cloud does not provide a general-purpose incident management product for your incidents. Personalized Service Health reports incidents affecting Google's services in your projects, which is a different and complementary signal. Plan your process around your paging vendor and the runbooks in Chapter 34, not around a first-party console.

Four things must be decided before an incident, not during one:

1. Who declares. A single named role, reachable in one step from the page.
2. Where the incident lives. One channel per incident, created automatically, with the incident ID in its name.
3. When security is engaged. Any incident whose cause is not explained within a stated window is treated as potentially a security incident until it is explained.
4. What is preserved. §34.13 owns log preservation; the decision to preserve is made at declaration, not at the postmortem.

Pitfall. Operational incident response deletes evidence by default — restarting the instance, rolling back the deployment, rotating the credential. Every one of those is correct for availability and destructive for investigation. The runbook must say to snapshot first (§34.12), and the only way that happens is if it is step one rather than a note at the end.

18.19 Monitoring as Code §

Every object in this chapter — dashboards, alert policies, notification channels, uptime checks, SLOs, log-based metrics — has a Terraform resource, and treating them as code is what makes monitoring auditable.

Console-created monitoring is unreviewable and undiscoverable. An alert policy edited in the console has no diff, no author, and no reason recorded. For an operational alert that is a minor annoyance; for a security alert it means the control's history is unknowable, which is exactly what §17.20 is trying to prevent for logs.

Module the pattern, not the policy. The security alert set in §18.15 is seven policies that differ only in filter, display name, and severity. One module taking a list of {name, filter, severity} objects produces all of them, and adding an eighth is a one-line change reviewed in a pull request.

Version the notification channels separately from the policies that reference them. Channels change rarely and carry secrets; policies change often and carry none. Splitting them means a routine alert change never requires access to the PagerDuty key.

Pitfall. google_monitoring_alert_policy conflicts badly with console edits: a policy edited in the console is reverted on the next apply, with no warning to the person who made the edit. Decide the ownership per policy, and put an explicit note in the policy's documentation field saying which repository owns it.

Chapter Summary §

• Cloud Monitoring is the only Google Cloud service that notifies a human, so every security alert in this book runs on it.
• A metrics scope determines what a project can observe, and adding a project to a scope grants visibility to everyone who can read that scope's metrics.
• Metric kind, value type, and label cardinality define a time series; unbounded label values are the usual way a team makes their own monitoring unusable.
• Log-based metrics — counter and distribution — are the bridge from audit logs to alerting.
• A PromQL alert condition must be the only condition in its policy; threshold and absence conditions can coexist.
• Set autoClose on every metric-based policy, or incidents from deleted resources stay open forever.
• Uptime checks follow redirects and do not run JavaScript; --validate-ssl gives certificate-expiry alerting almost free.
• Private uptime checks target a Service Directory endpoint and pick one at random when several exist.
• Synthetic monitors are GA and are Cloud Functions running your code, which makes them the only continuous test of a security control here — and a supply-chain surface to build accordingly.
• Managed collection is Google's recommendation for Managed Service for Prometheus in every Kubernetes environment, with full support; self-deployed carries limited support.
• The Ops Agent is Google's recommended Compute Engine telemetry path and speaks OTLP; INT64 OTLP metrics are translated to DOUBLE.
• Trace span names and OpenTelemetry attributes land in a store with different IAM than logs — keep identifiers and secrets out of them.
• Error Reporting groups stack traces and will faithfully retain any credential that appears in one.
• Alert on Admin Activity logs, because they cannot be disabled and the attempt to disable logging is itself one.
• PagerDuty, Slack, webhooks, and the mobile app share a single point of failure; pair one of them with email or SMS for anything that must arrive.
• Google Chat notifications are Preview and must not be a production dependency.
• Never snooze a security alert policy; a maintenance window is when a control change is least likely to be noticed.
• Every monitoring object has a Terraform resource, and console edits to a Terraform-managed alert policy are silently reverted on the next apply.

Security Checklist §

Sources §

• https://cloud.google.com/monitoring — Cloud Monitoring product overview (last validated 2026-09-03)
• https://cloud.google.com/monitoring/api/v3/kinds-and-types — metric kinds and value types (last validated 2026-09-03)
• https://cloud.google.com/monitoring/api/v3/metric-model — metric types, resource types, labels, and cardinality (last validated 2026-09-03)
• https://cloud.google.com/logging/docs/logs-based-metrics — counter and distribution log-based metrics (last validated 2026-09-03)
• https://cloud.google.com/monitoring/alerts — alert policy concepts (last validated 2026-09-03)
• https://cloud.google.com/monitoring/alerts/policies-in-api — condition types and the PromQL sole-condition rule (last validated 2026-09-03)
• https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies — alert policy resource and alertStrategy (last validated 2026-09-03)
• https://cloud.google.com/monitoring/support/notification-options — notification channel types and the shared failure domain (last validated 2026-09-03)
• https://cloud.google.com/monitoring/uptime-checks — uptime check protocols and regions (last validated 2026-09-03)
• https://cloud.google.com/monitoring/uptime-checks/private-checks — private uptime checks and Service Directory endpoints (last validated 2026-09-03)
• https://cloud.google.com/monitoring/synthetic-monitors — synthetic monitoring overview (last validated 2026-09-03)
• https://cloud.google.com/stackdriver/docs/managed-prometheus — managed versus self-deployed collection (last validated 2026-09-03)
• https://cloud.google.com/monitoring/agent/ops-agent — the Ops Agent and OTLP ingestion (last validated 2026-09-03)
• https://cloud.google.com/stackdriver/docs/instrumentation/overview — Google's instrumentation recommendation (last validated 2026-09-03)
• https://cloud.google.com/stackdriver/docs/reference/telemetry/v1.metrics — OTLP to Cloud Monitoring metric mapping (last validated 2026-09-03)
• https://cloud.google.com/trace/docs — Cloud Trace overview (last validated 2026-09-03)
• https://cloud.google.com/error-reporting/docs — Error Reporting overview (last validated 2026-09-03)