GCP cost investigation and Billing Guard
From a fee that jumped over 6,000x to an automatic breaker for runaway APIs.
My role
I ran the investigation and wrote the report. Billing Guard is all mine, 15 of 15 commits.
- R$ 128k
- per year in projected savings
- 2.5x
- gap between bytes read and billed, explained
- 1/h
- cap on automatic shutoffs
Problem
After a Google pricing change, one project's Cloud Storage transfer fee went from cents to more than R$ 10k a month. None of the internal theories matched the numbers.
Solution
I joined billing data with the job INFORMATION_SCHEMA across several projects and found the cause, dbt runs with 23 models in parallel reading external Parquet in a multi-region bucket from a single region. I also explained why billed bytes were 2.5x what BigQuery reported, since the Storage Read API pulls whole row groups. Google's product team confirmed it.
Highlights
- Billing Guard, in Go on Cloud Run, takes cost anomalies from Pub/Sub. During business hours it posts to Discord. Outside them it checks for live traffic in Cloud Monitoring and disables the API.
- Split IAM roles. The service can only disable, and only a person can re-enable.
- Firestore dedup and leases, a one-shutoff-per-hour circuit breaker, protected projects and a service allowlist (GKE is never touched).
Result
The fix projects R$ 128.9k a year in savings, and Billing Guard covers every project in the organization.