Live status of the GCP monitoring alarms defined in the infrastructure repository.
Jun 2026 - Sep 2026
Backend
BigQuery Transfer Failure — scheduled queries (incl. latency pipeline)
BigQuery Data Transfer (scheduled query) failures, incl. the latency pipeline (aggregate_15m, train_arima, detect_anomalies). Red when a transfer run fails.
99.951% uptime
90 days agoToday
Cloud Run Job Failed — sync-regulatory-alerts (production)
Cloud Run job 'lexroom-backend-sync-regulatory-alerts-production' execution health. Red when the job reports failed executions.
99.992% uptime
90 days agoToday
Cloud Run Job Failed — sync-regulatory-alerts (staging)
Cloud Run job 'lexroom-backend-sync-regulatory-alerts-staging' execution health. Red when the job reports failed executions.
Availability SLO error-budget burn for lexroom-backend-sse-production (99.9% / rolling 7d). Red on fast burn (14x/1h) or slow burn (5x/6h).
89.673% uptime
90 days agoToday
lexroom-backend-production Cloud Run Uptime Check uptime failure
Cloud Run HTTP uptime probe for lexroom-backend-production. Red when the uptime check fails.
99.954% uptime
90 days agoToday
Data / ETL pipelines
lex_etl_low_priority_failed
Dead-letter queue alert for Pub/Sub topic 'etl-low-priority-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.728% uptime
90 days agoToday
lex_etl_low_priority_remove_failed
Dead-letter queue alert for Pub/Sub topic 'etl-low-priority-remove-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.967% uptime
90 days agoToday
lex_etl_low_priority_remove_failed_test
Dead-letter queue alert for Pub/Sub topic 'etl-low-priority-remove-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.992% uptime
90 days agoToday
lex_etl_low_priority_reprocess_failed
Dead-letter queue alert for Pub/Sub topic 'etl-low-priority-reprocess-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.996% uptime
90 days agoToday
lex_etl_low_priority_reprocess_failed_test
Dead-letter queue alert for Pub/Sub topic 'etl-low-priority-reprocess-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.954% uptime
90 days agoToday
lex_etl_low_priority_failed_test
Dead-letter queue alert for Pub/Sub topic 'etl-low-priority-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
98.117% uptime
90 days agoToday
lex_etl_low_priority_update_metadata_failed
Dead-letter queue alert for Pub/Sub topic 'etl-low-priority-update-metadata-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.975% uptime
90 days agoToday
lex_etl_low_priority_update_metadata_failed_test
Dead-letter queue alert for Pub/Sub topic 'etl-low-priority-update-metadata-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.988% uptime
90 days agoToday
lex_etl_failed
Dead-letter queue alert for Pub/Sub topic 'etl-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.573% uptime
90 days agoToday
lex_etl_remove_failed
Dead-letter queue alert for Pub/Sub topic 'etl-remove-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.983% uptime
90 days agoToday
lex_etl_remove_failed_test
Dead-letter queue alert for Pub/Sub topic 'etl-remove-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.992% uptime
90 days agoToday
lex_etl_reprocess_failed
Dead-letter queue alert for Pub/Sub topic 'etl-reprocess-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.996% uptime
90 days agoToday
lex_etl_reprocess_failed_test
Dead-letter queue alert for Pub/Sub topic 'etl-reprocess-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.988% uptime
90 days agoToday
lex_etl_failed_test
Dead-letter queue alert for Pub/Sub topic 'etl-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.992% uptime
90 days agoToday
lex_etl_update_metadata_failed
Dead-letter queue alert for Pub/Sub topic 'etl-update-metadata-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.967% uptime
90 days agoToday
lex_etl_update_metadata_failed_test
Dead-letter queue alert for Pub/Sub topic 'etl-update-metadata-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.983% uptime
90 days agoToday
semantic_indexing_failed
Dead-letter queue alert for Pub/Sub topic 'indexing-prod-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.946% uptime
90 days agoToday
semantic_indexing_failed_test
Dead-letter queue alert for Pub/Sub topic 'indexing-test-dlq'. Red when messages are published to the DLQ (a downstream consumer failed).
99.987% uptime
90 days agoToday
spider_failed
Dead-letter queue alert for Pub/Sub topic 'spider-failed-trigger-topic'. Red when messages are published to the DLQ (a downstream consumer failed).
99.715% uptime
90 days agoToday
spider_failed_dev
Dead-letter queue alert for Pub/Sub topic 'spider-failed-trigger-topic-dev'. Red when messages are published to the DLQ (a downstream consumer failed).
99.855% uptime
90 days agoToday
spider_failed_test
Dead-letter queue alert for Pub/Sub topic 'spider-failed-trigger-topic-test'. Red when messages are published to the DLQ (a downstream consumer failed).
99.806% uptime
90 days agoToday
spider_starter_failed
Dead-letter queue alert for Pub/Sub topic 'spider-starter-dlq-topic'. Red when messages are published to the DLQ (a downstream consumer failed).
100.000% uptime
90 days agoToday
spider_starter_failed_dev
Dead-letter queue alert for Pub/Sub topic 'spider-starter-dlq-topic-dev'. Red when messages are published to the DLQ (a downstream consumer failed).
99.992% uptime
90 days agoToday
spider_starter_failed_test
Dead-letter queue alert for Pub/Sub topic 'spider-starter-dlq-topic-test'. Red when messages are published to the DLQ (a downstream consumer failed).
99.983% uptime
90 days agoToday
AI / Vertex
Vertex AI 4xx Error Rate (shared pool) — gemini-2.0-flash
Vertex AI shared-pool client-error (4xx) rate for model 'gemini-2.0-flash'. Red when the 4xx ratio exceeds 25% for 10 minutes (usually 429 quota throttling).
99.996% uptime
90 days agoToday
Vertex AI 4xx Error Rate (shared pool) — gemini-2.5-flash
Vertex AI shared-pool client-error (4xx) rate for model 'gemini-2.5-flash'. Red when the 4xx ratio exceeds 25% for 10 minutes (usually 429 quota throttling).
91.591% uptime
90 days agoToday
Vertex AI 4xx Error Rate (shared pool) — gemini-2.5-pro
Vertex AI shared-pool client-error (4xx) rate for model 'gemini-2.5-pro'. Red when the 4xx ratio exceeds 25% for 10 minutes (usually 429 quota throttling).
95.278% uptime
90 days agoToday
Vertex AI 4xx Error Rate (shared pool) — gemini-3.1-flash-lite
Vertex AI shared-pool client-error (4xx) rate for model 'gemini-3.1-flash-lite'. Red when the 4xx ratio exceeds 25% for 10 minutes (usually 429 quota throttling).
99.916% uptime
90 days agoToday
Vertex AI 4xx Error Rate (shared pool) — gemini-3.5-flash
Vertex AI shared-pool client-error (4xx) rate for model 'gemini-3.5-flash'. Red when the 4xx ratio exceeds 25% for 10 minutes (usually 429 quota throttling).
99.806% uptime
90 days agoToday
Vertex AI 5xx Error Rate — gemini-2.0-flash
Vertex AI server-error (5xx) rate for model 'gemini-2.0-flash'. Red when the 5xx ratio exceeds 5% for 10 minutes (Google-side model degradation).
99.987% uptime
90 days agoToday
Vertex AI 5xx Error Rate — gemini-2.5-flash
Vertex AI server-error (5xx) rate for model 'gemini-2.5-flash'. Red when the 5xx ratio exceeds 5% for 10 minutes (Google-side model degradation).
99.983% uptime
90 days agoToday
Vertex AI 5xx Error Rate — gemini-2.5-pro
Vertex AI server-error (5xx) rate for model 'gemini-2.5-pro'. Red when the 5xx ratio exceeds 5% for 10 minutes (Google-side model degradation).
99.983% uptime
90 days agoToday
Vertex AI 5xx Error Rate — gemini-3.1-flash-lite
Vertex AI server-error (5xx) rate for model 'gemini-3.1-flash-lite'. Red when the 5xx ratio exceeds 5% for 10 minutes (Google-side model degradation).
99.987% uptime
90 days agoToday
Vertex AI 5xx Error Rate — gemini-3.5-flash
Vertex AI server-error (5xx) rate for model 'gemini-3.5-flash'. Red when the 5xx ratio exceeds 5% for 10 minutes (Google-side model degradation).
99.975% uptime
90 days agoToday
Latency
Latency — lexroom-backend-production
Latency anomaly status for the lexroom-backend-production Cloud Run service. Red when the latency-status-check function reports a sustained latency anomaly.
95.106% uptime
90 days agoToday
Latency — lexroom-backend-sse-production
Latency anomaly status for the lexroom-backend-sse-production Cloud Run service. Red when the latency-status-check function reports a sustained latency anomaly.
Latency anomaly status for the /v1/ask_me_anything route on lexroom-backend-production. Red when the latency-status-check function reports a sustained latency anomaly.
Latency anomaly status for the /v1/drafting_atto route on lexroom-backend-production. Red when the latency-status-check function reports a sustained latency anomaly.
99.980% uptime
90 days agoToday
Latency — lexroom-backend-production /v1/library
Latency anomaly status for the /v1/library route on lexroom-backend-production. Red when the latency-status-check function reports a sustained latency anomaly.
99.947% uptime
90 days agoToday
Latency — lexroom-backend-production /v1/login
Latency anomaly status for the /v1/login route on lexroom-backend-production. Red when the latency-status-check function reports a sustained latency anomaly.
Latency anomaly status for the /v1/research/{research_id} route on lexroom-backend-production. Red when the latency-status-check function reports a sustained latency anomaly.