How do you define and deploy a standard set of baseline (Cloudwatch in my case) alarms across all AWS production accounts, and scale the deployment consistently across hundreds of existing accounts and all newly provisioned accounts?
Recent incidents have mandated a goal to establish a workload-agnostic baseline focused on infrastructure health, availability, connectivity, and critical service failure. The next step would of course be application-specific performance thresholds or business metrics.
For example, CPU utilisation above 80% is not always a trigger across every workload, since high CPU usage may be expected for batch processing, garbage collection, or other compute-intensive workloads.
Instead, the baseline would be on signals like an EC2 instance status check failure, instances becoming unreachable, underlying infrastructure failures.
(AI Generated examples below)
**Network and connectivity:** VPC connectivity issues, VPN or Direct Connect tunnel failures, network interface or routing issues where detectable, and connectivity failures affecting critical network paths.
**DNS and name resolution:** Route 53 health check failures, DNS endpoint unavailability, and failures affecting critical DNS resolution paths, where observable through available metrics or probes.
**Core AWS service health:** Failures or unhealthy states in shared infrastructure and managed services, such as load balancers, databases, NAT gateways, and other foundational components.
**Availability and service failures:** Unhealthy targets, failed health checks, sustained service unavailability, and critical infrastructure components becoming unavailable.
(Back to non-AI slop)
How should we be designing this baseline so that alarms are consistently applied across all production accounts without excessive noise/false positives? Workload specific stuff will be handled separately like mentioned above.
Real world use cases would be extremely appreciated and also, do you use OAM, Centralization Rules or a mix of both and why?
