# DataAgent fault catalog

Catalog 0efc434 · ruleset v0.1.0 · 35 fault primitives.

Machine-readable version: /coverage/ruleset.json — evaluation semantics: a rule
applies when ALL of its `requires` tokens are present in the stack's token set;
rules whose fault id starts with `NEW:` are roadmap gaps, not covered faults.
Interactive report: /coverage/ (accepts URL params, see /llms.txt).

## Plane: aws

- **postgres_s3_deny** (critical · class: iam_permission_drift) — IAM deny S3:* on postgres IRSA
  - Autonomous remediation: Remove the deny / restore the postgres role's S3 permissions.
- **checkout_sm_deny** (warning · class: iam_permission_drift) — IAM deny SM:GetSecretValue on checkout IRSA
  - Autonomous remediation: Restore the checkout role's GetSecretValue permission.
- **elb_unhealthy** (critical · class: external_service_failure) — ALB target group health check path → 404 → unhealthy
  - Autonomous remediation: Restore the target group's health-check path.
- **sg_rule_block** (warning · class: iam_permission_drift) — EC2 RevokeSecurityGroupIngress on a placeholder SG (CloudTrail-driven)
  - Autonomous remediation: Re-authorize the security-group ingress rule.
- **sg_db_block** (warning · class: dependency_unavailable) — Revoke 5432 ingress on the RDS security group → app loses its DB path (real network partition to RDS)
  - Autonomous remediation: Re-authorize the :5432 ingress on the RDS security group.
- **rds_failover** (critical · class: dependency_unavailable) — reboot-with-ForceFailover the Multi-AZ RDS instance → writer endpoint blips, connections drop
  - Autonomous remediation: Transient — ensure the app's pool reconnects/retries; investigate the failover trigger if unexpected.
- **sqs_receive_deny** (warning · class: iam_permission_drift) — IAM deny sqs:ReceiveMessage on the SQS-consumer IRSA role → AccessDenied on poll
  - Autonomous remediation: Restore the consumer role's sqs:ReceiveMessage permission.
- **karpenter_runinstances_deny** (warning · class: iam_permission_drift) — IAM deny ec2:RunInstances on the Karpenter controller role → no new nodes (DISRUPTIVE; tight window)
  - Autonomous remediation: Restore the Karpenter controller role's ec2:RunInstances/CreateFleet permissions.
- **sqs_consumer_stall** (critical · class: dependency_unavailable) — POST /admin/sqs-consumer enabled=false — consumer stops polling → SQS backlog (ApproximateNumberOfMessagesVisible) grows unbounded
  - Autonomous remediation: Restart/unblock the consumer so it drains the queue; investigate why acking stalled.
- **sqs_poison_messages** (warning · class: service_failure) — POST /admin/poison rate=0.5 — malformed messages exceed max-receive → DLQ depth spikes (DLQ storm)
  - Autonomous remediation: Drain/triage the DLQ, fix the message contract or consumer parsing, and replay valid messages.
- **external_secret_sync_failure** (warning · class: secret_sync_failure) — ExternalSecret sync failure (UpdateFailed)
  - Autonomous remediation: Fix the backing secret's IRSA permissions, or correct the ExternalSecret spec if the source key/path changed.

## Plane: k8s

- **netpol_block** (warning · class: network_partition) — NetPol block checkout→postgres:5432
  - Autonomous remediation: Add an egress rule to checkout-netpol allowing TCP 5432 to postgres.
- **cpu_squeeze** (critical · class: resource_exhaustion) — CPU limit 500m→10m + stress load
  - Autonomous remediation: Restore the checkout CPU limit (500m) / right-size requests+limits.
- **bad_image** (warning · class: crash_loop) — Bad ECR image → ImagePullBackOff
  - Autonomous remediation: Roll the deployment back to a valid (existing) image tag.
- **unschedulable_cpu** (critical · class: scheduling_failure) — Unschedulable resource requests
  - Autonomous remediation: Lower the CPU request to fit a node (or add capacity).
- **broken_probe** (warning · class: probe_failure) — Liveness probe misconfiguration → CrashLoop
  - Autonomous remediation: Fix the liveness probe path back to the real health endpoint.
- **init_container_fail** (warning · class: crash_loop) — Init container with bad image → Init:ImagePullBackOff
  - Autonomous remediation: Correct or remove the init-container image reference.
- **pvc_unbound** (warning · class: storage_failure) — Standalone PVC with missing StorageClass → Pending
  - Autonomous remediation: Point the PVC at a valid StorageClass (or create it).
- **pdb_blocking_rollout** (warning · class: bad_deployment) — PDB minAvailable matches replicas → eviction blocked
  - Autonomous remediation: Lower PDB minAvailable (or raise replicas) so a pod can be evicted.
- **sts_stuck_pvc** (warning · class: storage_failure) — StatefulSet PVC stuck Pending on missing StorageClass
  - Autonomous remediation: Fix the StatefulSet StorageClass to a provisionable one.
- **hpa_cannot_scale** (warning · class: resource_exhaustion) — ResourceQuota pods=3 blocks HPA scale-up
  - Autonomous remediation: Raise the ResourceQuota pod limit (or free quota).
- **service_zero_endpoints** (critical · class: service_failure) — Service selector swapped → Endpoints empty → ALB 503
  - Autonomous remediation: Fix the Service selector to match the deployment's pod labels.
- **configmap_drift** (warning · class: config_drift) — ConfigMap feature flag flipped → checkout returns 503
  - Autonomous remediation: Restore the ConfigMap value and roll the checkout pods.
- **rbac_denied** (warning · class: rbac_denial) — Probe pod's `list configmaps` denied — Role missing `list` verb
  - Autonomous remediation: Grant the missing RBAC verb in the Role/RoleBinding.
- **cert_expired** (critical · class: cert_failure) — cert-manager Certificate references a missing Issuer — never issued
  - Autonomous remediation: Create the missing Issuer, or point the Certificate at an existing one.
- **db_conn_exhaustion** (critical · class: resource_exhaustion) — Postgres max_connections capped + conn-hog → FATAL: too many clients (RDS conn-storm proxy)
  - Autonomous remediation: Kill the connection hog and/or raise max_connections; fix the connection leak.
- **dependency_egress_block** (warning · class: dependency_unavailable) — Drop checkout :443 egress → SM/STS/external HTTPS unreachable (managed-dep network partition; SQS/RDS-endpoint/Elastic proxy)
  - Autonomous remediation: Restore the :443 egress rule in the checkout NetworkPolicy.
- **dns_failure** (critical · class: dependency_unavailable) — Drop checkout :53 (DNS) egress → name resolution fails for postgres Service + SM hostname (UnknownHostException)
  - Autonomous remediation: Restore DNS (:53) egress / fix the pod dnsConfig.
- **redis_egress_block** (warning · class: dependency_unavailable) — Drop checkout-java :6379 egress → Redis cache unreachable → Lettuce RedisCommandTimeoutException on /checkout (503)
  - Autonomous remediation: Restore the :6379 egress rule in checkout-java-netpol (or fix the cache route).
- **kafka_egress_block** (critical · class: dependency_unavailable) — Drop checkout-java :9092 egress → Kafka broker unreachable → org.apache.kafka TimeoutException on /checkout (503)
  - Autonomous remediation: Restore the :9092 egress rule in checkout-java-netpol (or fix the broker route).
- **karpenter_capacity** (warning · class: capacity_exhaustion) — Workload requests more CPU than any NodePool instance type → Karpenter can't provision → pods Pending (FailedScheduling)
  - Autonomous remediation: Reduce the request to a satisfiable size, or widen the NodePool instance-type/limits.
- **jvm_oom** (critical · class: resource_exhaustion) — POST /admin/oom — JVM heap leak → java.lang.OutOfMemoryError → pod OOMKilled
  - Autonomous remediation: Stop the leak / restart the pod; right-size the JVM heap (-Xmx) and the container memory limit.
- **latency_brownout** (critical · class: resource_exhaustion) — POST /admin/latency — per-request handler delay → /checkout p95 climbs + thread pool saturates (pod stays Ready)
  - Autonomous remediation: Remove the injected delay; size the thread pool / add capacity to absorb the latency.
- **failed_prestop_hook** (warning · class: workload_lifecycle) — Pod preStop hook failure during termination (FailedPreStopHook)
  - Autonomous remediation: Fix the preStop hook command, or extend terminationGracePeriodSeconds so the hook reliably completes.
- **failed_daemon_pod** (warning · class: node_lifecycle) — DaemonSet pod scheduling failure during node lifecycle transition (FailedDaemonPod)
  - Autonomous remediation: Correlate with the node lifecycle event (drain, spot interruption, scale-down) and confirm the DaemonSet tolerates it.

## Applicability rules

- k8s → bad_image (high confidence) — K8s workloads pull bad or unavailable container images
- k8s → broken_probe (high confidence) — Liveness/readiness probe misconfiguration crash-loops pods
- k8s → init_container_fail (high confidence) — Init container failures block the main container from starting
- k8s → NEW:pod-not-ready (high confidence) — Pods enter not-ready state from resource constraints or config errors
- k8s → service_zero_endpoints (high confidence) — Services lose all ready endpoints when pods crash or fail probes
- k8s → cpu_squeeze (high confidence) — Container CPU limits cause throttling under burst load
- k8s → pvc_unbound (high confidence) — PersistentVolumeClaims fail to bind (storage class unavailable, quota)
- k8s → unschedulable_cpu (high confidence) — Pods fail to schedule when nodes lack available CPU or memory
- k8s → rbac_denied (high confidence) — K8s RBAC misconfigurations block service accounts from accessing resources
- k8s → configmap_drift (low confidence) — ConfigMap / env drift causes silent misconfiguration at rollout
- k8s → hpa_cannot_scale (low confidence) — HPA fails to scale out when metrics are absent or a PDB blocks rollout
- k8s → pdb_blocking_rollout (low confidence) — PodDisruptionBudget blocks rolling updates when min-available is tight
- k8s → netpol_block (low confidence) — NetworkPolicy rules accidentally block inter-service traffic
- k8s → cert_expired (low confidence) — TLS certificates expire and cause connection failures across any HTTPS service
- eks → karpenter_capacity (high confidence) — Karpenter node provisioning fails under spot interruption or capacity shortage
- eks → karpenter_runinstances_deny (low confidence) — IAM denies ec2:RunInstances — Karpenter nodes fail to provision
- microservices → dns_failure (high confidence) — Service-to-service calls rely on DNS; CoreDNS overload causes cascading failures
- microservices → dependency_egress_block (high confidence) — Outbound HTTP timeouts / connection resets propagate across service boundaries
- microservices → latency_brownout (low confidence) — Upstream latency cascades into p95 brownouts across service boundaries
- rds → db_conn_exhaustion (high confidence) — RDS max_connections exhausted under concurrent load
- rds → rds_failover (high confidence) — Multi-AZ failover causes brief database unavailability
- rds → sg_db_block (low confidence) — Security group rules can block application → RDS port connectivity
- rds → NEW:db-lock-contention (high confidence) — Postgres lock contention under concurrent writes
- rds + s3 → postgres_s3_deny (high confidence) — RDS WAL archiving fails when IAM denies s3:PutObject on the backup bucket
- sqs → sqs_consumer_stall (high confidence) — Queue backlog grows when consumers are too slow or crash
- sqs → sqs_poison_messages (low confidence) — Malformed messages cause consumer crash loops and queue growth
- sqs → sqs_receive_deny (low confidence) — IAM denies sqs:ReceiveMessage — consumer fails silently
- sqs + k8s → dependency_egress_block (low confidence) — NetworkPolicy or SG rules block pod → SQS endpoint connectivity
- sqs → NEW:sqs-dlq-overflow (low confidence) — Dead-letter queue fills up when poison messages are not retried or purged
- iam → sg_rule_block (low confidence) — Security group misconfiguration blocks inbound or outbound service traffic
- iam → checkout_sm_deny (low confidence) — SecretsManager IAM denials block services from fetching credentials at startup
- java → jvm_oom (high confidence) — JVM heap exhaustion under memory pressure causes OOMKilled pod crashes
- elasticache → latency_brownout (low confidence) — Redis connection saturation or eviction causes upstream latency brownouts
- elasticache → redis_egress_block (low confidence) — Security group / network rules block application → Redis connectivity
- alb → elb_unhealthy (high confidence) — ALB target groups become unhealthy when pods fail health checks
- alb → latency_brownout (low confidence) — ALB target group health issues increase p95 request latency
- cloudfront → latency_brownout (low confidence) — CDN edge cache misses or origin errors cause latency spikes
- opensearch → latency_brownout (low confidence) — OpenSearch query latency spikes under shard pressure or JVM GC pauses
- opensearch → dependency_egress_block (low confidence) — Network / SG rules can block application → OpenSearch connectivity
- eks + rds → sts_stuck_pvc (low confidence) — EBS CSI / STS token expiry causes PVC mount failures on pod restart
- k8s + cluster_medium → pdb_blocking_rollout (high confidence) — At medium scale (multi-replica services), PodDisruptionBudgets frequently block rolling updates under load
- k8s + cluster_medium → hpa_cannot_scale (high confidence) — At medium scale, HPA scaling failures during traffic spikes cause sustained saturation
- eks + cluster_medium → karpenter_capacity (high confidence) — At medium scale (on-demand + Spot mix), Karpenter capacity shortages become a common incident source
- k8s + cluster_large → unschedulable_cpu (high confidence) — At large scale (~50k pods), node bin-packing pressure causes frequent scheduling failures
- k8s + cluster_large → pdb_blocking_rollout (high confidence) — At large scale (1000+ services), PDB misconfigurations routinely block cluster-wide rollouts
- eks + cluster_large → karpenter_capacity (high confidence) — At large scale (multi-cloud, multi-region), node provisioning bottlenecks and spot interruptions are P1 incidents
- sqs + cluster_large → sqs_consumer_stall (high confidence) — At large scale (high-throughput queues), consumer stalls cause rapid queue depth growth with blast radius
- xray + k8s → dependency_egress_block (low confidence) — X-Ray daemon (UDP 2000) can be blocked by NetworkPolicy, silently dropping trace segments
- xray + microservices → NEW:xray-segment-loss (high confidence) — Trace segment overflow or misconfigured sampling rate causes trace data loss under high traffic
- bedrock → latency_brownout (high confidence) — LLM inference adds significant p95 latency; any calling service inherits that spike
- bedrock → NEW:bedrock-throttle (high confidence) — Bedrock per-model TPS limits trigger 429 ThrottlingException under concurrent inference load
- bedrock + microservices → NEW:bedrock-model-timeout (high confidence) — Inference latency exceeds calling service read-timeout, causing cascading upstream failure
- bedrock → dependency_egress_block (low confidence) — VPC endpoint misconfig or SG rules can block pod → Bedrock API endpoint connectivity
- bedrock + iam → rbac_denied (low confidence) — Missing bedrock:InvokeModel IAM permission causes silent model call failures
- elastic_cloud → latency_brownout (high confidence) — Elasticsearch query latency spikes under shard pressure, JVM GC, or hot threads
- elastic_cloud → dependency_egress_block (low confidence) — Network / SG rules can block application → Elastic Cloud endpoint connectivity
- elastic_cloud → jvm_oom (low confidence) — Elasticsearch JVM heap exhaustion under heavy indexing or query load
- elastic_cloud → NEW:elastic-shard-failure (high confidence) — Shard allocation failures or replica degradation cause partial data unavailability
- mongodb → db_conn_exhaustion (high confidence) — MongoDB connection pool exhaustion under concurrent client load
- mongodb → latency_brownout (low confidence) — MongoDB query latency spikes from missing indexes, lock contention, or RS elections
- mongodb + k8s → pvc_unbound (high confidence) — MongoDB data PVC fails to bind on pod restart (EBS AZ mismatch, storage class)
- mongodb + k8s → NEW:pod-not-ready (low confidence) — MongoDB pods enter not-ready during RS re-election or PVC mount delays
- mongodb → NEW:mongodb-replicaset-failover (high confidence) — ReplicaSet primary election causes a brief write-unavailability window
- mongodb → NEW:mongodb-lock-contention (low confidence) — Global write lock contention under high concurrency causes query queuing
- k8s + msk → kafka_egress_block (high confidence) — NetworkPolicy or SG rules block pod → MSK broker egress