AWS DevOps, Cost, and Monitoring

CloudWatch, CloudTrail, CloudFormation, Cost Explorer, and AWS Budgets commands for DevOps monitoring, IaC, and cost management.

CloudWatch — Metrics and Alarms

Create alarms on EC2 CPU and custom metrics.

BASH

## Create CPU alarm with SNS notification
aws cloudwatch put-metric-alarm \
  --alarm-name prod-web-high-cpu \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --dimensions Name=InstanceId,Value=i-0abc123def456789 \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --alarm-actions arn:aws:sns:ap-south-1:123456789012:devops-alerts \
  --ok-actions arn:aws:sns:ap-south-1:123456789012:devops-alerts \
  --treat-missing-data notBreaching \
  --region ap-south-1

## Push custom business metric
aws cloudwatch put-metric-data \
  --namespace "App/Orders" \
  --metric-name OrdersPerMinute \
  --value 1247 \
  --unit Count \
  --dimensions Environment=prod,Region=mumbai \
  --region ap-south-1

## Force alarm state for testing
aws cloudwatch set-alarm-state \
  --alarm-name prod-web-high-cpu \
  --state-value ALARM \
  --state-reason "Testing alarm pipeline" \
  --region ap-south-1

## List alarms in ALARM state
aws cloudwatch describe-alarms \
  --state-value ALARM \
  --query 'MetricAlarms[*].[AlarmName,StateReason]' \
  --output table \
  --region ap-south-1

Parameter Breakdown:

--evaluation-periods 2: Alarm fires only after 2 consecutive breaches — reduces false alerts
--treat-missing-data notBreaching: Missing data is not treated as a breach — handles gaps
--ok-actions: Sends notification when alarm recovers — not just when it fires
Always test with set-alarm-state — untested alarms often have misconfigured SNS targets

CloudWatch Logs Insights

Query logs across multiple log groups.

BASH

## Find all Lambda errors in the last hour
aws logs start-query \
  --log-group-names /aws/lambda/prod-order-processor \
  --start-time $(date -d "1 hour ago" +%s) \
  --end-time $(date +%s) \
  --query-string \
    'fields @timestamp, @message
     | filter @message like /ERROR/
     | sort @timestamp desc
     | limit 50' \
  --region ap-south-1

## Get query results
aws logs get-query-results \
  --query-id QUERY-ID-FROM-ABOVE \
  --region ap-south-1

## Query Lambda cold starts (filter REPORT lines)
aws logs start-query \
  --log-group-names /aws/lambda/prod-order-processor \
  --start-time $(date -d "1 hour ago" +%s) \
  --end-time $(date +%s) \
  --query-string \
    'filter @type = "REPORT"
     | stats avg(@duration), max(@duration),
             avg(@initDuration) by bin(5m)
     | sort by bin desc' \
  --region ap-south-1

Parameter Breakdown:

@initDuration: Present only on cold starts — if null, it was a warm invocation
avg(@duration): Average function execution time per 5-minute window
Combine with bin() to see latency trends over time

CloudTrail — Audit and Query

Enable CloudTrail and query API call history.

BASH

## Create CloudTrail Trail to S3
aws cloudtrail create-trail \
  --name prod-audit-trail \
  --s3-bucket-name prod-cloudtrail-logs-123456789012 \
  --include-global-service-events \
  --is-multi-region-trail \
  --enable-log-file-validation \
  --region ap-south-1

aws cloudtrail start-logging \
  --name prod-audit-trail \
  --region ap-south-1

## Enable S3 Data Events on sensitive bucket
aws cloudtrail put-event-selectors \
  --trail-name prod-audit-trail \
  --event-selectors '[{
    "ReadWriteType": "All",
    "IncludeManagementEvents": true,
    "DataResources": [{
      "Type": "AWS::S3::Object",
      "Values": ["arn:aws:s3:::razorpay-payment-logs/"]
    }]
  }]' \
  --region ap-south-1

## Look up specific event in Event History
aws cloudtrail lookup-events \
  --lookup-attributes \
    AttributeKey=EventName,AttributeValue=DeleteSecurityGroup \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-31T23:59:59Z \
  --query 'Events[*].[EventTime,Username,CloudTrailEvent]' \
  --region ap-south-1

Parameter Breakdown:

--enable-log-file-validation: Detects if log files are tampered with after creation
--is-multi-region-trail: One trail covers all regions — recommended for production
Data Events for S3: NOT logged by default — must explicitly enable for compliance
lookup-events: Searches 90-day Event History — for older events, query S3 with Athena

CloudFormation — Deploy and Manage Stacks

Deploy infrastructure from a YAML template.

BASH

## Validate template before deploying
aws cloudformation validate-template \
  --template-body file://prod-vpc.yaml \
  --region ap-south-1

## Deploy stack
aws cloudformation create-stack \
  --stack-name prod-vpc-stack \
  --template-body file://prod-vpc.yaml \
  --parameters \
    ParameterKey=Environment,ParameterValue=prod \
    ParameterKey=VpcCidr,ParameterValue=10.0.0.0/16 \
  --capabilities CAPABILITY_IAM \
  --on-failure ROLLBACK \
  --region ap-south-1

## Watch deployment events in real time
aws cloudformation describe-stack-events \
  --stack-name prod-vpc-stack \
  --query 'StackEvents[*].[Timestamp,ResourceStatus,ResourceType,LogicalResourceId]' \
  --output table \
  --region ap-south-1

## Create Change Set before updating production
aws cloudformation create-change-set \
  --stack-name prod-vpc-stack \
  --change-set-name add-sg-rule-20240115 \
  --template-body file://prod-vpc-v2.yaml \
  --parameters \
    ParameterKey=Environment,UsePreviousValue=true \
  --region ap-south-1

## Review then execute Change Set
aws cloudformation describe-change-set \
  --change-set-name add-sg-rule-20240115 \
  --stack-name prod-vpc-stack \
  --region ap-south-1

aws cloudformation execute-change-set \
  --change-set-name add-sg-rule-20240115 \
  --stack-name prod-vpc-stack \
  --region ap-south-1

## Detect drift from manual changes
aws cloudformation detect-stack-drift \
  --stack-name prod-vpc-stack \
  --region ap-south-1

Parameter Breakdown:

Change Set: preview exactly what changes before applying — mandatory for production
--on-failure ROLLBACK: Automatically rolls back on any resource creation failure
detect-stack-drift: Finds resources changed outside CloudFormation — fix or update template

EventBridge — Schedules and Rules

Create scheduled tasks and event-based triggers.

BASH

## Nightly cleanup at 2 AM IST (8:30 PM UTC)
aws events put-rule \
  --name nightly-log-cleanup \
  --schedule-expression "cron(30 20 * * ? *)" \
  --state ENABLED \
  --region ap-south-1

aws events put-targets \
  --rule nightly-log-cleanup \
  --targets '[{
    "Id": "cleanup-lambda",
    "Arn": "arn:aws:lambda:ap-south-1:123456789012:function:log-cleanup"
  }]' \
  --region ap-south-1

## Alert when root user logs in
aws events put-rule \
  --name root-login-alert \
  --event-pattern '{
    "source": ["aws.signin"],
    "detail-type": ["AWS Console Sign In via CloudTrail"],
    "detail": {
      "userIdentity": {"type": ["Root"]}
    }
  }' \
  --state ENABLED \
  --region ap-south-1

## Publish custom event from application
aws events put-events \
  --entries '[{
    "EventBusName": "prod-events",
    "Source": "com.swiggy.orders",
    "DetailType": "OrderPlaced",
    "Detail": "{\"orderId\":\"ORD-98765\",\"amount\":450}"
  }]' \
  --region ap-south-1

Parameter Breakdown:

cron(30 20 * * ? *): Every day at 20:30 UTC = 2:00 AM IST
aws.signin + Root userIdentity: Detects root account console login — fire immediately
put-events: Send custom business events to EventBridge from your own application

Cost Explorer and Budgets

Query costs and create budget alerts.

BASH

## Get this month's costs by service
aws ce get-cost-and-usage \
  --time-period \
    Start=$(date +%Y-%m-01),End=$(date +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --query 'ResultsByTime[0].Groups[*].[Keys[0],Metrics.BlendedCost.Amount]' \
  --output table \
  --region us-east-1

## Get EC2 cost by instance type
aws ce get-cost-and-usage \
  --time-period Start=2024-01-01,End=2024-01-31 \
  --granularity MONTHLY \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Elastic Compute Cloud - Compute"]}}' \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=INSTANCE_TYPE \
  --region us-east-1

## Create monthly budget with email alert
aws budgets create-budget \
  --account-id 123456789012 \
  --budget '{
    "BudgetName": "monthly-aws-total",
    "BudgetLimit": {"Amount": "5000","Unit": "USD"},
    "TimeUnit": "MONTHLY",
    "BudgetType": "COST"
  }' \
  --notifications-with-subscribers '[{
    "Notification": {
      "NotificationType": "ACTUAL",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 80,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{
      "SubscriptionType": "EMAIL",
      "Address": "devops@company.in"
    }]
  }]' \
  --region us-east-1

Parameter Breakdown:

Cost Explorer queries must go to us-east-1 regardless of your workload region
Threshold 80: Alert when 80% of budget used — gives time to investigate before hitting limit
ACTUAL vs FORECASTED: ACTUAL alerts on real spend, FORECASTED alerts before reaching limit

AWS CLI — Useful Shortcuts

BASH

## List all running EC2 instances with Name tag
aws ec2 describe-instances \
  --filters Name=instance-state-name,Values=running \
  --query 'Reservations[*].Instances[*].[Tags[?Key==`Name`]|[0].Value,InstanceId,InstanceType,State.Name]' \
  --output table \
  --region ap-south-1

## Find untagged EC2 instances (missing Environment tag)
aws ec2 describe-instances \
  --query 'Reservations[*].Instances[?!Tags[?Key==`Environment`]].[InstanceId]' \
  --output text \
  --region ap-south-1

## Get current IAM caller identity
aws sts get-caller-identity

## Switch to another AWS profile
export AWS_PROFILE=prod-account
aws sts get-caller-identity ## verify you switched correctly

Parameter Breakdown:

--query uses JMESPath — powerful filter language for extracting specific JSON fields
--output table: Human-readable table format for terminal use
AWS_PROFILE: Switch between named profiles in ~/.aws/credentials without reconfiguring

CloudWatch Alarm States Reference

State	Meaning	Action
OK	Metric within threshold	No action needed
ALARM	Threshold breached	SNS fires, ASG scales
INSUFFICIENT_DATA	Not enough data points	Check metric collection