Overview and What You Will Learn
Running a web application locally used to mean installing PostgreSQL, Redis, and RabbitMQ on your laptop, managing three different service configs, and hoping nothing conflicted with another project. Docker Compose replaces all of that with a single YAML file and a single command.
In this guide you will learn the complete Docker Compose file schema, how services, networks, and volumes work together, and every essential CLI command for managing a Compose application.
Core Principles
+------------------------------------------+| docker-compose.yml || || services: <- what to run || api: <- one container || postgres: <- another container || redis: <- another container || || networks: <- how they connect || app-network: || || volumes: <- where data lives || postgres-data: |+------------------------------------------+ | | docker compose up -d | v+------------------------------------------+| Three containers running || Connected on app-network || Data persisted in postgres-data volume || api reaches postgres by name |+------------------------------------------+Detailed Step-by-Step Practical Lab
Milestone 1: The Complete docker-compose.yml Schema
# docker-compose.ymlversion: "3.8" # Compose file format version — use 3.8 for most projects services: # ── Service Definition ───────────────────────────── payment-api: # What image to use (or build) image: registry.razorpay.in/payment-api:v3.1.0 # Or build from a Dockerfile build: context: . # Build context directory dockerfile: Dockerfile # Which Dockerfile (default: Dockerfile) args: BUILD_ENV: production # Container name (otherwise: projectname_servicename_1) container_name: payment-api # Port mapping: host:container ports: - "8080:8080" - "127.0.0.1:9090:9090" # Only localhost can access this port # Environment variables environment: NODE_ENV: production DB_HOST: postgres # Use service name as hostname DB_PORT: "5432" DB_NAME: payments # Load env vars from a file (keeps secrets out of compose file) env_file: - .env - .env.production # Volume mounts volumes: - ./config:/app/config:ro # Bind mount, read-only - payment-data:/app/data # Named volume - type: tmpfs target: /tmp # In-memory storage # Which networks this service joins networks: - app-network # Service dependency ordering depends_on: postgres: condition: service_healthy # Wait until postgres is healthy redis: condition: service_started # Just wait until redis starts # Restart policy restart: unless-stopped # Resource limits deploy: resources: limits: cpus: "1.5" memory: 512M reservations: cpus: "0.25" memory: 128M # Health check healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/health"] interval: 30s timeout: 5s retries: 3 start_period: 15s # Override the default command command: ["node", "dist/server.js", "--port", "8080"] # ── Database Service ─────────────────────────────── postgres: image: postgres:15-alpine environment: POSTGRES_DB: payments POSTGRES_USER: api_user POSTGRES_PASSWORD_FILE: /run/secrets/db-password # Secret from file volumes: - postgres-data:/var/lib/postgresql/data - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql # Init script networks: - app-network healthcheck: test: ["CMD-SHELL", "pg_isready -U api_user -d payments"] interval: 10s timeout: 5s retries: 5 # ── Cache Service ────────────────────────────────── redis: image: redis:7-alpine command: redis-server --appendonly yes # Enable persistence volumes: - redis-data:/data networks: - app-network # ── Networks ─────────────────────────────────────────networks: app-network: driver: bridge # ── Volumes ──────────────────────────────────────────volumes: postgres-data: # Docker creates and manages this redis-data: payment-data:Milestone 2: Essential Compose CLI Commands
# Start all services in backgrounddocker compose up -d # Start and rebuild images firstdocker compose up -d --build # Stop and remove containers (keeps volumes and networks)docker compose down # Stop, remove containers, AND remove named volumes (DELETES DATA)docker compose down -v # View status of all servicesdocker compose ps# NAME IMAGE STATUS PORTS# payment-api payment-api Up 5 minutes 0.0.0.0:8080->8080/tcp# postgres postgres:15 Up 5 minutes (healthy)# redis redis:7 Up 5 minutes # View logs from all servicesdocker compose logs # Follow logs from a specific servicedocker compose logs -f payment-api # Run a command in a running service containerdocker compose exec postgres psql -U api_user -d payments # Run a one-off command (starts a new container)docker compose run --rm payment-api npm run migrate # Scale a service (run multiple instances)docker compose up -d --scale payment-api=3 # Restart a specific servicedocker compose restart payment-api # Pull latest images for all servicesdocker compose pull # Build all services that have a build contextdocker compose build # See resource usage of running servicesdocker compose topMilestone 3: Compose v1 vs Compose v2
# Compose v1: separate binary (docker-compose)# Installed separately, uses Python, slowerdocker-compose up -d # Old way # Compose v2: built into Docker CLI as a plugin# Faster, Go-based, installed with Docker Desktop and Docker Engine 20.10+docker compose up -d # New way (space, not hyphen) # Check which version you havedocker compose version# Docker Compose version v2.23.0 # The compose file format is the same — only the binary changed# Always use 'docker compose' (v2) on modern systemsCommon Mistakes
| Mistake | Consequence | Fix |
|---|---|---|
Using depends_on without health checks |
Service starts before dependency is ready | Add condition: service_healthy with a healthcheck |
| Hardcoding secrets in compose file | Secrets in version control | Use env_file with .env in .gitignore |
| No resource limits | One service starves others | Always set deploy.resources.limits |
docker compose down -v by accident |
All database data deleted permanently | Know exactly what -v does before using it |
Using container_name with scaling |
Conflict when scaling to multiple instances | Remove container_name for services you want to scale |
PLACEMENT PRO TIP**Tip:** Use `docker compose config` to see the fully resolved compose file with all variable substitutions applied. This is the fastest way to verify that your `.env` file variables are being read correctly and that all service configurations look as expected.
REMEMBER THIS**Remember:** `docker compose down` keeps your volumes (your data is safe). `docker compose down -v` deletes your volumes (your data is gone permanently). This is the most common cause of accidental data loss with Docker Compose.
{ "title": "Docker Compose for Local Development — Full Stack in One Command", "slug": "docker-compose-local-development", "cluster": "docker", "description": "Build a complete local development stack with Docker Compose — API server, PostgreSQL, Redis, and a message queue — with hot reloading and proper config.", "primaryKeyword": "docker compose local development"}Docker Compose for Local Development — Full Stack in One Command
Overview and What You Will Learn
Every developer at CRED or Zerodha runs a complex application stack locally. Instead of installing PostgreSQL, Redis, and RabbitMQ on every developer's laptop — and dealing with version conflicts, OS differences, and "works on my machine" bugs — the team uses a single docker compose up command that gives every developer the exact same environment.
Core Principles
The ideal local development Compose file has three properties: hot reloading (code changes appear instantly without rebuild), isolated data (each developer has their own database), and production parity (the same images and config as production, with minor development overrides).
Complete Local Development Stack
# docker-compose.yml — base configurationversion: "3.8" services: api: build: context: . dockerfile: Dockerfile.dev # Development dockerfile with dev tools ports: - "3000:3000" - "9229:9229" # Node.js debugger port volumes: - ./src:/app/src # Bind mount for hot reloading - /app/node_modules # Keep container's node_modules environment: NODE_ENV: development DB_HOST: postgres REDIS_HOST: redis AMQP_URL: amqp://rabbitmq:5672 depends_on: postgres: condition: service_healthy redis: condition: service_started command: ["npm", "run", "dev"] # Dev command with nodemon/ts-node postgres: image: postgres:15-alpine ports: - "5432:5432" # Expose for local DB GUI (TablePlus, etc.) environment: POSTGRES_DB: myapp_dev POSTGRES_USER: developer POSTGRES_PASSWORD: devpassword123 volumes: - postgres-dev-data:/var/lib/postgresql/data - ./db/seed.sql:/docker-entrypoint-initdb.d/seed.sql healthcheck: test: ["CMD-SHELL", "pg_isready -U developer -d myapp_dev"] interval: 5s timeout: 3s retries: 5 redis: image: redis:7-alpine ports: - "6379:6379" # Expose for redis-cli on host rabbitmq: image: rabbitmq:3-management-alpine ports: - "5672:5672" # AMQP port - "15672:15672" # Management UI — http://localhost:15672 environment: RABBITMQ_DEFAULT_USER: developer RABBITMQ_DEFAULT_PASS: devpassword123 volumes: - rabbitmq-dev-data:/var/lib/rabbitmq mailhog: image: mailhog/mailhog # Email capture for development ports: - "1025:1025" # SMTP port - "8025:8025" # Web UI — http://localhost:8025 # Send emails to localhost:1025, view them in browser at :8025 volumes: postgres-dev-data: rabbitmq-dev-data:# Dockerfile.dev — development-specificFROM node:20-alpineWORKDIR /app # Install development toolsRUN apk add --no-cache git curl COPY package.json package-lock.json ./RUN npm install # Install ALL deps including devDependencies # Source is mounted via bind mount — not copied hereEXPOSE 3000 9229CMD ["npm", "run", "dev"] # nodemon watches for changesHot Reloading Setup
# The critical volume mounts for hot reloading:volumes: - ./src:/app/src # Your source code mounted into container - /app/node_modules # Anonymous volume — keeps container's node_modules # Without this, host's node_modules overrides container's # (Different OS = different native modules)// package.json{ "scripts": { "dev": "nodemon --watch src --ext ts,js,json --exec ts-node src/index.ts" }}# Start the full dev stackdocker compose up -d # Watch API logs while developingdocker compose logs -f api # Run database migrationsdocker compose exec api npm run migrate # Reset the databasedocker compose exec postgres psql -U developer -c "DROP DATABASE myapp_dev;"docker compose exec postgres psql -U developer -c "CREATE DATABASE myapp_dev;"docker compose restart postgres # Open a postgres shelldocker compose exec postgres psql -U developer -d myapp_devOverride File for Development
# docker-compose.override.yml# Auto-merged with docker-compose.yml when both are presentversion: "3.8" services: api: environment: DEBUG: "*" # Enable all debug logging LOG_LEVEL: debug volumes: - ./tests:/app/tests # Mount test files for running tests inside containerPLACEMENT PRO TIP**Tip:** Expose all service ports in your local development Compose file (postgres 5432, redis 6379, rabbitmq 15672) so you can connect with GUI tools like TablePlus, RedisInsight, and the RabbitMQ management UI. Remove these port exposures in production compose files — external access is not needed.
REMEMBER THIS**Remember:** The anonymous volume trick (`- /app/node_modules`) is critical on macOS and Windows. Without it, your macOS node_modules (compiled for macOS) override the container's node_modules (compiled for Linux), and native modules fail at runtime.
{ "title": "Docker Compose Health Checks and Dependency Ordering", "slug": "docker-compose-healthchecks-dependencies", "cluster": "docker", "description": "Configure service health checks and correct startup ordering in Docker Compose to prevent applications from starting before their dependencies are ready.", "primaryKeyword": "docker compose health check"}Docker Compose Health Checks and Dependency Ordering
Overview and What You Will Learn
The most common Docker Compose failure pattern: your API starts, tries to connect to PostgreSQL, gets connection refused because PostgreSQL is still initialising, crashes, and gets stuck in a restart loop. The fix is health checks combined with depends_on condition: service_healthy.
Core Principles
WRONG: depends_on without health check postgres starts -> api starts immediately postgres still initialising (takes 3-5 seconds) api tries to connect -> connection refused api crashes -> restart loop CORRECT: depends_on with service_healthy postgres starts postgres health check runs every 5s postgres becomes healthy (pg_isready returns 0) THEN api starts api connects successfully -> no crashHealth Check Configuration
version: "3.8" services: api: image: registry.razorpay.in/payment-api:v3.1.0 depends_on: postgres: condition: service_healthy # Wait until postgres passes health check redis: condition: service_healthy rabbitmq: condition: service_healthy postgres: image: postgres:15-alpine environment: POSTGRES_DB: payments POSTGRES_USER: api_user POSTGRES_PASSWORD: secret healthcheck: test: ["CMD-SHELL", "pg_isready -U api_user -d payments"] interval: 5s # Run health check every 5 seconds timeout: 3s # Health check must complete in 3 seconds retries: 5 # Fail after 5 consecutive failures start_period: 10s # Wait 10s after start before first check # Prevents false failures during initialisation redis: image: redis:7-alpine healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 5s timeout: 3s retries: 3 rabbitmq: image: rabbitmq:3-alpine healthcheck: test: ["CMD", "rabbitmq-diagnostics", "ping"] interval: 10s timeout: 5s retries: 5 start_period: 30s # RabbitMQ takes longer to startHealth Check Command Examples
# HTTP endpoint check (requires curl in image)healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/health"] # Use wget if curl is not available (Alpine images)healthcheck: test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/health"] # PostgreSQL readinesshealthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] # MySQL readinesshealthcheck: test: ["CMD", "mysqladmin", "ping", "-h", "localhost"] # Redis pinghealthcheck: test: ["CMD", "redis-cli", "ping"] # MongoDB readinesshealthcheck: test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"] # Simple TCP port checkhealthcheck: test: ["CMD-SHELL", "nc -z localhost 8080 || exit 1"] # Custom application check via exit codehealthcheck: test: ["CMD-SHELL", "node /app/scripts/healthcheck.js || exit 1"]Monitoring Health Status
# Check health status of all servicesdocker compose ps# NAME STATUS# api Up 2 minutes (healthy)# postgres Up 3 minutes (healthy)# redis Up 3 minutes (healthy) # If a service is unhealthy:# docker compose ps shows: Up 2 minutes (unhealthy) # See health check output for a containerdocker inspect postgres-container --format \ '{{json .State.Health}}' | jq# {# "Status": "healthy",# "FailingStreak": 0,# "Log": [# {# "Start": "2024-01-15T09:00:00Z",# "End": "2024-01-15T09:00:00.123Z",# "ExitCode": 0,# "Output": "/var/run/postgresql:5432 - accepting connections"# }# ]# } # If health check is failing, the Output field shows whydocker inspect unhealthy-container \ --format '{{(index .State.Health.Log 0).Output}}'# FATAL: database "payments" does not exist# -> Database not created yet — check POSTGRES_DB env varPLACEMENT PRO TIP**Tip:** Always set `start_period` on health checks for services that take time to initialise (like PostgreSQL, RabbitMQ, Elasticsearch). Without `start_period`, health check failures during the normal startup period count toward the failure threshold and can mark the service as unhealthy before it even had a chance to start properly.
COMMON MISTAKE / WARNING**Common Mistake:** Using `depends_on` without `condition: service_healthy`. The basic `depends_on: postgres` only waits for the container to start — not for PostgreSQL to be ready to accept connections. PostgreSQL takes 2-5 seconds after the container starts to become ready. Without `service_healthy`, your API will almost certainly try to connect before PostgreSQL is ready.
{ "title": "Docker Compose in CI/CD Pipelines — GitHub Actions Integration", "slug": "docker-compose-ci-cd", "cluster": "docker", "description": "Use Docker Compose in GitHub Actions CI/CD pipelines to run integration tests against real service dependencies — databases, caches, and message queues.", "primaryKeyword": "docker compose ci cd"}Docker Compose in CI/CD Pipelines — GitHub Actions Integration
Overview and What You Will Learn
Integration tests that mock the database are not real integration tests. Real integration tests hit a real PostgreSQL database, a real Redis cache, and a real message queue — the same services your application uses in production. Docker Compose makes running these real dependencies in GitHub Actions free and reliable.
Core Principles
GitHub Actions runner: ubuntu-latest has Docker pre-installed Your CI workflow: 1. docker compose up -d (start postgres, redis, etc.) 2. Wait for services to be healthy 3. Run your integration tests against real services 4. docker compose down (cleanup) Result: integration tests that test real behavior not mocked behaviorComplete GitHub Actions Workflow
# .github/workflows/ci.ymlname: CI Pipeline on: push: branches: [main, develop] pull_request: branches: [main] jobs: integration-tests: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Start services with Docker Compose run: | docker compose -f docker-compose.ci.yml up -d echo "Services started" - name: Wait for services to be healthy run: | echo "Waiting for postgres..." timeout 60 bash -c \ 'until docker compose -f docker-compose.ci.yml exec -T postgres \ pg_isready -U api_user -d payments; do sleep 2; done' echo "Waiting for redis..." timeout 30 bash -c \ 'until docker compose -f docker-compose.ci.yml exec -T redis \ redis-cli ping | grep -q PONG; do sleep 1; done' echo "All services ready" - name: Run database migrations run: | docker compose -f docker-compose.ci.yml exec -T api \ npm run migrate - name: Run integration tests run: | docker compose -f docker-compose.ci.yml exec -T api \ npm run test:integration env: CI: true - name: Collect logs on failure if: failure() run: | docker compose -f docker-compose.ci.yml logs > ci-logs.txt 2>&1 - name: Upload logs as artifact on failure if: failure() uses: actions/upload-artifact@v4 with: name: ci-logs path: ci-logs.txt - name: Teardown services if: always() # Run even if tests failed run: | docker compose -f docker-compose.ci.yml down -v # -v removes volumes — clean slate for next run# docker-compose.ci.yml — CI-specific compose fileversion: "3.8" services: api: build: . environment: NODE_ENV: test DB_HOST: postgres DB_PORT: "5432" DB_NAME: payments_test DB_USER: api_user DB_PASSWORD: testpassword REDIS_HOST: redis postgres: image: postgres:15-alpine environment: POSTGRES_DB: payments_test POSTGRES_USER: api_user POSTGRES_PASSWORD: testpassword healthcheck: test: ["CMD-SHELL", "pg_isready -U api_user -d payments_test"] interval: 5s timeout: 3s retries: 5 # No volume — data does not need to persist in CI # Faster than using a volume, clean state every run redis: image: redis:7-alpine healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 3s retries: 5 # No volume — clean state every runCaching Docker Layers in CI
# Significantly speeds up CI builds by caching built layers- name: Build with cache uses: docker/build-push-action@v5 with: context: . push: false load: true tags: myapp:test cache-from: type=gha # Read cache from GitHub Actions cache cache-to: type=gha,mode=max # Write new layers to cacheCommon Mistakes
| Mistake | Consequence | Fix |
|---|---|---|
| Not waiting for health checks | Tests fail intermittently — race condition | Use timeout bash -c 'until condition; do sleep 1; done' |
| Using volumes in CI | State bleeds between runs | No volumes in CI compose — in-memory or ephemeral storage |
Not cleaning up with always() |
Runner disk fills up with old containers and volumes | Always use if: always() on teardown step |
| Running tests directly on runner | Inconsistent dependencies between runners | Run tests inside the compose service container |
PLACEMENT PRO TIP**Tip:** Use `docker compose -f docker-compose.ci.yml exec -T service command` (note the `-T` flag) for non-interactive commands in CI. Without `-T`, GitHub Actions may hang waiting for TTY input that never comes.
REMEMBER THIS**Remember:** Add `if: always()` to your teardown step. If tests fail and you do not tear down, the next CI run may fail because ports are already in use from the previous run's containers still running on the same GitHub Actions runner.
{ "title": "Docker Compose Production Patterns — What to Use and What to Avoid", "slug": "docker-compose-production-patterns", "cluster": "docker", "description": "Understand what Docker Compose is appropriate for in production, its limitations compared to Kubernetes, and patterns for small-team production deployments.", "primaryKeyword": "docker compose production"}Docker Compose Production Patterns — What to Use and What to Avoid
Overview and What You Will Learn
Docker Compose is excellent for local development and CI. In production it is appropriate for specific situations — small teams, single-host deployments, internal tools — and completely wrong for others. Understanding where the line is prevents you from building on the wrong foundation.
When Compose Is Appropriate in Production
Compose is GOOD for production when: * Single host deployment (one server) * Team of 1-5 engineers * Non-critical internal tools * Staging / preview environments * Startups with simple infrastructure needs Compose is NOT appropriate when: * Multiple hosts needed (high availability) * Self-healing required (container fails = outage) * Rolling updates with zero downtime required * Horizontal scaling across nodes required * Production traffic is business-criticalProduction-Ready Compose Configuration
# docker-compose.prod.ymlversion: "3.8" services: api: image: registry.razorpay.in/payment-api:v3.1.0 restart: unless-stopped # Restart on crash, not on manual stop ports: - "127.0.0.1:8080:8080" # Only localhost — nginx proxies externally environment: NODE_ENV: production env_file: - .env.production # Production secrets from file logging: driver: "json-file" options: max-size: "100m" # Log rotation — prevents disk fill max-file: "3" # Keep 3 log files deploy: resources: limits: cpus: "1.5" memory: 512M healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/health"] interval: 30s timeout: 5s retries: 3 nginx: image: nginx:1.25-alpine restart: unless-stopped ports: - "80:80" - "443:443" volumes: - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro - ./nginx/ssl:/etc/nginx/ssl:ro - nginx-logs:/var/log/nginx postgres: image: postgres:15-alpine restart: unless-stopped volumes: - postgres-data:/var/lib/postgresql/data logging: driver: "json-file" options: max-size: "50m" max-file: "5" volumes: postgres-data: driver: local nginx-logs:Deploying Updates Without Downtime
Compose does not have built-in rolling updates. This is the pattern for zero-downtime updates:
# Pull new imagesdocker compose pull # Recreate only the service that changed (api, not postgres)docker compose up -d --no-deps api# --no-deps: do not recreate dependencies (postgres, redis)# This pulls new api image and recreates only the api container # Note: there IS a brief downtime during the container recreation# For true zero downtime, use Kubernetes or Traefik with multiple replicas # For simple single-host zero-downtime:# Run nginx in front, bring up new container on different port,# update nginx config, reload nginx (zero downtime), remove old containerCompose vs Kubernetes Decision
| Need | Compose | Kubernetes |
|---|---|---|
| Local development | Excellent | Overkill |
| Single host production | Good | Overkill |
| Multi-host / HA | Cannot do it | Designed for this |
| Self-healing | No (manual restart) | Yes (automatic) |
| Rolling updates (zero downtime) | Manual workaround | Built-in |
| Horizontal autoscaling | No | Built-in (HPA) |
| Service mesh / mTLS | No | Yes (Istio, Linkerd) |
| Complexity | Low | High |
| Ops cost | Very low | Significant |
PLACEMENT PRO TIP**Tip:** Use `unless-stopped` restart policy (not `always`) for production services. `always` restarts even when you intentionally stop the container for maintenance. `unless-stopped` restarts on crash but respects manual `docker stop`.
COMMON MISTAKE / WARNING**Common Mistake:** Using Compose in production for business-critical services without understanding its limitations. If the server goes down, all your containers go down. If a container crashes and the restart fails (e.g., port conflict), the service is down until you SSH in and fix it manually. Plan for this reality before choosing Compose over Kubernetes for production.