Overview and What You Will Learn
This guide covers how to build environment promotion pipelines that deploy automatically to dev and staging, then require explicit human approval before touching production. You will learn the three-environment model used by most production Kubernetes teams, how to manage per-environment configuration with Kustomize overlays and Helm values files, how to implement GitOps-native promotion where the same immutable image digest is promoted across environments rather than rebuilt, and how to set up GitHub Actions environment protection rules so that production deployments require a named reviewer to approve them before anything runs.
Why This Matters in Production
The cost of an environment promotion bug is asymmetric: a bad change deployed to dev breaks one developer's test cycle; the same change reaching production can break every customer. The three-environment model with a manual prod gate gives teams the latency they need -- automated staging deploys mean staging is always close to production without anyone doing it manually, but the human gate at production means a senior engineer has explicitly looked at what is about to go out before it goes out. PhonePe and Swiggy both operate pipeline models where staging is fully automated (every merge to main goes to staging within minutes) and production requires explicit approval with a named approver on record for the compliance and audit trail.
COMMON MISTAKE / WARNING**Common Mistake:** Rebuilding the Docker image for each environment from the same Git commit hash. Rebuilding is not the same artifact: base image updates, dependency resolution, or non-deterministic build steps can all produce a different binary even from the same commit. The immutable artifact should be the same image digest that passed staging -- not a new build from the same source.
Core Principles
Three-environment promotion flow
+------------------------------------------+| Git push triggers CI: build, test, scan |+------------------------------------------+ | v+------------------------------------------+| Tests pass: auto-deploy image to dev || namespace |+------------------------------------------+ | v+------------------------------------------+| Merge to main: promote same image to || staging |+------------------------------------------+ | v+------------------------------------------+| Smoke tests pass on staging: request || prod approval |+------------------------------------------+ | v+------------------------------------------+| Reviewer approves: deploy to production || namespace |+------------------------------------------+Separate app repo and config repo
+------------------------------------------+| APP REPO (code) |+------------------------------------------+| base/: shared manifests || overlays/dev/: dev config || overlays/staging/: staging config |+------------------------------------------+ ^ | v+------------------------------------------+| CONFIG REPO (state) |+------------------------------------------+| overlays/prod/: prod config || replicas, resources, || secrets per env |+------------------------------------------+```textconfig-repo/ base/ deployment.yaml # shared spec service.yaml overlays/ dev/ kustomization.yaml # image tag, replica: 1, debug env vars staging/ kustomization.yaml # image tag, replica: 2, staging secrets ref prod/ kustomization.yaml # image tag, replica: 10, prod secrets refkustomization.yaml in each overlay patches only what differs -- the base
manifest is not duplicated. The image tag is the only field CI needs to update
on each promotion.
# overlays/staging/kustomization.yamlapiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomizationresources: * ../../baseimages: * name: order-gateway newTag: "a3f8c12" # the immutable commit SHA, not 'latest'patches: * patch: |- * op: replace path: /spec/replicas value: 2 target: kind: Deployment name: order-gatewayCOMMON MISTAKE / WARNING**Security:** Use image digest (`@sha256:...`) rather than an image tag in production overlays where possible. Tags are mutable -- `v1.4.2` can be force-pushed to point at a different image. Digests are not.
GitHub Actions environment protection rules
GitHub Environments can require one or more named reviewers to approve a deployment before the job runs. The approval record (who approved, when, which SHA) is stored in GitHub's deployment history and is queryable via the API.
jobs: deploy-staging: runs-on: ubuntu-latest environment: staging # no protection rules on staging steps: * uses: actions/checkout@v4 * name: Update staging overlay image tag run: | cd config-repo/overlays/staging kustomize edit set image order-gateway=\ 418773912004.dkr.ecr.ap-south-1.amazonaws.com/order-gateway:${{ github.sha }} git commit -am "staging: deploy ${{ github.sha }}" git push deploy-production: needs: deploy-staging runs-on: ubuntu-latest environment: production # requires reviewer approval in GitHub settings steps: * uses: actions/checkout@v4 * name: Update production overlay image tag run: | cd config-repo/overlays/prod kustomize edit set image order-gateway=\ 418773912004.dkr.ecr.ap-south-1.amazonaws.com/order-gateway:${{ github.sha }} git commit -am "prod: promote ${{ github.sha }}" git pushImage promotion vs image rebuild
# CI pipeline -- runs once per commit* name: Build and push image run: | IMAGE=418773912004.dkr.ecr.ap-south-1.amazonaws.com/order-gateway docker buildx build --push \ -t $IMAGE:${{ github.sha }} \ -t $IMAGE:dev \ .# Staging promotion -- same digest, new tag* name: Promote to staging tag run: | IMAGE=418773912004.dkr.ecr.ap-south-1.amazonaws.com/order-gateway DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' $IMAGE:${{ github.sha }}) docker buildx imagetools create \ --tag $IMAGE:staging $DIGEST# Production promotion -- same digest again* name: Promote to production tag run: | DIGEST=$(aws ecr describe-images \ --repository-name order-gateway \ --image-ids imageTag=${{ github.sha }} \ --query 'imageDetails[0].imageDigest' --output text) aws ecr batch-get-image \ --repository-name order-gateway \ --image-ids imageDigest=$DIGEST \ --query 'images[0].imageManifest' --output text \ | aws ecr put-image \ --repository-name order-gateway \ --image-tag production \ --image-manifest file:///dev/stdinPLACEMENT PRO TIP**Tip:** Tag images with the Git commit SHA as the primary tag, then promote by adding env-named tags (dev, staging, production) to the same digest. The SHA tag is permanent and auditable; the env tags are mutable pointers you can query to know exactly what is running where.
Drift detection
In a GitOps setup, ArgoCD will detect and alert on any divergence between the
production overlay in Git and what is actually running in the cluster -- so a
manual kubectl apply or an emergency patch applied directly gets flagged
immediately. Without GitOps, drift detection requires a scheduled job that
compares the running image digest with the expected tag from the config repo.
Promotion audit trail
Every promotion generates an auditable record:
- GitHub Actions deployment history -- who triggered, which SHA, which environment, which reviewer approved and when
- Git commit on the config repo -- what changed, authored by CI bot
- ArgoCD sync history -- when the cluster converged to the new state
Together these three form a complete chain of custody from "who wrote this code" to "when did it reach production and who approved it".
Detailed Step-by-Step Practical Lab
This lab builds a full three-environment promotion pipeline for Swiggy's
order-gateway service using GitHub Actions, Kustomize overlays, and GitHub
environment protection rules.
Milestone 1 — Set up the config repo structure
mkdir -p config-repo/overlays/{dev,staging,prod}cp base/deployment.yaml config-repo/base/cp base/service.yaml config-repo/base/ cat > config-repo/base/kustomization.yaml << 'EOF'apiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomizationresources: * deployment.yaml * service.yamlEOFAt this point you have the shared base and three environment overlay directories ready for per-environment patches.
Milestone 2 — Define dev overlay
# config-repo/overlays/dev/kustomization.yamlapiVersion: kustomize.config.k8s.io/v1beta1kind: Kustomizationresources: * ../../baseimages: * name: order-gateway newTag: "placeholder"patches: * patch: |- * op: replace path: /spec/replicas value: 1 target: kind: Deployment name: order-gatewayAt this point the dev overlay is defined. CI will update newTag on every
commit to main.
Milestone 3 — Create GitHub environments with protection rules
In your GitHub repo Settings -> Environments:
- Create
dev-- no protection rules - Create
staging-- no protection rules - Create
production-- add Required Reviewers: list the on-call engineer or the team lead; optionally add a 5-minute wait timer
At this point GitHub will pause any workflow step referencing the production
environment until a named reviewer approves it.
Milestone 4 — Write the CI workflow with three deploy jobs
name: Build and Promote on: push: branches: [main] jobs: build: runs-on: ubuntu-latest outputs: image-tag: ${{ github.sha }} steps: * uses: actions/checkout@v4 * uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::418773912004:role/cicd-ecr-push aws-region: ap-south-1 * uses: aws-actions/amazon-ecr-login@v2 * run: | docker buildx build --push \ -t 418773912004.dkr.ecr.ap-south-1.amazonaws.com/order-gateway:${{ github.sha }} . deploy-dev: needs: build environment: dev runs-on: ubuntu-latest steps: * uses: actions/checkout@v4 with: repository: swiggy-platform/config-repo token: ${{ secrets.CONFIG_REPO_PAT }} * run: | cd overlays/dev kustomize edit set image \ order-gateway=418773912004.dkr.ecr.ap-south-1.amazonaws.com/order-gateway:${{ github.sha }} git config user.email "ci@swiggy.in" git config user.name "CI Bot" git commit -am "dev: ${{ github.sha }}" && git push deploy-staging: needs: deploy-dev environment: staging runs-on: ubuntu-latest steps: * uses: actions/checkout@v4 with: repository: swiggy-platform/config-repo token: ${{ secrets.CONFIG_REPO_PAT }} * run: | cd overlays/staging kustomize edit set image \ order-gateway=418773912004.dkr.ecr.ap-south-1.amazonaws.com/order-gateway:${{ github.sha }} git config user.email "ci@swiggy.in" git config user.name "CI Bot" git commit -am "staging: ${{ github.sha }}" && git push deploy-production: needs: deploy-staging environment: production runs-on: ubuntu-latest steps: * uses: actions/checkout@v4 with: repository: swiggy-platform/config-repo token: ${{ secrets.CONFIG_REPO_PAT }} * run: | cd overlays/prod kustomize edit set image \ order-gateway=418773912004.dkr.ecr.ap-south-1.amazonaws.com/order-gateway:${{ github.sha }} git config user.email "ci@swiggy.in" git config user.name "CI Bot" git commit -am "prod: ${{ github.sha }}" && git pushAt this point the full promotion pipeline exists, with dev and staging deploying automatically and production blocked on human approval.
Milestone 5 — Verify staging smoke tests before production gate
Add a smoke test step to the staging job before the config repo update:
deploy-staging: needs: deploy-dev environment: staging runs-on: ubuntu-latest steps: * name: Wait for ArgoCD staging sync run: | argocd app wait order-gateway-staging \ --health --timeout 120 * name: Smoke test staging run: | STATUS=$(curl -s -o /dev/null -w "%{http_code}" \ https://staging.order-gateway.swiggy.in/healthz) test "$STATUS" = "200"At this point staging must be healthy before the production approval request even appears in the GitHub UI.
Milestone 6 — Verify audit trail after a production deploy
# List recent deployments with approver info via GitHub APIgh api repos/swiggy-platform/order-gateway/deployments \ --jq '.[] | {ref:.ref, environment:.environment, created_at:.created_at}' # Check ArgoCD sync historyargocd app history order-gateway-production # Check config repo git loggit -C config-repo log --oneline overlays/prod/At this point you have a queryable audit trail covering every production promotion: when, what SHA, who approved it.
REMEMBER THIS**Remember:** The GitHub environment protection approval is tied to the workflow run, not the commit -- if the workflow is re-run after approval, the reviewer must approve again. This is the correct security behaviour; a new run could be picking up different secrets or a different runner environment.
Production Best Practices & Common Pitfalls
- Use image digests in production overlays rather than mutable tags -- tags can be overwritten, digests cannot.
- Keep the config repo as a separate repository from the app repo -- mixing application code and cluster state in the same repo makes the GitOps diff signal noisy and complicates access control.
- Set a wait timer on the production environment protection rule (five minutes is common) to give the team a window to pull a promotion if something is noticed at the last second after approving.
- Automate staging smoke tests before requesting production approval -- do not make the human reviewer check staging manually before they can approve prod.
- Use
kustomize edit set imagerather thansedfor updating image tags in overlays -- sed is fragile if the tag format changes; kustomize edit is structured and idempotent.
Quick Reference & Troubleshooting Commands
| Symptom | Command | What to Look For |
|---|---|---|
| Production deploy waiting for approval | Check GitHub UI Actions -> the pending deploy job | Named reviewer needs to click Approve |
| Staging running different image than expected | kubectl get deployment order-gateway -n staging -o jsonpath='{.spec.template.spec.containers[0].image}' |
Tag mismatch -- overlay not updated or ArgoCD sync pending |
| ArgoCD shows staging OutOfSync | argocd app get order-gateway-staging |
Config repo updated but ArgoCD sync not triggered -- enable auto-sync |
| Image not found on deploy | aws ecr describe-images --repository-name order-gateway --image-ids imageTag=$SHA |
Build job did not push, or SHA var was empty in the push step |