Environment Promotion — Dev to Staging to Production Pipelines

Overview and What You Will Learn

This guide covers how to build environment promotion pipelines that deploy automatically to dev and staging, then require explicit human approval before touching production. You will learn the three-environment model used by most production Kubernetes teams, how to manage per-environment configuration with Kustomize overlays and Helm values files, how to implement GitOps-native promotion where the same immutable image digest is promoted across environments rather than rebuilt, and how to set up GitHub Actions environment protection rules so that production deployments require a named reviewer to approve them before anything runs.

Why This Matters in Production

The cost of an environment promotion bug is asymmetric: a bad change deployed to dev breaks one developer's test cycle; the same change reaching production can break every customer. The three-environment model with a manual prod gate gives teams the latency they need -- automated staging deploys mean staging is always close to production without anyone doing it manually, but the human gate at production means a senior engineer has explicitly looked at what is about to go out before it goes out. PhonePe and Swiggy both operate pipeline models where staging is fully automated (every merge to main goes to staging within minutes) and production requires explicit approval with a named approver on record for the compliance and audit trail.

COMMON MISTAKE / WARNING

**Common Mistake:** Rebuilding the Docker image for each environment from the same Git commit hash. Rebuilding is not the same artifact: base image updates, dependency resolution, or non-deterministic build steps can all produce a different binary even from the same commit. The immutable artifact should be the same image digest that passed staging -- not a new build from the same source.

Core Principles

Three-environment promotion flow

Separate app repo and config repo

kustomization.yaml in each overlay patches only what differs -- the base manifest is not duplicated. The image tag is the only field CI needs to update on each promotion.

COMMON MISTAKE / WARNING

**Security:** Use image digest (`@sha256:...`) rather than an image tag in production overlays where possible. Tags are mutable -- `v1.4.2` can be force-pushed to point at a different image. Digests are not.

GitHub Actions environment protection rules

GitHub Environments can require one or more named reviewers to approve a deployment before the job runs. The approval record (who approved, when, which SHA) is stored in GitHub's deployment history and is queryable via the API.

Image promotion vs image rebuild

PLACEMENT PRO TIP

**Tip:** Tag images with the Git commit SHA as the primary tag, then promote by adding env-named tags (dev, staging, production) to the same digest. The SHA tag is permanent and auditable; the env tags are mutable pointers you can query to know exactly what is running where.

Drift detection

In a GitOps setup, ArgoCD will detect and alert on any divergence between the production overlay in Git and what is actually running in the cluster -- so a manual kubectl apply or an emergency patch applied directly gets flagged immediately. Without GitOps, drift detection requires a scheduled job that compares the running image digest with the expected tag from the config repo.

Promotion audit trail

Every promotion generates an auditable record:

• GitHub Actions deployment history -- who triggered, which SHA, which environment, which reviewer approved and when
• Git commit on the config repo -- what changed, authored by CI bot
• ArgoCD sync history -- when the cluster converged to the new state

Together these three form a complete chain of custody from "who wrote this code" to "when did it reach production and who approved it".

Detailed Step-by-Step Practical Lab

This lab builds a full three-environment promotion pipeline for Swiggy's order-gateway service using GitHub Actions, Kustomize overlays, and GitHub environment protection rules.

Milestone 1 — Set up the config repo structure

At this point you have the shared base and three environment overlay directories ready for per-environment patches.

Milestone 2 — Define dev overlay

At this point the dev overlay is defined. CI will update newTag on every commit to main.

Milestone 3 — Create GitHub environments with protection rules

In your GitHub repo Settings -> Environments:

• Create dev -- no protection rules
• Create staging -- no protection rules
• Create production -- add Required Reviewers: list the on-call engineer or the team lead; optionally add a 5-minute wait timer

At this point GitHub will pause any workflow step referencing the production environment until a named reviewer approves it.

Milestone 4 — Write the CI workflow with three deploy jobs

At this point the full promotion pipeline exists, with dev and staging deploying automatically and production blocked on human approval.

Milestone 5 — Verify staging smoke tests before production gate

Add a smoke test step to the staging job before the config repo update:

At this point staging must be healthy before the production approval request even appears in the GitHub UI.

Milestone 6 — Verify audit trail after a production deploy

At this point you have a queryable audit trail covering every production promotion: when, what SHA, who approved it.

REMEMBER THIS

**Remember:** The GitHub environment protection approval is tied to the workflow run, not the commit -- if the workflow is re-run after approval, the reviewer must approve again. This is the correct security behaviour; a new run could be picking up different secrets or a different runner environment.

Production Best Practices & Common Pitfalls

• Use image digests in production overlays rather than mutable tags -- tags can be overwritten, digests cannot.
• Keep the config repo as a separate repository from the app repo -- mixing application code and cluster state in the same repo makes the GitOps diff signal noisy and complicates access control.
• Set a wait timer on the production environment protection rule (five minutes is common) to give the team a window to pull a promotion if something is noticed at the last second after approving.
• Automate staging smoke tests before requesting production approval -- do not make the human reviewer check staging manually before they can approve prod.
• Use kustomize edit set image rather than sed for updating image tags in overlays -- sed is fragile if the tag format changes; kustomize edit is structured and idempotent.

Quick Reference & Troubleshooting Commands