Terraform Remote State with S3 and DynamoDB — Team-Safe Backends

Overview and What You Will Learn

Local state — a terraform.tfstate file sitting on your laptop — works fine when you are the only engineer. The moment a second engineer exists, local state becomes dangerous. They have a different copy, or no copy at all. Two different state files means two different views of reality. Concurrent applies corrupt infrastructure.

Remote state on S3 with DynamoDB locking solves this completely. The state file lives in S3. Every engineer and every CI/CD pipeline reads the same file. DynamoDB ensures only one apply runs at a time. S3 versioning means any accidental corruption is recoverable.

This is the production standard for Terraform on AWS. By the end of this lab you will have:

• Created the S3 bucket and DynamoDB table for state storage and locking
• Configured the S3 backend in your Terraform configuration
• Migrated existing local state to the remote backend
• Understood partial backend configuration for multi-environment setups
• Set up correct IAM permissions for the Terraform apply role

Why This Matters in Production

At Swiggy, before adopting remote state, two engineers applying to the same environment caused a state file corruption that took half a day to recover from. One engineer had the "correct" state. The other had an older copy. When both applied, the second write overwrote the first — resources were orphaned and the state no longer matched reality.

After moving to S3 remote state with DynamoDB locking, the problem became structurally impossible. The first apply acquires the DynamoDB lock. The second engineer's apply sees the lock and waits. When the first completes, the lock is released and the second can proceed safely — reading the updated state file.

Core Principles

The Remote State Architecture

Detailed Step-by-Step Practical Lab

Step 1 — Create the Bootstrap Resources

You need an S3 bucket and a DynamoDB table before you can configure the remote backend. These are created once — manually or with a separate bootstrap configuration that you never store in the state bucket itself.

Alternatively, manage the bootstrap resources as Terraform — in a separate configuration that stores its own state locally (bootstrapping the bootstrapper):

Step 2 — Configure the S3 Backend

Add the backend block to your main configuration's versions.tf:

Step 3 — Run terraform init to Migrate State

After adding the backend block, run terraform init. Terraform detects the new backend and offers to migrate any existing local state:

Step 4 — Set Up IAM Permissions

The IAM role or user that runs Terraform needs specific permissions on the S3 bucket and DynamoDB table:

Step 5 — Partial Backend Configuration for Multiple Environments

Backend blocks cannot use variables — all values must be hardcoded. For multiple environments sharing the same bucket but with different state keys, use partial backend configuration:

This pattern is especially useful in CI/CD pipelines where the environment name is a pipeline variable.

Step 6 — Verify the Setup is Working

Production Best Practices and Common Pitfalls

• Use a separate AWS account for state storage. The state bucket contains state for all your environments. Keeping it in a dedicated tools or shared-services account prevents a compromised dev account from exposing prod state. Terraform assumes the state bucket's account role using cross-account IAM.

• Key naming convention matters. Use a consistent key hierarchy: <environment>/<service>/terraform.tfstate. This makes state files easy to find, easy to restrict with IAM conditions, and easy to audit. A flat structure like terraform.tfstate for every service causes collisions and confusion.

• Enable S3 access logging on the state bucket. Every read and write to the state file is an infrastructure event. S3 access logs let you audit who accessed state and when — valuable during incident response.

• Add prevent_destroy to the state bucket resource. The state bucket is the most critical single resource in your infrastructure setup. One accidental terraform destroy would orphan all managed infrastructure. lifecycle { prevent_destroy = true } makes this impossible.

• Never share the same DynamoDB lock table across different S3 buckets. The lock key includes the full bucket and path — one DynamoDB table can safely serve multiple state files in the same bucket. But mixing state from different buckets in one lock table creates confusion when debugging stuck locks.

• Test state recovery before you need it. Use S3 versioning to restore a previous state version: download an older version, review it, then push it back. Do this in a dev environment once so you know the procedure when a production emergency happens.

Quick Reference and Troubleshooting Commands