StackGuardian Business Continuity guide

How to keep managing your infrastructure if StackGuardian becomes unavailable.

Overview

StackGuardian runs your Infrastructure as Code, but it does not hold it hostage. When you use a private runner, every artifact a workflow produces (its sources, its state, its outputs, its inputs and its environment) is written to object storage you own, in your own cloud account.

That is the foundation of business continuity here: the data needed to keep going is already yours, and reading it requires nothing from StackGuardian.

sg-dr is the supported tool that turns those artifacts back into a runnable working directory on your laptop. It never contacts the StackGuardian API, because the situation it exists for is the one where that API is unreachable.

note

Download sg-dr and exercise it before you need it. A disaster recovery tool that has never been run is not a recovery tool.

What "recovery" means in this guide

Recovery means: for a given workflow, reconstruct on your own machine

the exact source tree the last healthy run executed, and
the input variables and environment variables it ran with, and
a Terraform/OpenTofu backend pointing at your existing state,

so that tofu plan produces the same answer StackGuardian would have, and tofu apply continues managing the same resources — no import, no state surgery, no drift.

The three guarantees

Guarantee	Why it matters
Read-only. No code path in `sg-dr` writes, modifies or deletes an object in your bucket or container.	A recovery attempt cannot make an outage worse.
Your credentials only. `sg-dr` authenticates with your AWS profile, your `az login`, your git configuration.	Credentials StackGuardian stored alongside a run are never read — they are not even modelled in the code. Read access to an artifact bucket does not become access to everything the bucket describes.
No StackGuardian dependency. No API call, no license check, no phone home.	The tool works with StackGuardian completely down, and works years later from a binary you archived.

Scope: what can and cannot be recovered

Private runners only

Runner	Recoverable	Why
Private runner	Yes	Artifacts are written to your S3 bucket or Azure container.
Shared runner	No	Artifacts are written to StackGuardian's own storage, which you cannot read.

sg-dr detects a shared-runner workflow and says so plainly rather than failing obscurely later.

note

If business continuity matters for a workflow, run it on a private runner. This is the single most important preparation step in this guide.

By workflow type

Workflow type	Sources & inputs recovered	Run by `sg-dr`
Terraform	Yes	Yes — `--init`, `--plan`
OpenTofu	Yes	Yes — `--init`, `--plan`
Ansible	Yes	No — you run it
Helm	Yes	No — you run it
Kubernetes / kubectl	Yes	No — you run it
CloudFormation	Yes	No — you run it
Pulumi	Yes	No — you run it
Steampipe	Yes	No — you run it
Other custom workflows	Yes	No — you run it

Everything is recovered. Only Terraform and OpenTofu are executed, because running a Helm release or an Ansible play needs access to a cluster or to hosts that only you have. For everything else sg-dr reconstructs the sources, the resolved inputs and the environment, and hands them to you.

What cannot be recovered at all

Secret values. StackGuardian resolves ${secret::…} (its own secret store) and ${ext-secret::azure-kv::…} (an external vault) at run time. Those values are never written to object storage, so no offline tool can obtain them.

sg-dr does not fail at the first one. It collects every unresolvable value into a single sg-secrets.yaml file, annotated with where each one is used, and asks you to fill it in once. See §8.

Before an incident: preparation

Recovery is mostly something you do in advance. During an incident you should be executing a rehearsed procedure, not discovering prerequisites.

Checklist

Business-critical workflows run on private runners.
You know your organization name, and your runner's bucket/container name and region (or storage account). Record them somewhere reachable when StackGuardian is not.
Someone on call has read access to that bucket or container.
Someone on call has credentials for the cloud accounts the workflows deploy to.
Someone on call can clone the source repositories with their own git access.
sg-dr is downloaded, signature-verified and stored somewhere that does not depend on StackGuardian or on a single laptop.
sg-dr verify runs clean, on a schedule.
A recovery drill has been performed end-to-end at least once.

Storage permissions

sg-dr needs read and list access, nothing more.

AWS — attach to the role or user the responder assumes:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SgDrRead",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::YOUR-RUNNER-BUCKET/*"
    },
    {
      "Sid": "SgDrList",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::YOUR-RUNNER-BUCKET"
    }
  ]
}

Azure — the Storage Blob Data Reader role on the storage account or container is sufficient.

note

A later apply needs more. Recovering and planning are read-only. If you go on to apply, Terraform must write the state object back, which requires s3:PutObject on the state key (or Storage Blob Data Contributor on Azure). Grant that separately, and deliberately.

Protect the storage itself

The artifact bucket is now a recovery dependency. Treat it like one:

Enable versioning on the bucket/container, so an accidental or malicious deletion of a state file is recoverable.
Enable cross-region replication if a regional outage is in your threat model.
Restrict deletes — the runner needs write access; almost nobody needs delete access.
Monitor it. An artifact bucket that stops receiving writes is a signal.

Store the binary where you will still have it

Keep a verified copy of the sg-dr archive alongside your incident runbook — an internal artifact repository, a bucket in a second cloud, an encrypted USB drive in a safe. The archive is named for the version it holds (sg-dr_1.0.0_linux_amd64.tar.gz), so a file kept for years still says what it is, and CHANGELOG.md ships inside every archive.

Use a package manager as well, to stay current — a recovery tool three versions old is its own kind of risk.

Installing `sg-dr`

Releases are published at github.com/StackGuardian/sg-dr. No authentication is required to download them, and nothing StackGuardian runs is in the path.

Builds exist for Linux, macOS and Windows, on amd64 and arm64. They are static, so the machine you recover on needs nothing else installed — no Go, no Python, no AWS CLI.

Homebrew (macOS and Linux)

brew trust --tap stackguardian/tap
brew install stackguardian/tap/sg-dr

Homebrew 6.0 requires explicit trust for third-party taps and refuses to load an untrusted one rather than prompting, hence the first command. Trusting the tap covers every StackGuardian formula, now and later. To grant strictly less — Homebrew's own recommendation if sg-dr is all you want:

brew trust --formula stackguardian/tap/sg-dr

Upgrade later with brew upgrade sg-dr.

Scoop (Windows)

scoop bucket add sg-dr https://github.com/StackGuardian/sg-dr
scoop install sg-dr/sg-dr

Archives (every platform)

This is the path to use for the copy you archive, because it is the one you can verify.

VERSION=1.0.0
BASE=https://github.com/StackGuardian/sg-dr/releases/download/v$VERSION

curl -fsSLO $BASE/sg-dr_${VERSION}_linux_amd64.tar.gz
curl -fsSLO $BASE/checksums.txt
curl -fsSLO $BASE/checksums.txt.sig
curl -fsSLO $BASE/checksums.txt.pem

tar xzf sg-dr_${VERSION}_linux_amd64.tar.gz

Verify it before you store it

Homebrew and Scoop pin the same checksums, so they cover integrity. Only this path exercises the signature, which is what establishes where the binary came from.

# 1. Integrity
sha256sum --ignore-missing -c checksums.txt

# 2. Provenance
cosign verify-blob checksums.txt \
  --certificate checksums.txt.pem \
  --signature checksums.txt.sig \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github.com/StackGuardian/business-continuity/\.github/workflows/release\.yml@refs/tags/'

Signing is keyless: the certificate names the workflow and tag that built the release rather than a key someone has to keep, and the signature is recorded in the public Rekor transparency log. This check still works years from now, with StackGuardian unreachable.

Finally, prove the binary runs on the machine you intend to recover from:

sg-dr --version

Review what it touches

Before pointing an unfamiliar binary at your infrastructure:

sg-dr help files

That prints every file, directory, subprocess and network destination the tool uses. It ships inside the binary rather than in a document on purpose — you cannot end up reading a description of a version other than the one you are running.

Connecting to your storage

Every command needs to know your organization and where your runner writes.

AWS S3

sg-dr verify \
  --org acme \
  --bucket acme-runner-storage \
  --region eu-central-1 \
  --profile acme-prod

Azure Blob Storage

az login

sg-dr verify \
  --provider azure \
  --org acme \
  --account acmerunnerstorage \
  --container runner

If your deployment writes under a path inside the bucket or container, add --prefix. This cannot be discovered from storage — the metadata that would name it lives under the prefix — so you supply it.

Environment variables

Every connection flag has an environment variable, so you can export once and drop the flags from every command afterwards. Precedence is flags → environment → defaults.

Flag	Environment variable	Default
`--provider`, `-P`	`STORAGE_PROVIDER`	`aws`
`--org`, `-O`	`ORG_NAME`	—
`--bucket`, `-b`	`BUCKET_NAME`	—
`--region`	`AWS_REGION`	taken from the workflow run when omitted
`--profile`	`AWS_PROFILE`	default credential chain
`--account`, `-a`	`AZURE_STORAGE_ACCOUNT`	—
`--container`, `-c`	`AZURE_CONTAINER_NAME`	`runner`
`--resource-group`, `-R`	`AZURE_RESOURCE_GROUP`	—
`--prefix`	`STORAGE_PREFIX`	none

export ORG_NAME=acme
export BUCKET_NAME=acme-runner-storage
export AWS_REGION=eu-central-1
export AWS_PROFILE=acme-prod

sg-dr verify
sg-dr recover --workflow payments-vpc

AWS credentials come from the standard chain: environment variables, shared config/credentials files, then IMDS. Azure credentials come from DefaultAzureCredential: environment variables, workload identity, managed identity, then your az login session.

Readiness: `sg-dr verify`

This is the command to run before an incident — and on a schedule.

sg-dr verify --org acme --bucket acme-runner-storage --region eu-central-1

It answers the question that actually matters: not "does the binary run", but "if StackGuardian went away right now, which of my workflows could I get back, and why not the rest".

It walks every workflow in the organization and, for each, performs the same checks a real recovery performs, in the same order — so a workflow reported recoverable is one that would recover, and a reason given here is the reason a recovery would give.

Nothing is written, nothing is downloaded, and no state is touched.

Sample output

→ Checking what could be recovered from acme

  storage   s3://acme-runner-storage/
  workflows 7

✓ acme/platform/network-base  (opentofu)
✓ acme/platform/k8s-stack/cluster  (opentofu)
✓ acme/platform/k8s-stack/addons  (helm, recovered but not run by sg-dr)
            needs values only StackGuardian holds; you would have to supply them
✓ acme/apps/payments-api  (terraform)
            private repository, so a clone would need your git access
Error: acme/apps/legacy-batch
            ran on a shared runner, so its artifacts are in StackGuardian's storage

Error: 1 of 7 workflows cannot be recovered

Recoverable workflows are listed first, so the report reads as "here is what you have" before "here is what you do not".

Qualifiers you may see

Message	Meaning	What to do
`recovered but not run by sg-dr`	Non-Terraform workflow.	Nothing — expected. Sources and inputs still come back; you run the tool.
`needs values only StackGuardian holds`	The workflow uses `${secret::…}` or `${ext-secret::…}`.	Know in advance which secrets you would need to supply. See §8.
`private repository, so a clone would need your git access`	Only relevant in the rare case where the run has no snapshot.	Confirm the responder can clone it.
`ran on a shared runner`	Not recoverable.	Move it to a private runner if it matters.
`workflow has no runs` / `no completed run left infrastructure standing`	Nothing worth recovering.	Investigate, or accept for a workflow that was intentionally destroyed.

Exit code

sg-dr verify exits non-zero when any workflow cannot be recovered. That is deliberate: wire it into a scheduled job and it becomes the alarm that tells you something stopped being recoverable while nobody was looking.

# e.g. a nightly CI job
sg-dr verify --org acme --bucket acme-runner-storage --region eu-central-1 \
  || notify-oncall "sg-dr readiness regression"

Recovering a workflow

The basic command

sg-dr recover \
  --org acme \
  --bucket acme-runner-storage \
  --region eu-central-1 \
  --workflow payments-vpc

This writes a runnable working directory under ./recovery/ and stops. Nothing is applied. No state is changed. No command is run.

What it does, step by step

Find the workflow. In current storage a workflow lives at orgs/{org}/wfs/{ksuid}/, and nothing in that path contains its name — the name-to-ID mapping exists only inside run metadata. So sg-dr reads one run payload per workflow to build a name index. If you already know the ID, --workflow-ksuid skips this entirely and is noticeably faster in a large organization.
Choose a run — by default the most recent run that left infrastructure standing. Runs are ordered by their recorded timestamp, not by identifier.
- Failed runs are skipped. They are usually a bug or a bad configuration change, which is exactly what you want to recover away from.
- Completed destroy runs are skipped too. Their state is valid but empty, so recovering one produces a working directory whose plan proposes creating your entire infrastructure from scratch — with nothing to signal that anything is wrong.
Both remain reachable by --run-id when you want them.
Materialize the sources from that run's snapshot — a tarball of the workspace exactly as it ran. This is the preferred source for three reasons: it needs no git access and no reachable git host; it is the only source carrying files the workflow created while running, including file mounts and anything your hooks generated; and it is byte-for-byte what produced your current state.

If the run has no snapshot (which happens for a run paused for approval), sg-dr falls back to a fresh clone using your git credentials, and warns you that runtime-generated files will be missing.
Reconstruct what the snapshot does not carry — the input variables and environment variables (StackGuardian passed those in from outside the working directory), and a backend pointing at your state.
Resolve inter-workflow references — ${workflow::…} values are resolved from the referenced workflow's stored outputs. Anything that cannot be obtained is collected rather than guessed.

Reading the report

→ Recovering payments-vpc

  workflow  acme/platform/payments-vpc
  run       3HEUyanoMhUUSk11T2RAfNjNPuY  completed (apply)  2 days ago
  source    snapshot of this run
  recovered 47 files
            12 inputs to terraform.tfvars.json
            8 environment variables to terraform.env, 2 sensitive
            state at orgs/acme/wfs/2NxKz7abc123/artifacts/tfstate.json

note:     the source repository is private; recovery used the snapshot, so no git access was needed

✓ recovery/platform/payments-vpc/3HEUyanoMhUUSk11T2RAfNjNPuY/user/infra/vpc
            this points at live state; an apply here writes to it
            add --init --plan to compare it against live infrastructure

run tells you which run this is, its outcome, and how long ago — the question you actually have when several runs exist.
source is either snapshot of this run or fresh clone of …. If it says fresh clone, read the warning that accompanies it.
note: is information. Warning: is something that may be wrong. The distinction is maintained strictly, which is what keeps a warning worth reading.
All diagnostics go to stderr. stdout carries the working directory path alone, so the command composes:
```
cd "$(sg-dr recover --org acme --workflow payments-vpc)"
```

Output layout

recovery/{group}[/{stack}]/{workflow}/{run-id}/     0700
├── user/
│   └── {repo}/…                                    dirs 0700, files 0600
│       └── {workingDir}/                           ← your working directory
│           ├── (the workflow's own sources)
│           ├── terraform.tfvars.json  or  sg.iac-inputs.json
│           ├── terraform.env          or  sg.env
│           └── backend.tf
├── filemount/
└── sg-secrets.yaml                                 (only if something was unresolvable)

The directory is built from human coordinates, not internal IDs, so you can find your own output without consulting an index. Everything is 0600/0700 because a recovered workflow holds resolved references, which for real workflows means kubeconfigs and private keys.

The generated files

File	Written for	Purpose
`terraform.tfvars.json`	Terraform / OpenTofu	Input variables. Terraform loads it automatically — no flag needed.
`sg.iac-inputs.json`	Everything else	The same input values, under a neutral name. Calling it `terraform.tfvars.json` on a Helm chart would invite you to reach for the wrong tool; the report tells you what the workflow's own step does with these values (a Helm values file, Ansible extra-vars, …).
`terraform.env` / `sg.env`	All	The run's environment variables, shell-sourceable. Values resolved from sensitive workflow outputs are marked with a comment.
`backend.tf`	Terraform / OpenTofu with StackGuardian-managed state	Points at your existing state object in your own storage.
`sg-secrets.yaml`	Only when needed	Placeholders for values only you can supply.

sg-dr writes the environment file for you to source; it does not source it itself:

set -a; . ./terraform.env; set +a

(--init and --plan export those variables into the tool's process directly, so you do not need to source anything for those.)

About the generated backend

StackGuardian ran your workflow with a local backend inside the runner container and uploaded the resulting state file afterwards. That container path does not exist on your machine, so sg-dr re-points the same state object as a proper remote backend:

terraform {
  backend "s3" {
    bucket = "acme-runner-storage"
    key    = "orgs/acme/wfs/2NxKz7abc123/artifacts/tfstate.json"
    region = "eu-central-1"
  }
}

Three behaviours worth knowing:

It is only generated when StackGuardian managed the state. If your workflow brought its own backend (via hooks), that backend is already in the snapshot and sg-dr writes none — it says so in a note.
It refuses rather than overrides. If the sources already declare a backend, sg-dr errors out naming the file. Terraform accepts only one, and silently overwriting yours would redirect your state.
It is skipped if no state object exists. A backend pointing at a non-existent object is worse than none: init succeeds against an empty state and the plan proposes recreating everything, with no hint that anything is off. sg-dr warns instead.

note

The generated backend declares bucket, key and region only. There is no lock table or lock file. Concurrent applies are not protected against. During a recovery, make sure exactly one person is applying. This is your live state. An apply in this directory writes to the same object StackGuardian reads. That is what makes recovery seamless, and it is also why sg-dr never applies anything for you.

Choosing a specific run

sg-dr recover --org acme --workflow payments-vpc \
  --run-id 3HEO... --accept-state-mismatch

--run-id recovers a named run, including one that failed — a legitimate way to investigate an incident.

--accept-state-mismatch is required whenever that run is not the most recent one. The reason: artifacts are stored per workflow, not per run, and every run overwrites them. So an older run's configuration would be paired with the workflow's current state and outputs, which belong to a newer run. That is sometimes exactly what you want and sometimes badly wrong, so sg-dr makes you say which.

Useful flags

# You already know the workflow's ID — skips building the name index
sg-dr recover --org acme --workflow-ksuid 3H9Jkn4X9bug38zT5vkuFj8bsAF

# Two workflows share a name
sg-dr recover --org acme --workflow api --group platform

# Write somewhere other than ./recovery
sg-dr recover --org acme --workflow api --output /mnt/incident-2026-08-18

# Keep a full record of the run, including detail the terminal filtered out
sg-dr recover --org acme --workflow api --log recovery.log

--log truncates the file it is given. Point it at a new path.

Secrets you must supply

StackGuardian resolves ${secret::…} and ${ext-secret::azure-kv::…} at run time from a store that is unreachable when it is down. sg-dr cannot obtain those values, and will not emit the literal placeholder text into a variable where it would fail somewhere far less obvious.

Instead it collects every one of them in a single pass and writes sg-secrets.yaml next to the recovered workflow:

# Values sg-dr could not resolve.
#
# StackGuardian resolves these at run time, from its own secret store or an
# external vault. They are not written to object storage, so they cannot be
# recovered — only you can supply them.
#
# Fill in every value below, then re-run with:
#   sg-dr recover ... --secrets recovery/.../sg-secrets.yaml
#
# Once filled this file holds live secrets.
# Keep it out of version control, and delete it when the recovery is done.

secrets:

  # ${secret::db_admin_password}
  # source: StackGuardian secret store
  # used by: environment variable TF_VAR_db_password
  db_admin_password: ""

  # ${ext-secret::azure-kv::prod-vault.api-token}
  # source: Azure Key Vault
  # used by: environment variable API_TOKEN
  # used by: step deploy setting config.token
  azure-kv/prod-vault/api-token: ""

Each entry carries the original reference, its source, and every place it is used — so filling the file does not require going back to StackGuardian to work out what each one was for, which is rather the point.

The recovery exits non-zero while values are outstanding: recovery is genuinely incomplete until they are supplied.

Fill in every value, then re-run the same command with --secrets:

sg-dr recover --org acme --workflow payments-api \
  --secrets recovery/apps/payments-api/3HE.../sg-secrets.yaml

Notes:

A secret referenced from three places is filled in once — entries are deduplicated by value, not by use site.
A blank value is rejected. An empty string substituted into a password or a token fails somewhere much harder to diagnose.
sg-dr will never overwrite the file you passed to --secrets. If more values turn out to be needed, it tells you to add them to the file you already filled in.
Keep the filled file with your runbook if your compliance posture allows it, in a secret manager — it turns a two-pass recovery into a one-pass one. Otherwise delete it when the incident is over; sg-dr cleanup --recovered removes it along with everything else.

Inter-workflow references are different

${workflow::group.workflow.outputs.name.value} references — one workflow reading another's outputs — are recovered. Outputs are stored in your bucket, including sensitive ones, so sg-dr resolves them from there and marks the resulting environment variables as sensitive in the generated env file.

Two cases produce a warning rather than a value:

The referenced workflow publishes no outputs. Outputs are written only by an apply or a destroy, so a workflow whose last run was a plan has none.
The output exists but is empty, which usually means that workflow never applied successfully.

Recovering a whole stack

sg-dr recover \
  --org acme \
  --bucket acme-runner-storage \
  --region eu-central-1 \
  --stack k8s-stack

Naming a stack instead of a workflow recovers every workflow in it, in the order its dependencies require.

This ordering is the main thing a stack recovery gives you beyond running recover several times. The order is read from the stack's own dependency graph, as recorded in a stack-triggered run, and topologically sorted (ties broken by name, so the result is deterministic).

Getting it wrong is silently wrong rather than loudly wrong — a Helm release applied before its cluster exists simply fails — and alphabetical order is not merely arbitrary but frequently inverted, because dependents often sort before their dependencies.

→ Recovering stack k8s-stack

  stack     k8s-stack
  order     cluster  →  addons  →  workloads

→ cluster
  ...
→ addons
  ...
→ workloads
  ...

✓ Recovered 3 workflows in k8s-stack
            work through them in the order above

When the order is a guess

If no run of the stack recorded a dependency graph — which happens when the workflows were always triggered individually rather than as a stack — sg-dr says so explicitly:

Warning: this order is a guess, not the stack's own: no run of this stack
         recorded its dependency order, which happens when the workflows were
         triggered individually rather than as a stack
         check the dependencies yourself before applying anything

Partial failures

A stack recovery does not stop at the first failure. During an incident, knowing that three of four workflows recovered is more useful than knowing the second one failed. Every workflow is attempted, every report is printed, and every sg-secrets.yaml is written, so you leave with everything in hand rather than discovering problems one re-run at a time. The command then exits non-zero with a summary of what failed.

--init/--plan on a stack are skipped unless the whole stack recovered, and stop at the first failure when they do run. Later workflows read earlier workflows' outputs and state, so a plan against a partial stack looks authoritative and is not.

Running the recovered workflow

Recovery stops after materializing, by default. To go further, ask explicitly.

sg-dr recover --org acme --workflow payments-vpc --init --plan

--init runs init -input=false, which contacts your live state storage.
--plan runs plan -input=false, and requires --init — a plan refreshes state, so the backend has to be configured first.
The recovered environment is exported into the tool's process. This matters: without it, required TF_VAR_* variables are missing and variables with defaults silently take them, producing spurious diffs.
sg-dr never runs apply. That is your decision, made with the plan in front of you.

A clean run looks like this:

  using     tofu 1.11.13, already downloaded

Initializing the backend...
...
No changes. Your infrastructure matches the configuration.

"No changes" is the success criterion of a recovery drill. It means the sources, the inputs, the environment and the state all line up with what StackGuardian last did.

Binary version resolution

Running a recovered workflow with the wrong tool version is not cosmetic: a newer binary upgrades the state file format on write, irreversibly. So sg-dr goes to some length to obtain the exact version the run used.

--tf-binary wins over everything, so if you manage versions with tfenv, mise or asdf you keep control.
Otherwise the version comes from the workflow's configuration. Real workflows commonly pin a wildcard (OPENTOFU-1.11.x), in which case the exact patch level is read from the run's own log, which records what the step actually resolved. A wildcard with no log to resolve it is an error, not a guess.
The binary is then taken from the shared cache, or from your PATH if the version there already matches exactly, or downloaded.
Downloads are verified against the publisher's SHA256SUMS.

The report tells you which of those happened — already downloaded, downloaded just now, found on your PATH, --tf-binary — because those mean different things if the version turns out to be wrong.

note

Terraform is only downloaded up to 1.5.7, the last release under the MPL. Everything after it is BUSL-licensed and cannot be redistributed. For a newer Terraform, install it yourself and pass --tf-binary /path/to/terraform. OpenTofu has no such restriction.

Binaries are cached in ~/.cache/sg/bin/ — shared with other StackGuardian tools, so a version fetched once is not fetched again. Relocate it with SG_CACHE_HOME, or its parent with XDG_CACHE_HOME.

Applying

When you have read the plan and decided to proceed, run the tool yourself from the recovered working directory:

cd recovery/platform/payments-vpc/3HEU.../user/infra/vpc
set -a; . ./terraform.env; set +a
tofu apply

Remember: this writes to your live state object, there is no lock, and StackGuardian will read the same object when it comes back. Coordinate.

Non-Terraform workflows

--init/--plan are refused for Helm, Ansible, kubectl, CloudFormation and custom workflows, with an explanation. The sources, resolved inputs (sg.iac-inputs.json) and environment (sg.env) are all there; run the tool yourself:

cd recovery/platform/k8s-stack/addons/3HEV.../user/helm-deployments/wfs-demo
set -a; . ./sg.env; set +a
helm upgrade --install wfs-demo . -f sg.iac-inputs.json

sg-dr also notes any non-Terraform steps a Terraform workflow had — a pre-apply script, a notification step — because it does not re-run those either.

After the incident: `sg-dr cleanup`

A recovered workflow contains the inputs and environment a workflow ran with, including any secrets that were resolved for it. Removing that once an incident is over is worth doing deliberately.

# See what is on this machine — removes nothing
sg-dr cleanup

# Remove recovered workflows, and the secrets in them
sg-dr cleanup --recovered

# Remove cached OpenTofu/Terraform binaries (shared with other SG tools)
sg-dr cleanup --binaries

# Remove everything sg-dr created
sg-dr cleanup --all

Run without flags, it only reports. Nothing is removed unless you say so, and it refuses to touch a filesystem root, your home directory, or the parent of home directories. Files written by --log are never touched — delete those yourself if they matter.

Also delete any filled sg-secrets.yaml you kept outside the recovery directory.

Returning to StackGuardian

Because the recovered workflow writes to the same state object StackGuardian reads, there is no migration back. When the service is available again:

Stop applying locally. Make sure no one is mid-apply.
Reconcile. If you applied changes during the outage, the state object already reflects them. StackGuardian's next plan will see the current reality — no import, no terraform state push.
Push source changes. If you edited the recovered sources, commit those changes to the repository and branch the workflow tracks. The snapshot is a copy of what ran, not a checkout you can push from, so make the edits in a real clone.
Re-inject secrets you rotated. If you rotated any credential during the incident, update it in StackGuardian's secret store or your vault.
Run a plan in StackGuardian before the next apply, and confirm it matches what you expect.
Clean up. sg-dr cleanup --all.

Command reference

Global flags

Available on every command.

Flag	Short	Description
`--provider`	`-P`	Storage provider: `aws` or `azure`. Default `aws`.
`--org`	`-O`	StackGuardian organization name. Required.
`--bucket`	`-b`	S3 bucket your runner writes to (AWS).
`--region`		Region of that bucket; taken from the workflow run when omitted (AWS).
`--profile`		Named profile from your AWS config, as `aws --profile` (AWS).
`--account`	`-a`	Azure storage account.
`--container`	`-c`	Azure blob container. Default `runner`.
`--resource-group`	`-R`	Azure resource group (optional; carried into the generated backend).
`--prefix`		Path inside the bucket/container the runner writes under, if your deployment set one.
`--log`	`-l`	Also write a timestamped record of the run to a file. Truncates.
`--verbose`	`-v`	Show extra detail.
`--quiet`	`-q`	Show only errors and warnings.
`--version`		Print version, commit and build date.

`sg-dr recover`

Reconstruct a workflow's environment on local disk.

Flag	Short	Description
`--workflow`	`-w`	Workflow name.
`--workflow-ksuid`		Workflow ID — skips the name lookup, which is faster.
`--stack`	`-S`	Stack to recover whole, or which stack a workflow belongs to.
`--group`	`-g`	Workflow group, when two workflows share a name.
`--run-id`	`-r`	Recover a specific run instead of the last healthy one.
`--accept-state-mismatch`		Recover an older run even though state and outputs belong to a newer one.
`--output`	`-o`	Where to write. Default `recovery`.
`--secrets`		File of values only you can supply, filled in from an earlier run.
`--init`	`-i`	Also run `init`, which contacts your state storage.
`--plan`	`-p`	Also run `plan`. Requires `--init`.
`--tf-binary`		Use this OpenTofu/Terraform binary instead of the version the run used.
`--bin-cache`		Where to keep downloaded binaries. Default `~/.cache/sg/bin`.

One of --workflow, --workflow-ksuid or --stack is required.

`sg-dr verify`

Report what could be recovered, changing nothing. Takes global flags only.

`sg-dr cleanup`

Remove what recovering left on this machine.

Flag	Description
`--recovered`	Remove recovered workflows, including any secrets resolved into them.
`--binaries`	Remove downloaded OpenTofu and Terraform versions.
`--all`	Remove everything `sg-dr` created.
`--output`, `-o`	Directory recovered workflows were written to. Default `recovery`.
`--bin-cache`	Directory downloaded binaries were kept in.

`sg-dr help files`

Print the full inventory of files, directories, subprocesses and network destinations the tool touches.

Exit codes

Code	Meaning
`0`	Success.
non-zero	`verify`: at least one workflow cannot be recovered. `recover`: recovery failed, or completed but still needs values only you can supply.

Other environment variables

Variable	Effect
`SG_CACHE_HOME`	Relocate the whole StackGuardian cache.
`XDG_CACHE_HOME`	Relocate its parent (honoured on every platform).
`NO_COLOR`	Disable coloured output.

Security model

Worth reviewing before an incident, and worth showing to whoever must approve the tool.

It never writes to your storage

The storage backend is opened read-only. No code path uploads, modifies or deletes an object in your bucket or container.

It never uses credentials found in your storage

A run's stored payload contains live credentials: AWS credential and config file contents, VCS tokens, and the workflow's secret map. None of them are read. They are not modelled in the code at all, and the project's test suite fails if any survives decoding. Repository URLs have embedded tokens stripped at the JSON-decoding boundary, so no part of the program can reach one.

Two reasons:

A stored credential resurrected during an incident may long outlive its intended lifetime.
A tool that reads credentials out of a bucket turns read access to that bucket into access to everything it describes.

The consequence is that you must already have the access you are recovering with — the cloud accounts, the repositories, the clusters. sg-dr verify flags private repositories up front so you learn that before a clone fails rather than after.

Everything it writes is owner-only

Recovered trees are 0700, files 0600. A recovered workflow holds resolved references, which for real workflows means kubeconfigs and private keys.

It refuses unsafe archives

An archive entry that would escape the destination directory is refused, not sanitized. Rewriting ../evil.tf to evil.tf could overwrite a real file. Per-file and total size limits guard against a malformed archive filling a disk during an incident.

Network destinations

Destination	When
Your S3 bucket or Azure container	Always. Reads only.
`github.com`	Fetching an OpenTofu version not already cached.
`releases.hashicorp.com`	Fetching a Terraform version (1.5.7 or older).
Your git host	Only on the clone fallback.
The provider registry	During `--init`, by OpenTofu/Terraform itself.
The StackGuardian API	Never.

Subprocesses

git, only on the clone fallback, run with prompting disabled so an unreachable private repository fails instead of hanging. tofu or terraform, for --init and --plan and a version probe.

Troubleshooting

Message	Cause	Fix
`no organization: pass --org or set ORG_NAME`	Missing organization.	Set `--org` / `ORG_NAME`.
`no S3 bucket given: pass --bucket`	Missing bucket.	Set `--bucket` / `BUCKET_NAME`.
`no AWS region: pass --region, set AWS_REGION, or configure a profile`	No region resolvable from flags, environment or profile.	Set `--region`.
`no workflows found in {org}: check the organization and bucket`	Wrong org name, wrong bucket, or a `--prefix` your deployment uses that you did not pass.	Verify all three.
`"api" is ambiguous, it matches …: narrow it with --group or --stack`	Names are unique within a group, not across an organization.	Add `--group` or `--stack`.
`workflow ran on a shared runner, so its artifacts live in StackGuardian's own storage`	Not recoverable.	Move the workflow to a private runner.
`no completed run left infrastructure standing`	Every run failed, or the newest completed run was a destroy.	Use `--run-id` to materialize a specific run anyway.
`run … is not the most recent (… is)`	Older run; its config would pair with newer state.	Add `--accept-state-mismatch` if that is what you want.
`run … has no snapshot, and cloning the repository failed. It is private…`	Recovery fell back to a clone and your git access is insufficient.	Confirm you can clone the repository yourself, with your own credentials.
`the sources already declare a backend in [main.tf]`	Terraform accepts only one backend.	Remove the existing one, or recover the workflow without backend generation.
`no state was found in storage for this workflow` (warning)	The workflow may never have applied, or its state lives under a different path.	Check `--prefix`; check whether the workflow ever applied.
`this run records its storage path as "x" but --prefix is "y"` (warning)	Your `--prefix` disagrees with what the run recorded. The run's own path wins.	Check which is right if `init` cannot find the state.
`the workflow pins OPENTOFU-1.11.x … and the run log does not record which one it used`	Wildcard version, no log to resolve it.	Pass `--tf-binary`.
`Terraform 1.6.0 is licensed under the BUSL and cannot be downloaded by this tool`	Licence restriction.	Install it yourself and pass `--tf-binary`.
`source workflow … publishes no outputs`	Outputs are written only by an apply or destroy; a workflow whose last run was a plan has none.	Recover/apply that workflow first, or supply the value manually.
`recovery is incomplete: N values still needed`	Unresolvable secrets.	Fill `sg-secrets.yaml`, re-run with `--secrets`.
`secrets file still has empty values for: …`	A blank entry.	Fill every value; blanks are rejected on purpose.

Getting more detail. --verbose shows what the tool is doing; --log recovery.log records everything including detail the terminal filtered out — useful when handing an incident to someone else.

Appendix A: Storage layout reference

Useful for auditing your bucket, for building your own tooling, and for the manual fallback in Appendix B.

Current layout (ID-addressed)

orgs/{org}/wfs/{workflow-id}/
├── artifacts/
│   ├── tfstate.json               ← Terraform state (managed state only)
│   ├── sg.outputs.json            ← outputs, written by apply and destroy only
│   └── sg.outputs_masked.json     ← same, with sensitive values masked
├── cache/
│   └── cache.tar.gz               ← workspace cache (per workflow)
└── wfruns/{run-id}/
    ├── metrics/details.json       ← the complete run payload
    ├── snapshot/snapshot.tar.gz   ← the workspace exactly as it ran
    └── logs/wfrun.log             ← console log, records the resolved tool version

Legacy layout (name-addressed)

Older organizations may still have workflows addressed by name:

orgs/{org}/wfgrps/{group}/wfs/{workflow}/                     standalone
orgs/{org}/wfgrps/{group}/stacks/{stack}/wfs/{workflow}/      stack member

with the same artifacts/, cache/ and wfruns/ beneath. Workflow groups can span several path segments (platform/kubernetes), so parse by the wfgrps, stacks and wfs markers rather than by position.

Two lifetimes, and why it matters

Per run: wfruns/{run-id}/… — the snapshot, the payload, the log. These accumulate.
Per workflow: artifacts/… — state and outputs. Every run overwrites them.

That asymmetry is the reason --accept-state-mismatch exists: recovering an older run pairs that run's configuration with the workflow's current state.

Key fields in `details.json`

Path	Meaning
`LatestStatus`	`COMPLETED`, `ERRORED`, …
`CreatedAt`	Unix milliseconds; the authoritative ordering key for runs
`ParentId`	`/orgs/{org}/wfgrps/{group}[/stacks/{stack}]/wfs/{name}` — the workflow's human coordinates
`RuntimeParameters.wfType`	`TERRAFORM`, `OPENTOFU` or `CUSTOM`
`RuntimeParameters.runnerConstraints.type`	`private` or `shared`
`RuntimeParameters.storageBackendConfig`	Bucket/region or storage account/container, and any path prefix. Absent for shared-runner runs.
`RuntimeParameters.terraformConfig.terraformVersion`	e.g. `OPENTOFU-1.11.x`, `TERRAFORM-1.5.7`, `CUSTOM-/opt/tofu`
`RuntimeParameters.terraformConfig.managedTerraformState`	Whether StackGuardian owned the state
`RuntimeParameters.terraformAction.action`	`apply` or `destroy`
`RuntimeParameters.vcsConfig.iacInputData.data`	Input variables, before reference interpolation
`RuntimeParameters.userProvidedEnvVars`	Base64 of `{"userProvidedEnvVars":[{name,value}]}`, before interpolation
`RuntimeParameters.wfStepsConfig[]`	Each step's container template ID and settings
`SGInternals.resolvedVCSconfig`	The repository, ref and working directory actually checked out
`TriggerDetails.Action.order`	The stack dependency graph, on stack-triggered runs only

Values in iacInputData.data and userProvidedEnvVars are stored before interpolation — the raw ${workflow::…} and ${secret::…} templates. That is deliberate and is exactly what a recovery needs, since it must resolve references against what it can actually reach.

note

details.json also contains live credentials — AWS credential file contents, VCS tokens, the workflow's secret map. If you write your own tooling against these payloads, treat the whole file as a secret, and prefer not to read those fields at all.

Appendix B: Manual fallback, without `sg-dr`

sg-dr exists because doing this by hand is error-prone: a wrong state key points a plan at the wrong infrastructure, and the mistake is not obvious. Use this only if the tool is genuinely unavailable to you, and check your work.

BUCKET=acme-runner-storage
ORG=acme
WF_ID=2NxKz7abc123           # from the storage listing
PREFIX="orgs/$ORG/wfs/$WF_ID"

# 1. Which runs exist
aws s3 ls "s3://$BUCKET/$PREFIX/wfruns/"

# 2. Pick the newest COMPLETED, non-destroy run
for run in $(aws s3 ls "s3://$BUCKET/$PREFIX/wfruns/" | awk '{print $2}' | tr -d '/'); do
  d=$(aws s3 cp "s3://$BUCKET/$PREFIX/wfruns/$run/metrics/details.json" - 2>/dev/null)
  [ -z "$d" ] && continue
  echo "$(jq -r '.CreatedAt' <<<"$d") $run $(jq -r '.LatestStatus' <<<"$d") $(jq -r '.RuntimeParameters.terraformAction.action // "-"' <<<"$d")"
done | sort -rn

RUN_ID=<the run you chose>

# 3. Fetch its payload and its workspace snapshot
aws s3 cp "s3://$BUCKET/$PREFIX/wfruns/$RUN_ID/metrics/details.json" ./details.json
aws s3 cp "s3://$BUCKET/$PREFIX/wfruns/$RUN_ID/snapshot/snapshot.tar.gz" ./snapshot.tar.gz
mkdir -p work && tar xzf snapshot.tar.gz -C work

# 4. Enter the working directory
REPO=$(jq -r '.SGInternals.resolvedVCSconfig.repo_name' details.json)
WD=$(jq -r '.SGInternals.resolvedVCSconfig.workingDir // "."' details.json)
cd "work/user/$REPO/$WD"

# 5. Remove StackGuardian's backend override — it points inside the runner container
rm -f *_stackguardian_override.tf

# 6. Reconstruct the inputs
jq '.RuntimeParameters.vcsConfig.iacInputData.data' ../../../../details.json \
  > terraform.tfvars.json

# 7. Reconstruct the environment
jq -r '.RuntimeParameters.userProvidedEnvVars' ../../../../details.json \
  | base64 -d \
  | jq -r '.userProvidedEnvVars[] | "export \(.name)=\((.value)|@sh)"' \
  > terraform.env

# 8. Point at your existing state
REGION=$(jq -r '.RuntimeParameters.storageBackendConfig.awsRegion' ../../../../details.json)
SB=$(jq -r '.RuntimeParameters.storageBackendConfig.s3BucketName' ../../../../details.json)
cat > backend.tf <<EOF
terraform {
  backend "s3" {
    bucket = "$SB"
    key    = "$PREFIX/artifacts/tfstate.json"
    region = "$REGION"
  }
}
EOF

# 9. Use the version the run used — check the log
aws s3 cp "s3://$BUCKET/$PREFIX/wfruns/$RUN_ID/logs/wfrun.log" - | grep -m1 'Resolved '

set -a; . ./terraform.env; set +a
tofu init && tofu plan

Things this shortcut does not do, which sg-dr does:

Skip completed destroy runs (recovering one yields a plan that recreates everything).
Resolve ${workflow::…} references from the other workflow's stored outputs.
Detect and collect ${secret::…} values instead of writing the literal placeholder into an environment variable.
Refuse to generate a backend when the sources already declare one, or when no state object exists.
Fetch and verify the exact tool version the run used.
Handle managedTerraformState: false, the legacy storage layout, a storage prefix, or Azure.

Appendix C: Recovery drill

Run this quarterly, and after any change to your runner configuration or storage layout. It costs an afternoon and it is the only thing that turns this document into a capability.

Readiness. Run sg-dr verify for each organization. Every workflow that matters should be green. Investigate every one that is not.
Cold start. Have someone who did not write the runbook do the rest — from the archived binary, following this document only.
Verify the binary. sha256sum -c, cosign verify-blob, sg-dr --version.
Recover a real workflow. Pick a production one, not a toy. sg-dr recover --workflow … --init --plan.
Confirm "No changes". That is the pass criterion. Any diff is a finding: drift, a missing environment variable, or a version mismatch.
Recover a stack, and confirm the reported order matches your architecture.
Exercise the secrets path. Recover a workflow that needs secrets, fill in sg-secrets.yaml, and re-run with --secrets. Time it — this is the step that surprises people.
Time the whole thing. Record it. That number is your real RTO, and it is the one to quote to auditors.
Clean up. sg-dr cleanup --all, and delete any filled secrets file.
Write down what went wrong, and fix it before the next drill.

Getting help

Raise anything through your usual StackGuardian support channel. Include the output of sg-dr --version, and — if you can share it — a --log file from the run in question.

Overview​

What "recovery" means in this guide​

The three guarantees​

Scope: what can and cannot be recovered​

Private runners only​

By workflow type​

What cannot be recovered at all​

Before an incident: preparation​

Checklist​

Storage permissions​

Protect the storage itself​

Store the binary where you will still have it​

Installing sg-dr​

Homebrew (macOS and Linux)​

Scoop (Windows)​

Archives (every platform)​

Verify it before you store it​

Review what it touches​

Connecting to your storage​

AWS S3​

Azure Blob Storage​

Environment variables​

Readiness: sg-dr verify​

Sample output​

Qualifiers you may see​

Exit code​

Recovering a workflow​

The basic command​

What it does, step by step​

Reading the report​

Output layout​

The generated files​

About the generated backend​

Choosing a specific run​

Useful flags​

Secrets you must supply​

Inter-workflow references are different​

Recovering a whole stack​

When the order is a guess​

Partial failures​

Running the recovered workflow​

Binary version resolution​

Applying​

Non-Terraform workflows​

After the incident: sg-dr cleanup​

Returning to StackGuardian​

Command reference​

Global flags​

sg-dr recover​

sg-dr verify​

sg-dr cleanup​

sg-dr help files​

Exit codes​

Other environment variables​

Security model​

It never writes to your storage​

It never uses credentials found in your storage​

Everything it writes is owner-only​

It refuses unsafe archives​

Network destinations​

Subprocesses​

Troubleshooting​

Appendix A: Storage layout reference​

Current layout (ID-addressed)​

Legacy layout (name-addressed)​

Two lifetimes, and why it matters​

Key fields in details.json​

Appendix B: Manual fallback, without sg-dr​

Appendix C: Recovery drill​

Getting help​

Overview

What "recovery" means in this guide

The three guarantees

Scope: what can and cannot be recovered

Private runners only

By workflow type

What cannot be recovered at all

Before an incident: preparation

Checklist

Storage permissions

Protect the storage itself

Store the binary where you will still have it

Installing `sg-dr`

Homebrew (macOS and Linux)

Scoop (Windows)

Archives (every platform)

Verify it before you store it

Review what it touches

Connecting to your storage

AWS S3

Azure Blob Storage

Environment variables

Readiness: `sg-dr verify`

Sample output

Qualifiers you may see

Exit code

Recovering a workflow

The basic command

What it does, step by step

Reading the report

Output layout

The generated files

About the generated backend

Choosing a specific run

Useful flags

Secrets you must supply

Inter-workflow references are different

Recovering a whole stack

When the order is a guess

Partial failures

Running the recovered workflow

Binary version resolution

Applying

Non-Terraform workflows

After the incident: `sg-dr cleanup`

Returning to StackGuardian

Command reference

Global flags

`sg-dr recover`

`sg-dr verify`

`sg-dr cleanup`

`sg-dr help files`

Exit codes

Other environment variables

Security model

It never writes to your storage

It never uses credentials found in your storage

Everything it writes is owner-only

It refuses unsafe archives

Network destinations

Subprocesses

Troubleshooting

Appendix A: Storage layout reference

Current layout (ID-addressed)

Legacy layout (name-addressed)

Two lifetimes, and why it matters

Key fields in `details.json`

Appendix B: Manual fallback, without `sg-dr`

Appendix C: Recovery drill

Getting help