# Backup & Restore Runbook — Vigilant Entities

> ISO 27001 A.12.3 / SOC 2 CC9.1 / NIST 800-53 CP-9 compliant.
> Reviewed quarterly. Drill annually (Q3 tabletop).

## 1. Backup architecture

```
                  ┌─────────────────────┐
   Cron @02:00UTC │ scheduler_service   │ ─► db.aws_backup_runs
   (every day)    │ .mongo_backup_task  │
                  └──────────┬──────────┘
                             ▼
                  ┌─────────────────────┐
                  │ mongodump → tarball │
                  └──────────┬──────────┘
                             ▼
                  ┌─────────────────────┐
                  │ AWS S3 (versioned)  │  ← tenant's own AWS account
                  │ Glacier transition  │     via STS-assumed role
                  │ Lifecycle 90d / 1y  │
                  └─────────────────────┘
```

## 2. What gets backed up

| Asset                   | Method                              | Frequency | Retention            |
|-------------------------|-------------------------------------|-----------|----------------------|
| MongoDB (all DBs)       | `mongodump --gzip --archive`        | Daily 02:00 UTC | 30d hot + 1y Glacier |
| Uploaded files (S3)     | Native S3 versioning + replication  | Continuous| Versioned forever    |
| `audit_logs` chain      | Same as Mongo + integrity verify    | Daily     | 7 years (regulatory) |
| Tenant `.env` configs   | Encrypted (KMS) in S3               | On change | Versioned forever    |
| TLS cert + DKIM keys    | Encrypted (KMS) in S3               | On rotation | Forever             |
| Server image (Docker)   | ECR tag per deploy                  | Per deploy| 12 deploys           |

## 3. Backup verification

Every backup run logs an entry in `db.aws_backup_runs`:
```js
{
  run_id, started_at, finished_at, status,
  collections, document_count, bytes,
  s3_key, sha256, etag,
  err?
}
```

A monthly **restore-test job** (Q3 drill at minimum) downloads a recent
backup into a sandbox Mongo, runs `audit_chain.verify_chain()` for every
tenant, and confirms `db.users.count() > 0`. Anomaly → P1 incident.

## 4. RTO / RPO

| Metric | Target | Max acceptable |
|--------|--------|----------------|
| RPO    | 1 hour | 24 hours       |
| RTO    | 2 hours| 4 hours        |

## 5. Restore procedure

### 5.1 Full database restore (P0 — full corruption / ransomware)

```bash
# 1. Spin a clean replacement Mongo instance.
#    (Provision via your cloud provider / IaC pipeline.)

# 2. Identify the most recent verified backup.
aws s3 ls s3://vigilant-backups/mongo/ --recursive \
  | sort -k1,2 -r | head -5

# 3. Download.
aws s3 cp s3://vigilant-backups/mongo/2026-02-12T02-00-00.archive.gz \
  /tmp/restore.archive.gz

# 4. Validate sha256 vs aws_backup_runs.sha256.
mongo mongodb://localhost:27017 --eval '
  db.aws_backup_runs.findOne({s3_key: "mongo/2026-02-12T02-00-00.archive.gz"})
'
sha256sum /tmp/restore.archive.gz

# 5. Restore.
mongorestore --gzip --archive=/tmp/restore.archive.gz \
  --drop \
  --uri="$MONGO_URL"

# 6. Re-bind the live app.
sudo supervisorctl restart backend

# 7. Verify audit-chain integrity end-to-end.
curl -H "Authorization: Bearer $MAKER_JWT" \
  "$BACKEND/api/security/audit/verify"
# expect: {ok: true, total: <approx pre-incident count>}
```

### 5.2 Tenant-scoped restore (P1 — single tenant data loss)

If a single tenant accidentally bulk-deleted data:

```bash
# 1. Extract just the tenant's documents from the backup.
mongorestore --gzip --archive=/tmp/restore.archive.gz \
  --nsInclude "vigilant.*" \
  --nsExclude "vigilant.audit_logs" \
  --queryFile /tmp/tenant-filter.json \
  --uri="mongodb://localhost:27017/restore_scratch"

# /tmp/tenant-filter.json:
# { "tenant_id": "acme-corp" }

# 2. Diff against current state.
node tools/diff_tenant_state.js acme-corp

# 3. Selectively merge using db.bulkWrite operations into prod.
```

### 5.3 Single-document restore (P3)

Use the in-app history view (if the resource type supports it), or
contact platform team. Most user-mutable docs have soft-delete + 30d
retention.

## 6. Encryption at rest

| Layer        | Method                            | Key custody                  |
|--------------|-----------------------------------|------------------------------|
| Mongo        | Disk-level (cloud-provider)       | Cloud provider KMS           |
| S3 backups   | SSE-KMS with customer master key  | Tenant's AWS KMS in their account |
| `.env` files | KMS envelope encryption           | Tenant's AWS KMS             |
| Field-level  | Reserved for future (PCI scope)   | n/a                          |

## 7. Tabletop drill log

| Date       | Scenario                  | RTO actual | RPO actual | Findings              | Owner |
|------------|---------------------------|------------|------------|-----------------------|-------|
| 2026-Q3    | _scheduled_               | _tbd_      | _tbd_      | _tbd_                 | Ops   |
| 2025-09-15 | Mongo corruption          | 1h45       | 22 min     | RTO target met        | Ops   |

## 8. Key contacts

| Role        | Email                                |
|-------------|--------------------------------------|
| Ops Lead    | ops@vigilantentities.com             |
| Security    | security@vigilantentities.com        |
| AWS support | (per support contract)               |

## Last reviewed: 2026-02-12 · Next: 2026-05-12