Disaster Recovery¶

How to recover SwarmCracker from node failures, Raft corruption, and cluster outages.

Overview¶

SwarmCracker cluster state is stored in multiple locations:

Scenario 1: Worker Node Failure¶

Detection¶

swarmcracker cluster health
# Will show healthz endpoint unreachable for the failed node
swarmctl --socket /var/lib/swarmkit/swarm.sock node ls
# Status will show "down" for the failed worker

Recovery Steps¶

1. Verify the node is truly dead:

   ssh dead-worker -- swarmcracker cluster health
   

2. Remove the dead node from the cluster:

   swarmctl --socket /var/lib/swarmkit/swarm.sock node rm <NODE_ID>
   

3. VMs on the dead node are lost. SwarmKit will reschedule services with desired replicas. Verify:

   swarmcracker service ps <service-name>
   

4. Replace the worker:

   # Get a new join token on the manager
   swarmcracker cluster token create --role worker
   
   # On the replacement node: install binary + setup deps, then join
   curl -fsSL https://raw.githubusercontent.com/restuhaqza/SwarmCracker/main/install.sh | sudo bash
   sudo swarmcracker setup install --download-kernel --download-rootfs
   sudo swarmcracker setup network
   sudo swarmcracker setup config --non-interactive
   sudo swarmcracker cluster join --token <TOKEN> <MANAGER_IP>:4242
   

Scenario 2: Manager Node Failure¶

⚠️ Manager failure requires immediate action — the Raft consensus log is on the manager.

If you have >1 manager (recommended for production)¶

SwarmKit's Raft consensus handles this automatically. The remaining managers elect a new leader.

# Check which manager is now leader
swarmctl --socket /var/lib/swarmkit/swarm.sock node ls
# Look for "Leader" in the MANAGER STATUS column

If you have only 1 manager (typical for small clusters)¶

You need to promote a worker or set up a new manager.

Option A: Promote a worker to manager¶

1. On a healthy worker, stop the swarmd service:

   sudo systemctl stop swarmd-firecracker
   

2. Clear old worker state (prevents CA conflicts):

   sudo rm -rf /var/lib/swarmkit/certificates
   sudo rm -rf /var/lib/swarmkit/worker
   

3. Start as manager with force-new-cluster:

   sudo swarmd-firecracker \
     --manager \
     --force-new-cluster \
     --state-dir /var/lib/swarmkit \
     --listen-remote-api 0.0.0.0:4242 \
     --advertise-remote-api <THIS_NODE_IP>:4242 \
     --bridge-name swarm-br0 \
     --enable-cni
   

4. Re-join remaining workers:

   # Get new join token
   sudo cat /var/lib/swarmkit/join-tokens.txt
   
   # On each worker
   sudo swarmcracker cluster join --token <TOKEN> <NEW_MANAGER_IP>:4242
   

Option B: Restore from Raft backup (if you have one)¶

# Restore the Raft snapshot on a new node
sudo swarmd-firecracker \
  --manager \
  --state-dir /var/lib/swarmkit \
  --raft-snapshot /backup/raft-snapshot.db \
  --listen-remote-api 0.0.0.0:4242

Scenario 3: Raft Log Corruption¶

Symptoms: Manager won't start, or swarmctl commands return inconsistent results.

Recovery¶

1. Stop the manager:

   sudo systemctl stop swarmd-firecracker
   

2. Try Raft recovery:

   # SwarmKit has a built-in recovery mechanism
   sudo swarmd-firecracker \
     --manager \
     --force-new-cluster \
     --state-dir /var/lib/swarmkit \
     --listen-remote-api 0.0.0.0:4242 \
     --advertise-remote-api <IP>:4242
   

This creates a new Raft log from the most recent consistent state.

1. If recovery fails, reset and rebuild:

   sudo rm -rf /var/lib/swarmkit/raft
   sudo swarmcracker cluster reset --force
   sudo swarmcracker cluster init --advertise-addr <IP>:4242
   # Re-join all workers
   

⚠️ Resetting loses all running VM state. Use this only as a last resort.

Scenario 4: VM Snapshot Recovery¶

If a VM's state is lost but you have snapshots:

# List available snapshots
swarmcracker vm snapshot list --task <VM_ID>

# Restore from snapshot
swarmcracker vm snapshot restore <SNAPSHOT_ID>

# Verify
swarmcracker cluster status <VM_ID>

Automated snapshot backup¶

Set up a cron job to copy snapshots to remote storage:

# /etc/cron.d/swarmcracker-backup
0 2 * * * root rsync -avz /var/lib/firecracker/snapshots/ backup-server:/backups/swarmcracker/

Scenario 5: Full Cluster Outage¶

All nodes lose power or network.

Recovery Steps¶

1. Power on the manager node first.

2. Wait for it to start:

   # Manager should auto-start via systemd
   sudo systemctl status swarmd-firecracker
   swarmcracker cluster health
   

3. Power on worker nodes one at a time:

   # On each worker
   sudo systemctl start swarmd-firecracker
   sudo journalctl -u swarmd-firecracker -f
   # Look for "Node joined" in logs
   

4. Verify cluster:

   swarmcracker cluster health --format json | jq .
   swarmctl --socket /var/lib/swarmkit/swarm.sock node ls
   

5. Verify VMs:

   swarmcracker vm list
   swarmcracker service ps --all
   

6. VMs that were running before the outage will need to be recreated:

   # If using services (recommended), SwarmKit handles this automatically
   swarmcracker service update <service> --force
   

Preventive Measures¶

1. Regular Raft backups¶

# Run daily via cron on the manager node
#!/bin/bash
BACKUP_DIR="/backup/swarmcracker/$(date +%Y-%m-%d)"
mkdir -p "$BACKUP_DIR"

# Backup Raft state
swarmctl --socket /var/lib/swarmkit/swarm.sock cluster snapshot \
  > "$BACKUP_DIR/raft-snapshot-$(date +%H%M).db"

# Backup config
cp /etc/swarmcracker/config.yaml "$BACKUP_DIR/"

# Backup join tokens
cp /var/lib/swarmkit/join-tokens.txt "$BACKUP_DIR/"

2. Multiple managers (production)¶

For production clusters with >3 nodes, run 3 managers. SwarmKit's Raft consensus requires odd numbers.

# Add a second manager
swarmcracker cluster token manager | grep TOKEN
# On new node: use the manager token
swarmcracker cluster join --role manager --token <TOKEN> <LEADER_IP>:4242

3. VM state redundancy¶

Use SwarmKit services instead of raw VMs for stateless workloads. Services automatically reschedule tasks when a worker fails.

# Good: service with replicas
swarmcracker service create --name web --replicas 3 nginx:alpine

# Instead of: single VM
swarmcracker vm create web-vm --image nginx:alpine

4. Health monitoring¶

# Nagios/Icinga compatible exit code
swarmcracker cluster health --format nagios
# CRITICAL: 2 checks failed | kvm=pass firecracker=pass ...
# OK: all checks passed | kvm=pass ...

# JSON for scripting/monitoring
swarmcracker cluster health --format json | jq '.healthy'

Quick Reference¶