Backup Solutions

Backup Solutions

Introduction

Regular backups are essential for protecting your data from hardware failures, accidental deletions, or system corruption. MidnightBSD provides several built-in backup tools and supports many third-party solutions through mports. This guide covers various backup methods to help you implement a robust backup strategy tailored to your needs.

Backup solutions range from simple file copying to sophisticated, encrypted, deduplicated backups to remote servers. The right approach depends on your data volume, sensitivity, and recovery requirements.

Backup Strategy Basics

Follow the 3-2-1 rule for critical data:

Consider these factors when choosing backup tools:

Built-in Tools

dump and restore

The dump and restore utilities are traditional Unix tools for backing up entire file systems. They are included in the MidnightBSD base system and work at the file system level.

Key features:

Basic usage:

# Full backup of a file system
# dump -0 -f backupfile.dump /dev/ada0s1a

# Incremental backup (levels 1-9)
# dump -1 -f backupfile.dump /dev/ada0s1a

# Restore from backup
# cd /destination
# restore -r -f backupfile.dump

For more details, see the Dump & Restore documentation.

cpdup

cpdup is a fast file copying utility designed for efficient directory synchronization. It's particularly useful for maintaining mirrors of file systems or creating backup copies with minimal overhead.

Key features:

Installation:

# mport install cpdup

Basic usage:

# Local backup
cpdup /source/directory /backup/directory

# Backup to remote server via SSH
cpdup -v -x -s user@remotehost:/backup/directory /local/directory

# Dry run (show what would be copied)
cpdup -v -n /source /backup

Common options:

Automated backup script with cpdup:

#!/bin/sh
# Backup home directories to external drive
BACKUP_SRC="/home"
BACKUP_DEST="/mnt/backup/home"
LOG_FILE="/var/log/cpdup_backup.log"

# Ensure backup destination is mounted
if ! mount | grep -q "$BACKUP_DEST"; then
    mount /dev/da1s1 /mnt/backup
    if [ $? -ne 0 ]; then
        echo "Failed to mount backup drive" | logger
        exit 1
    fi
fi

# Run the backup
echo "Starting cpdup backup: $(date)" >> "$LOG_FILE"
cpdup -v -x "$BACKUP_SRC" "$BACKUP_DEST" >> "$LOG_FILE" 2>&1

# Unmount backup drive
umount /mnt/backup

echo "Backup completed: $(date)" >> "$LOG_FILE"
rsync

rsync is a versatile file synchronization tool that efficiently transfers and synchronizes files between directories, local drives, or remote servers. It's widely used for backups due to its efficiency and flexibility.

Key features:

Installation:

# mport install rsync

Basic usage:

# Local directory sync
rsync -av /source/directory/ /backup/directory/

# Remote backup via SSH
rsync -av -e ssh /local/source/ user@remote:/backup/destination/

# Dry run (show what would be transferred)
rsync -avn /source/ /backup/

# Compressed transfer
rsync -avz /source/ remote:/backup/

Common options:

Excluding files and directories:

# Exclude specific files
rsync -av --exclude='*.tmp' --exclude='*.bak' /source/ /backup/

# Exclude using a file
rsync -av --exclude-from='/etc/rsync_exclude.txt' /source/ /backup/

Incremental backup with rsync:

#!/bin/sh
# Incremental backup with date-stamped directories
BACKUP_SRC="/home"
BACKUP_DEST="/backup/home"
DATE=$(date +%Y%m%d_%H%M%S)

# Create date-stamped backup directory
mkdir -p "${BACKUP_DEST}/${DATE}"

# Run rsync with compression and progress
rsync -avzP --delete "${BACKUP_SRC}/" "${BACKUP_DEST}/${DATE}/"

# Create latest symlink for easy access
rm -f "${BACKUP_DEST}/latest"
ln -s "${BACKUP_DEST}/${DATE}" "${BACKUP_DEST}/latest"

Backup to external drive with rsync:

#!/bin/sh
EXTERNAL_MOUNT="/mnt/external"
BACKUP_SRC="/home"
LOG_FILE="/var/log/rsync_backup.log"

# Mount external drive
mount /dev/da0s1 "$EXTERNAL_MOUNT"
if [ $? -ne 0 ]; then
    echo "Failed to mount external drive" | logger
    exit 1
fi

# Run backup
echo "Starting rsync backup: $(date)" >> "$LOG_FILE"
rsync -av --delete --stats "$BACKUP_SRC/" "$EXTERNAL_MOUNT/backup/" >> "$LOG_FILE" 2>&1

# Unmount external drive
umount "$EXTERNAL_MOUNT"

echo "Backup completed: $(date)" >> "$LOG_FILE"

Third-Party Tools via mports

Restic

restic is a modern backup program that is fast, efficient, and secure. It supports deduplication, compression, and encrypted backups to various storage backends.

Key features:

Installation:

# mport install restic

Initializing a repository:

# Create a local repository
restic -r /backup/restic-repo init

# Create a repository on a remote server via SFTP
restic -r sftp:user@remotehost:/backup/restic-repo init

# Create a repository on Amazon S3
restic -r s3:s3.amazonaws.com/bucket-name init

Basic backup and restore:

# Backup a directory
restic -r /backup/restic-repo backup /home

# List snapshots
restic -r /backup/restic-repo snapshots

# Restore latest snapshot to /restore
restic -r /backup/restic-repo restore latest --target /restore

# Mount a repository for browsing
restic -r /backup/restic-repo mount /mnt/restic &

Automating with restic:

#!/bin/sh
# Automated restic backup script
REPO="/backup/restic-repo"
BACKUP_DIRS="/home /etc /var"
PASSWORD_FILE="/root/.restic-password"

# Set environment variables for password
export RESTIC_PASSWORD_FILE="$PASSWORD_FILE"

# Run backup for each directory
for dir in $BACKUP_DIRS; do
    restic -r "$REPO" backup "$dir" --verbose
    if [ $? -ne 0 ]; then
        echo "Backup failed for $dir" | logger
    fi
done

# Forget old snapshots (keep last 30 days)
restic -r "$REPO" forget --keep-daily 30 --group-by paths,tags

# Check repository integrity
restic -r "$REPO" check --read-data-subset=10%

Backup to remote server:

# Backup to a remote server via SFTP
restic -r sftp:user@backup-server:/backups/midnightbsd backup /home /etc

# Set password via environment (or use password file)
export RESTIC_PASSWORD="your-secure-password"
restic -r sftp:user@backup-server:/backups/midnightbsd backup /home
Tarsnap

tarsnap is a secure, efficient online backup service for Unix-like operating systems. It uses Amazon S3 for storage and provides client-side deduplication, compression, and encryption.

Key features:

Installation:

# mport install tarsnap

Setup and configuration:

# Register and create account at https://www.tarsnap.com/
# After registration, you'll receive an account key

# Create configuration directory
tarsnap-keygen --user your-email@example.com --machine mymidnightbsd

# Store your key file securely
mv your-email+my-midnightbsd.key /root/.tarsnap/keyfile
chmod 600 /root/.tarsnap/keyfile

Basic usage:

# Create a backup archive
# tarsnap -c -f backup-$(date +%Y%m%d) /etc /home /var

# More efficient: use cache directory for deduplication
tarsnap --cachedir /var/cache/tarsnap -c -f backup-$(date +%Y%m%d) /etc /home /var

# List all archives
tarsnap --list-archives

# Restore files
tarsnap -x -f backup-20250101 -C /restore etc/passwd

# Delete old archives
tarsnap -d -f backup-20241201

Automated backup script:

#!/bin/sh
# Automated tarsnap backup
CACHE_DIR="/var/cache/tarsnap"
LOG_FILE="/var/log/tarsnap_backup.log"
BACKUP_NAME="midnightbsd-$(date +%Y%m%d-%H%M%S)"

# Create cache directory if it doesn't exist
mkdir -p "$CACHE_DIR"

# Run backup
echo "Starting tarsnap backup: $(date)" >> "$LOG_FILE"
tarsnap --cachedir "$CACHE_DIR" -c -f "$BACKUP_NAME" /etc /home /usr/local/etc >> "$LOG_FILE" 2>&1

# List archives
echo "Current archives:" >> "$LOG_FILE"
tarsnap --list-archives >> "$LOG_FILE" 2>&1

echo "Backup completed: $(date)" >> "$LOG_FILE"
Duopoly

duopoly is a powerful backup tool that creates deduplicated, encrypted, and compressed archives. It's designed to be efficient and secure for both local and remote backups.

Key features:

Installation:

# mport install duopoly

Creating your first repository:

# Create a new duopoly repository
duopoly init /backup/duopoly-repo

# For remote repositories (SSH)
duopoly init user@remotehost:/backup/duopoly-repo

Backup and restore:

# Create a backup
# duopoly backup /home /etc --repository /backup/duopoly-repo

# Create backup with description
duopoly backup /home /etc --repository /backup/duopoly-repo --description "Daily backup"

# List all backups
duopoly list --repository /backup/duopoly-repo

# Restore a backup
duopoly restore latest --repository /backup/duopoly-repo --target /restore

# Restore specific backup
duopoly restore backup-20250101 --repository /backup/duopoly-repo --target /restore

Encryption and security:

# Create encrypted repository with password
duopoly init /backup/secure-repo --password-file /root/.duopoly-pass

# Backup with encryption
duopoly backup /home --repository /backup/secure-repo --password-file /root/.duopoly-pass

Automated backup with duopoly:

#!/bin/sh
# Automated duopoly backup
REPO="/backup/duopoly-repo"
PASS_FILE="/root/.duopoly-pass"
LOG_FILE="/var/log/duopoly_backup.log"

# Run backup
echo "Starting duopoly backup: $(date)" >> "$LOG_FILE"
duopoly backup /home /etc /var --repository "$REPO" --password-file "$PASS_FILE" --description "Daily backup $(date +%Y%m%d)" >> "$LOG_FILE" 2>&1

# Prune old backups (keep last 30 days)
duopoly prune --repository "$REPO" --password-file "$PASS_FILE" --keep-daily 30 >> "$LOG_FILE" 2>&1

# Verify repository
duopoly check --repository "$REPO" --password-file "$PASS_FILE" >> "$LOG_FILE" 2>&1

echo "Backup completed: $(date)" >> "$LOG_FILE"
BorgBackup

borg is a deduplicating backup program that supports compression, authentication, and encryption. It's optimized for fast and efficient backups with powerful search and restore capabilities.

Key features:

Installation:

# mport install borgbackup

Basic usage:

# Initialize a repository
borg init --encryption=repokey /backup/borg-repo

# Create a backup
borg create --stats /backup/borg-repo::home-$(date +%Y%m%d) /home

# List archives
borg list /backup/borg-repo

# Extract files from an archive
borg extract /backup/borg-repo::home-20250101

# Mount repository for browsing
borg mount /backup/borg-repo /mnt/borg

Backup with encryption:

# Create encrypted repository
borg init --encryption=keyfile /backup/borg-repo

# Backup with passphrase
BORG_PASSPHRASE="your-passphrase" borg create /backup/borg-repo::mybackup /home /etc

# Use key file for encryption
BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg create /backup/borg-repo::mybackup /home

Automated borg backup:

#!/bin/sh
# Automated borg backup
REPO="/backup/borg-repo"
PASS_FILE="/root/.borg-passphrase"
LOG_FILE="/var/log/borg_backup.log"
ARCHIVE_NAME="midnightbsd-{now:%Y-%m-%d_%H-%M-%S}"

# Run backup
echo "Starting borg backup: $(date)" >> "$LOG_FILE"
BORG_PASSCOMMAND="cat $PASS_FILE" borg create --stats --progress \
    "$REPO::$ARCHIVE_NAME" \
    /home /etc /usr/local/etc >> "$LOG_FILE" 2>&1

# Prune old archives (keep daily for 7 days, weekly for 4 weeks, monthly for 6 months)
BORG_PASSCOMMAND="cat $PASS_FILE" borg prune --stats \
    --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
    "$REPO" >> "$LOG_FILE" 2>&1

# Check repository integrity
BORG_PASSCOMMAND="cat $PASS_FILE" borg check "$REPO" >> "$LOG_FILE" 2>&1

echo "Backup completed: $(date)" >> "$LOG_FILE"

Offsite Backup Strategies

Offsite backups are crucial for disaster recovery. Here are several approaches for backing up data to remote locations:

SSH-based Backups

Using SSH for secure remote backups is a common and reliable approach:

# Using rsync over SSH
rsync -av -e "ssh -p 2222" /home/user/ user@remote:/backup/home

# Using cpdup over SSH
cpdup -v -x -s user@remote:/backup/destination /local/source

# Using tar over SSH
tar czf - /home | ssh user@remote "cat > /backup/home-$(date +%Y%m%d).tar.gz"

SSH key-based authentication:

# Generate SSH key on backup server
ssh-keygen -t ed25519 -f /root/.ssh/backup_key

# Copy public key to remote server
ssh-copy-id -i /root/.ssh/backup_key.pub user@remote

# Use specific key for backup scripts
rsync -av -e "ssh -i /root/.ssh/backup_key" /home user@remote:/backup
Cloud Storage Backups

Many tools support cloud storage providers for offsite backups:

Restic with cloud providers:

# Amazon S3
restic -r s3:s3.amazonaws.com/bucket-name backup /home

# Backblaze B2
restic -r b2:bucket-name backup /home

# Microsoft Azure
restic -r azure:container-name backup /home

# Google Cloud Storage
restic -r gs:bucket-name backup /home

Configuration for cloud providers:

# Set environment variables for AWS S3
AWS_ACCESS_KEY_ID="your-access-key"
AWS_SECRET_ACCESS_KEY="your-secret-key"
restic -r s3:s3.amazonaws.com/bucket-name backup /home

# Or use configuration file
cat > ~/.aws/credentials << EOF
[default]
aws_access_key_id = your-access-key
aws_secret_access_key = your-secret-key
EOF
Network Attached Storage (NAS)

Using NAS devices for backups:

# Mount NAS share
mount -t nfs nas-server:/volume1/backups /mnt/nas

# Backup using rsync
rsync -av /home /mnt/nas/midnightbsd-backup

# Unmount after backup
umount /mnt/nas
External Hard Drives

Rotating external drives provide a simple offsite solution:

# List available disks
dmesg | grep da

# Mount external drive
mount /dev/da0s1 /mnt/backup

# Run backup
rsync -av --delete /home /mnt/backup/home-$(date +%Y%m%d)

# Unmount when done
umount /mnt/backup

Automated external drive backup:

#!/bin/sh
# Rotating drive backup script
MOUNT_POINT="/mnt/backup"
LOG_FILE="/var/log/external_backup.log"

# Check for external drive
if ! ls /dev/da*; then
    echo "No external drive found" | logger
    exit 1
fi

# Get the first external drive
DRIVE=$(ls /dev/da* | head -1)

# Create mount point and mount
mkdir -p "$MOUNT_POINT"
mount "$DRIVE" "$MOUNT_POINT"
if [ $? -ne 0 ]; then
    echo "Failed to mount $DRIVE" | logger
    exit 1
fi

# Run backup
echo "Starting external drive backup: $(date)" >> "$LOG_FILE"
rsync -av --delete /home "$MOUNT_POINT/backup-$(date +%Y%m%d)" >> "$LOG_FILE" 2>&1

# Sync and unmount
sync
umount "$MOUNT_POINT"

echo "External backup completed: $(date)" >> "$LOG_FILE"

Automating Backups

Automating backups ensures they happen regularly and consistently. Here are several approaches:

Cron Jobs

Use cron for scheduled backups:

# Edit root's crontab
crontab -e

# Daily backup at 2:00 AM
0 2 * * * /usr/local/bin/daily-backup.sh >> /var/log/daily-backup.log 2>&1

# Weekly full backup on Sundays at 3:00 AM
0 3 * * 0 /usr/local/bin/weekly-backup.sh >> /var/log/weekly-backup.log 2>&1

# Monthly backup on 1st at 4:00 AM
0 4 1 * * /usr/local/bin/monthly-backup.sh >> /var/log/monthly-backup.log 2>&1

Cron examples for different backup tools:

# Daily rsync backup
0 2 * * * rsync -av --delete /home /mnt/backup/home >> /var/log/rsync.log 2>&1

# Weekly restic backup
0 3 * * 0 restic -r /backup/restic-repo backup /home /etc >> /var/log/restic.log 2>&1

# Monthly tarsnap backup
0 4 1 * * tarsnap -c -f monthly-$(date +%Y%m) /home /etc >> /var/log/tarsnap.log 2>&1
Systemd Timers (if available)

For systems with systemd, timers can be used instead of cron:

# Create a service file: /usr/local/etc/systemd/system/backup.service
[Unit]
Description=Daily Backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/daily-backup.sh
User=root
StandardOutput=journal
StandardError=journal

# Create a timer file: /usr/local/etc/systemd/system/backup.timer
[Unit]
Description=Daily Backup Timer

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

# Enable the timer
# systemctl enable backup.timer
# systemctl start backup.timer
Backup Wrapper Script

A comprehensive backup wrapper script that handles multiple backup methods:

#!/bin/sh
# Comprehensive backup wrapper
BACKUP_CONFIG="/etc/backup.conf"
LOG_DIR="/var/log/backups"

# Load configuration
. "$BACKUP_CONFIG"

# Create log directory
mkdir -p "$LOG_DIR"

# Function for logging
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "${LOG_DIR}/backup-$(date +%Y%m%d).log"
}

# Function to check if command exists
command_exists() {
    command -v "$1" >/dev/null 2>&1
}

# Run backups based on configuration
log "Starting backup sequence"

# If using restic
if [ "$USE_RESTIC" = "yes" ] && command_exists restic; then
    log "Running restic backup"
    restic -r "$RESTIC_REPO" backup "$RESTIC_SOURCES" >> "${LOG_DIR}/restic.log" 2>&1
    restic -r "$RESTIC_REPO" forget --keep-daily "$RESTIC_KEEP_DAILY" >> "${LOG_DIR}/restic.log" 2>&1
fi

# If using rsync
if [ "$USE_RSYNC" = "yes" ] && command_exists rsync; then
    log "Running rsync backup"
    rsync -av --delete "$RSYNC_SOURCES" "$RSYNC_DEST" >> "${LOG_DIR}/rsync.log" 2>&1
fi

# If using cpdup
if [ "$USE_CPDUP" = "yes" ] && command_exists cpdup; then
    log "Running cpdup backup"
    cpdup "$CPDUP_SOURCES" "$CPDUP_DEST" >> "${LOG_DIR}/cpdup.log" 2>&1
fi

log "Backup sequence completed"

Sample configuration file (/etc/backup.conf):

# Enable/disable different backup methods
USE_RESTIC="yes"
USE_RSYNC="yes"
USE_CPDUP="no"

# Restic configuration
RESTIC_REPO="/backup/restic-repo"
RESTIC_SOURCES="/home /etc /var"
RESTIC_KEEP_DAILY=30

# Rsync configuration
RSYNC_SOURCES="/home"
RSYNC_DEST="/mnt/backup/home"

# Cpdup configuration
CPDUP_SOURCES="/home"
CPDUP_DEST="/mnt/backup/home"
Email Notifications

Set up email notifications for backup success/failure:

#!/bin/sh
# Backup with email notifications
LOG_FILE="/var/log/backup.log"
EMAIL="admin@example.com"

# Run backup
rsync -av /home /backup/home >> "$LOG_FILE" 2>&1

# Check if backup succeeded
if [ $? -eq 0 ]; then
    SUBJECT="Backup Success: $(date +%Y%m%d)"
    echo "Backup completed successfully" | mail -s "$SUBJECT" "$EMAIL"
else
    SUBJECT="Backup Failed: $(date +%Y%m%d)"
    echo "Backup failed. See $LOG_FILE for details" | mail -s "$SUBJECT" "$EMAIL"
    cat "$LOG_FILE" | mail -s "Backup Log: $(date +%Y%m%d)" "$EMAIL"
fi

Backup Verification and Testing

Regular verification and testing of backups is crucial to ensure data can be restored when needed.

Verifying Backup Integrity

Most backup tools provide commands to verify repository integrity:

# Restic repository check
restic -r /backup/restic-repo check

# Borg repository check
borg check /backup/borg-repo

# Duopoly repository check
duopoly check --repository /backup/duopoly-repo

# Tarsnap integrity check (list archives)
tarsnap --list-archives
Testing Restores

Regularly test restoring files to ensure your backups work:

# Test restore with restic
restic -r /backup/restic-repo restore latest --target /tmp/test-restore
ls -la /tmp/test-restore
rm -rf /tmp/test-restore

# Test restore with borg
borg extract /backup/borg-repo::latest /tmp/test-restore
ls -la /tmp/test-restore
rm -rf /tmp/test-restore

# Test restore with rsync (verify files exist)
rsync -av --dry-run /backup/home/ /tmp/test-restore/

Automated verification script:

#!/bin/sh
# Backup verification script
VERIFY_LOG="/var/log/backup-verify.log"
TEST_DIR="/tmp/backup-verify-$(date +%Y%m%d)"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$VERIFY_LOG"
}

mkdir -p "$TEST_DIR"

# Test restic restore
if command -v restic >/dev/null 2>&1; then
    log "Testing restic restore"
    if restic -r /backup/restic-repo restore latest --target "$TEST_DIR/restic" >> "$VERIFY_LOG" 2>&1; then
        log "Restic restore: SUCCESS"
    else
        log "Restic restore: FAILED"
    fi
fi

# Test borg restore
if command -v borg >/dev/null 2>&1; then
    log "Testing borg restore"
    if borg extract /backup/borg-repo::latest "$TEST_DIR/borg" >> "$VERIFY_LOG" 2>&1; then
        log "Borg restore: SUCCESS"
    else
        log "Borg restore: FAILED"
    fi
fi

# Clean up
rm -rf "$TEST_DIR"
log "Verification completed"
Backup Log Analysis

Regularly review backup logs to catch issues early:

# Check recent backup logs
tail -n 50 /var/log/backup.log

# Search for errors
grep -i "error\|failed\|warning" /var/log/backup*.log

# Check disk space usage
du -sh /backup/*

# Check when backups were last run
ls -lt /var/log/backup*.log

Recovery Procedures

Knowing how to restore data is just as important as creating backups. Here are recovery procedures for different scenarios:

Single File Recovery

Recovering individual files from various backup types:

# From restic
restic -r /backup/restic-repo mount /mnt/restic
# Browse /mnt/restic to find your file, then copy it
cp /mnt/restic/latest/home/user/important-file /home/user/
umount /mnt/restic

# From borg
borg mount /backup/borg-repo /mnt/borg
cp /mnt/borg/latest/home/user/document.txt /home/user/
umount /mnt/borg

# From dump/restore
# Find the archive containing the file
ls -la /backup/dump/
# Mount dump archive and extract
# This typically requires knowing which dump file contains the file

# From rsync backups
cp /backup/home/user/missing-file /home/user/
Directory Recovery

Recovering entire directories:

# Using restic
restic -r /backup/restic-repo restore latest --target /restore --include /home/user

# Using borg
borg extract /backup/borg-repo::latest home/user /restore/

# Using rsync
rsync -av /backup/home/user/ /restore/user/

# Using cpdup
cpdup /backup/home/user /restore/user
Full System Recovery

For complete system recovery, you'll need a recovery environment:

  1. Boot from MidnightBSD installation media or live CD
  2. Mount your backup destination (external drive, network share, etc.)
  3. Restore critical system files first (/etc, /root, etc.)
  4. Restore user data
  5. Reinstall any packages if needed
  6. Verify system configuration

Full system restore using restic:

# Boot from live media
# Mount backup repository (adjust for your setup)
mount /dev/da0s1 /mnt

# Restore system files
restic -r /mnt/restic-repo restore latest --target / --host midnightbsd

# For ZFS systems, restore to alternate location first
restic -r /mnt/restic-repo restore latest --target /mnt/restored

# Then manually handle ZFS pool imports and data restoration

Full system restore using dump:

# Boot from live media
# Mount backup drive
mount /dev/da0s1 /mnt

# Restore each file system
cd /mnt
restore -r -f /mnt/dump/root.dump
restore -r -f /mnt/dump/var.dump
restore -r -f /mnt/dump/usr.dump
restore -r -f /mnt/dump/home.dump

Disaster Recovery Checklist:

Recovery Documentation Template:

# System Recovery Documentation
# System: midnightbsd-server
# Last Updated: $(date)

## Backup Locations
- Local: /backup
- Remote: backup-server:/backups/midnightbsd
- Cloud: s3://my-backups/midnightbsd

## Backup Tools Used
- Primary: restic
- Secondary: rsync
- Configuration: /etc/backup.conf

## Recovery Procedures
1. Boot from MidnightBSD installation USB
2. Mount backup repository: mount /dev/da0s1 /mnt
3. Run: restic -r /mnt/restic-repo restore latest --target /restore
4. Copy critical files: cp -R /restore/etc /etc
5. Reboot and verify

## Important Files to Restore First
- /etc/rc.conf
- /etc/fstab
- /etc/ssh/sshd_config
- /etc/passwd
- /etc/group
- /home/*/.ssh/authorized_keys

## Contacts for Recovery Assistance
- Admin: admin@example.com (555-1234)
- Backup Provider: support@backup-service.com