Backup SolutionsRegular backups are essential for protecting your data from hardware failures, accidental deletions, or system corruption. MidnightBSD provides several built-in backup tools and supports many third-party solutions through mports. This guide covers various backup methods to help you implement a robust backup strategy tailored to your needs.
Backup solutions range from simple file copying to sophisticated, encrypted, deduplicated backups to remote servers. The right approach depends on your data volume, sensitivity, and recovery requirements.
Follow the 3-2-1 rule for critical data:
Consider these factors when choosing backup tools:
The dump and restore utilities are traditional Unix tools for backing up
entire file systems. They are included in the MidnightBSD base system and work at the file system level.
Key features:
Basic usage:
# Full backup of a file system # dump -0 -f backupfile.dump /dev/ada0s1a # Incremental backup (levels 1-9) # dump -1 -f backupfile.dump /dev/ada0s1a # Restore from backup # cd /destination # restore -r -f backupfile.dump
For more details, see the Dump & Restore documentation.
cpdup is a fast file copying utility designed for efficient directory synchronization. It's
particularly useful for maintaining mirrors of file systems or creating backup copies with minimal
overhead.
Key features:
Installation:
# mport install cpdup
Basic usage:
# Local backup cpdup /source/directory /backup/directory # Backup to remote server via SSH cpdup -v -x -s user@remotehost:/backup/directory /local/directory # Dry run (show what would be copied) cpdup -v -n /source /backup
Common options:
-v - Verbose output-x - Cross-device copy (for different file systems)-s - Source is on remote host (via SSH)-n - Dry run (no actual copying)-0 - Copy all files (ignore timestamps)-u - Update only (skip newer files)Automated backup script with cpdup:
#!/bin/sh
# Backup home directories to external drive
BACKUP_SRC="/home"
BACKUP_DEST="/mnt/backup/home"
LOG_FILE="/var/log/cpdup_backup.log"
# Ensure backup destination is mounted
if ! mount | grep -q "$BACKUP_DEST"; then
mount /dev/da1s1 /mnt/backup
if [ $? -ne 0 ]; then
echo "Failed to mount backup drive" | logger
exit 1
fi
fi
# Run the backup
echo "Starting cpdup backup: $(date)" >> "$LOG_FILE"
cpdup -v -x "$BACKUP_SRC" "$BACKUP_DEST" >> "$LOG_FILE" 2>&1
# Unmount backup drive
umount /mnt/backup
echo "Backup completed: $(date)" >> "$LOG_FILE"
rsync is a versatile file synchronization tool that efficiently transfers and synchronizes
files between directories, local drives, or remote servers. It's widely used for backups due to its
efficiency and flexibility.
Key features:
Installation:
# mport install rsync
Basic usage:
# Local directory sync rsync -av /source/directory/ /backup/directory/ # Remote backup via SSH rsync -av -e ssh /local/source/ user@remote:/backup/destination/ # Dry run (show what would be transferred) rsync -avn /source/ /backup/ # Compressed transfer rsync -avz /source/ remote:/backup/
Common options:
-a - Archive mode (preserves permissions, ownership, timestamps)-v - Verbose output-z - Compress data during transfer-P - Show progress during transfer--delete - Delete files in destination that no longer exist in source--dry-run - Perform a trial run without transferring files--exclude=PATTERN - Exclude files matching patternExcluding files and directories:
# Exclude specific files rsync -av --exclude='*.tmp' --exclude='*.bak' /source/ /backup/ # Exclude using a file rsync -av --exclude-from='/etc/rsync_exclude.txt' /source/ /backup/
Incremental backup with rsync:
#!/bin/sh
# Incremental backup with date-stamped directories
BACKUP_SRC="/home"
BACKUP_DEST="/backup/home"
DATE=$(date +%Y%m%d_%H%M%S)
# Create date-stamped backup directory
mkdir -p "${BACKUP_DEST}/${DATE}"
# Run rsync with compression and progress
rsync -avzP --delete "${BACKUP_SRC}/" "${BACKUP_DEST}/${DATE}/"
# Create latest symlink for easy access
rm -f "${BACKUP_DEST}/latest"
ln -s "${BACKUP_DEST}/${DATE}" "${BACKUP_DEST}/latest"
Backup to external drive with rsync:
#!/bin/sh
EXTERNAL_MOUNT="/mnt/external"
BACKUP_SRC="/home"
LOG_FILE="/var/log/rsync_backup.log"
# Mount external drive
mount /dev/da0s1 "$EXTERNAL_MOUNT"
if [ $? -ne 0 ]; then
echo "Failed to mount external drive" | logger
exit 1
fi
# Run backup
echo "Starting rsync backup: $(date)" >> "$LOG_FILE"
rsync -av --delete --stats "$BACKUP_SRC/" "$EXTERNAL_MOUNT/backup/" >> "$LOG_FILE" 2>&1
# Unmount external drive
umount "$EXTERNAL_MOUNT"
echo "Backup completed: $(date)" >> "$LOG_FILE"
restic is a modern backup program that is fast, efficient, and secure. It supports
deduplication, compression, and encrypted backups to various storage backends.
Key features:
Installation:
# mport install restic
Initializing a repository:
# Create a local repository restic -r /backup/restic-repo init # Create a repository on a remote server via SFTP restic -r sftp:user@remotehost:/backup/restic-repo init # Create a repository on Amazon S3 restic -r s3:s3.amazonaws.com/bucket-name init
Basic backup and restore:
# Backup a directory restic -r /backup/restic-repo backup /home # List snapshots restic -r /backup/restic-repo snapshots # Restore latest snapshot to /restore restic -r /backup/restic-repo restore latest --target /restore # Mount a repository for browsing restic -r /backup/restic-repo mount /mnt/restic &
Automating with restic:
#!/bin/sh
# Automated restic backup script
REPO="/backup/restic-repo"
BACKUP_DIRS="/home /etc /var"
PASSWORD_FILE="/root/.restic-password"
# Set environment variables for password
export RESTIC_PASSWORD_FILE="$PASSWORD_FILE"
# Run backup for each directory
for dir in $BACKUP_DIRS; do
restic -r "$REPO" backup "$dir" --verbose
if [ $? -ne 0 ]; then
echo "Backup failed for $dir" | logger
fi
done
# Forget old snapshots (keep last 30 days)
restic -r "$REPO" forget --keep-daily 30 --group-by paths,tags
# Check repository integrity
restic -r "$REPO" check --read-data-subset=10%
Backup to remote server:
# Backup to a remote server via SFTP restic -r sftp:user@backup-server:/backups/midnightbsd backup /home /etc # Set password via environment (or use password file) export RESTIC_PASSWORD="your-secure-password" restic -r sftp:user@backup-server:/backups/midnightbsd backup /home
tarsnap is a secure, efficient online backup service for Unix-like operating systems. It uses
Amazon S3 for storage and provides client-side deduplication, compression, and encryption.
Key features:
Installation:
# mport install tarsnap
Setup and configuration:
# Register and create account at https://www.tarsnap.com/ # After registration, you'll receive an account key # Create configuration directory tarsnap-keygen --user your-email@example.com --machine mymidnightbsd # Store your key file securely mv your-email+my-midnightbsd.key /root/.tarsnap/keyfile chmod 600 /root/.tarsnap/keyfile
Basic usage:
# Create a backup archive # tarsnap -c -f backup-$(date +%Y%m%d) /etc /home /var # More efficient: use cache directory for deduplication tarsnap --cachedir /var/cache/tarsnap -c -f backup-$(date +%Y%m%d) /etc /home /var # List all archives tarsnap --list-archives # Restore files tarsnap -x -f backup-20250101 -C /restore etc/passwd # Delete old archives tarsnap -d -f backup-20241201
Automated backup script:
#!/bin/sh # Automated tarsnap backup CACHE_DIR="/var/cache/tarsnap" LOG_FILE="/var/log/tarsnap_backup.log" BACKUP_NAME="midnightbsd-$(date +%Y%m%d-%H%M%S)" # Create cache directory if it doesn't exist mkdir -p "$CACHE_DIR" # Run backup echo "Starting tarsnap backup: $(date)" >> "$LOG_FILE" tarsnap --cachedir "$CACHE_DIR" -c -f "$BACKUP_NAME" /etc /home /usr/local/etc >> "$LOG_FILE" 2>&1 # List archives echo "Current archives:" >> "$LOG_FILE" tarsnap --list-archives >> "$LOG_FILE" 2>&1 echo "Backup completed: $(date)" >> "$LOG_FILE"
duopoly is a powerful backup tool that creates deduplicated, encrypted, and compressed
archives. It's designed to be efficient and secure for both local and remote backups.
Key features:
Installation:
# mport install duopoly
Creating your first repository:
# Create a new duopoly repository duopoly init /backup/duopoly-repo # For remote repositories (SSH) duopoly init user@remotehost:/backup/duopoly-repo
Backup and restore:
# Create a backup # duopoly backup /home /etc --repository /backup/duopoly-repo # Create backup with description duopoly backup /home /etc --repository /backup/duopoly-repo --description "Daily backup" # List all backups duopoly list --repository /backup/duopoly-repo # Restore a backup duopoly restore latest --repository /backup/duopoly-repo --target /restore # Restore specific backup duopoly restore backup-20250101 --repository /backup/duopoly-repo --target /restore
Encryption and security:
# Create encrypted repository with password duopoly init /backup/secure-repo --password-file /root/.duopoly-pass # Backup with encryption duopoly backup /home --repository /backup/secure-repo --password-file /root/.duopoly-pass
Automated backup with duopoly:
#!/bin/sh # Automated duopoly backup REPO="/backup/duopoly-repo" PASS_FILE="/root/.duopoly-pass" LOG_FILE="/var/log/duopoly_backup.log" # Run backup echo "Starting duopoly backup: $(date)" >> "$LOG_FILE" duopoly backup /home /etc /var --repository "$REPO" --password-file "$PASS_FILE" --description "Daily backup $(date +%Y%m%d)" >> "$LOG_FILE" 2>&1 # Prune old backups (keep last 30 days) duopoly prune --repository "$REPO" --password-file "$PASS_FILE" --keep-daily 30 >> "$LOG_FILE" 2>&1 # Verify repository duopoly check --repository "$REPO" --password-file "$PASS_FILE" >> "$LOG_FILE" 2>&1 echo "Backup completed: $(date)" >> "$LOG_FILE"
borg is a deduplicating backup program that supports compression, authentication, and
encryption. It's optimized for fast and efficient backups with powerful search and restore capabilities.
Key features:
Installation:
# mport install borgbackup
Basic usage:
# Initialize a repository borg init --encryption=repokey /backup/borg-repo # Create a backup borg create --stats /backup/borg-repo::home-$(date +%Y%m%d) /home # List archives borg list /backup/borg-repo # Extract files from an archive borg extract /backup/borg-repo::home-20250101 # Mount repository for browsing borg mount /backup/borg-repo /mnt/borg
Backup with encryption:
# Create encrypted repository borg init --encryption=keyfile /backup/borg-repo # Backup with passphrase BORG_PASSPHRASE="your-passphrase" borg create /backup/borg-repo::mybackup /home /etc # Use key file for encryption BORG_PASSCOMMAND="cat /root/.borg-passphrase" borg create /backup/borg-repo::mybackup /home
Automated borg backup:
#!/bin/sh
# Automated borg backup
REPO="/backup/borg-repo"
PASS_FILE="/root/.borg-passphrase"
LOG_FILE="/var/log/borg_backup.log"
ARCHIVE_NAME="midnightbsd-{now:%Y-%m-%d_%H-%M-%S}"
# Run backup
echo "Starting borg backup: $(date)" >> "$LOG_FILE"
BORG_PASSCOMMAND="cat $PASS_FILE" borg create --stats --progress \
"$REPO::$ARCHIVE_NAME" \
/home /etc /usr/local/etc >> "$LOG_FILE" 2>&1
# Prune old archives (keep daily for 7 days, weekly for 4 weeks, monthly for 6 months)
BORG_PASSCOMMAND="cat $PASS_FILE" borg prune --stats \
--keep-daily 7 --keep-weekly 4 --keep-monthly 6 \
"$REPO" >> "$LOG_FILE" 2>&1
# Check repository integrity
BORG_PASSCOMMAND="cat $PASS_FILE" borg check "$REPO" >> "$LOG_FILE" 2>&1
echo "Backup completed: $(date)" >> "$LOG_FILE"
Offsite backups are crucial for disaster recovery. Here are several approaches for backing up data to remote locations:
Using SSH for secure remote backups is a common and reliable approach:
# Using rsync over SSH rsync -av -e "ssh -p 2222" /home/user/ user@remote:/backup/home # Using cpdup over SSH cpdup -v -x -s user@remote:/backup/destination /local/source # Using tar over SSH tar czf - /home | ssh user@remote "cat > /backup/home-$(date +%Y%m%d).tar.gz"
SSH key-based authentication:
# Generate SSH key on backup server ssh-keygen -t ed25519 -f /root/.ssh/backup_key # Copy public key to remote server ssh-copy-id -i /root/.ssh/backup_key.pub user@remote # Use specific key for backup scripts rsync -av -e "ssh -i /root/.ssh/backup_key" /home user@remote:/backup
Many tools support cloud storage providers for offsite backups:
Restic with cloud providers:
# Amazon S3 restic -r s3:s3.amazonaws.com/bucket-name backup /home # Backblaze B2 restic -r b2:bucket-name backup /home # Microsoft Azure restic -r azure:container-name backup /home # Google Cloud Storage restic -r gs:bucket-name backup /home
Configuration for cloud providers:
# Set environment variables for AWS S3 AWS_ACCESS_KEY_ID="your-access-key" AWS_SECRET_ACCESS_KEY="your-secret-key" restic -r s3:s3.amazonaws.com/bucket-name backup /home # Or use configuration file cat > ~/.aws/credentials << EOF [default] aws_access_key_id = your-access-key aws_secret_access_key = your-secret-key EOF
Using NAS devices for backups:
# Mount NAS share mount -t nfs nas-server:/volume1/backups /mnt/nas # Backup using rsync rsync -av /home /mnt/nas/midnightbsd-backup # Unmount after backup umount /mnt/nas
Rotating external drives provide a simple offsite solution:
# List available disks dmesg | grep da # Mount external drive mount /dev/da0s1 /mnt/backup # Run backup rsync -av --delete /home /mnt/backup/home-$(date +%Y%m%d) # Unmount when done umount /mnt/backup
Automated external drive backup:
#!/bin/sh
# Rotating drive backup script
MOUNT_POINT="/mnt/backup"
LOG_FILE="/var/log/external_backup.log"
# Check for external drive
if ! ls /dev/da*; then
echo "No external drive found" | logger
exit 1
fi
# Get the first external drive
DRIVE=$(ls /dev/da* | head -1)
# Create mount point and mount
mkdir -p "$MOUNT_POINT"
mount "$DRIVE" "$MOUNT_POINT"
if [ $? -ne 0 ]; then
echo "Failed to mount $DRIVE" | logger
exit 1
fi
# Run backup
echo "Starting external drive backup: $(date)" >> "$LOG_FILE"
rsync -av --delete /home "$MOUNT_POINT/backup-$(date +%Y%m%d)" >> "$LOG_FILE" 2>&1
# Sync and unmount
sync
umount "$MOUNT_POINT"
echo "External backup completed: $(date)" >> "$LOG_FILE"
Automating backups ensures they happen regularly and consistently. Here are several approaches:
Use cron for scheduled backups:
# Edit root's crontab crontab -e # Daily backup at 2:00 AM 0 2 * * * /usr/local/bin/daily-backup.sh >> /var/log/daily-backup.log 2>&1 # Weekly full backup on Sundays at 3:00 AM 0 3 * * 0 /usr/local/bin/weekly-backup.sh >> /var/log/weekly-backup.log 2>&1 # Monthly backup on 1st at 4:00 AM 0 4 1 * * /usr/local/bin/monthly-backup.sh >> /var/log/monthly-backup.log 2>&1
Cron examples for different backup tools:
# Daily rsync backup 0 2 * * * rsync -av --delete /home /mnt/backup/home >> /var/log/rsync.log 2>&1 # Weekly restic backup 0 3 * * 0 restic -r /backup/restic-repo backup /home /etc >> /var/log/restic.log 2>&1 # Monthly tarsnap backup 0 4 1 * * tarsnap -c -f monthly-$(date +%Y%m) /home /etc >> /var/log/tarsnap.log 2>&1
For systems with systemd, timers can be used instead of cron:
# Create a service file: /usr/local/etc/systemd/system/backup.service [Unit] Description=Daily Backup [Service] Type=oneshot ExecStart=/usr/local/bin/daily-backup.sh User=root StandardOutput=journal StandardError=journal # Create a timer file: /usr/local/etc/systemd/system/backup.timer [Unit] Description=Daily Backup Timer [Timer] OnCalendar=daily Persistent=true [Install] WantedBy=timers.target # Enable the timer # systemctl enable backup.timer # systemctl start backup.timer
A comprehensive backup wrapper script that handles multiple backup methods:
#!/bin/sh
# Comprehensive backup wrapper
BACKUP_CONFIG="/etc/backup.conf"
LOG_DIR="/var/log/backups"
# Load configuration
. "$BACKUP_CONFIG"
# Create log directory
mkdir -p "$LOG_DIR"
# Function for logging
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "${LOG_DIR}/backup-$(date +%Y%m%d).log"
}
# Function to check if command exists
command_exists() {
command -v "$1" >/dev/null 2>&1
}
# Run backups based on configuration
log "Starting backup sequence"
# If using restic
if [ "$USE_RESTIC" = "yes" ] && command_exists restic; then
log "Running restic backup"
restic -r "$RESTIC_REPO" backup "$RESTIC_SOURCES" >> "${LOG_DIR}/restic.log" 2>&1
restic -r "$RESTIC_REPO" forget --keep-daily "$RESTIC_KEEP_DAILY" >> "${LOG_DIR}/restic.log" 2>&1
fi
# If using rsync
if [ "$USE_RSYNC" = "yes" ] && command_exists rsync; then
log "Running rsync backup"
rsync -av --delete "$RSYNC_SOURCES" "$RSYNC_DEST" >> "${LOG_DIR}/rsync.log" 2>&1
fi
# If using cpdup
if [ "$USE_CPDUP" = "yes" ] && command_exists cpdup; then
log "Running cpdup backup"
cpdup "$CPDUP_SOURCES" "$CPDUP_DEST" >> "${LOG_DIR}/cpdup.log" 2>&1
fi
log "Backup sequence completed"
Sample configuration file (/etc/backup.conf):
# Enable/disable different backup methods USE_RESTIC="yes" USE_RSYNC="yes" USE_CPDUP="no" # Restic configuration RESTIC_REPO="/backup/restic-repo" RESTIC_SOURCES="/home /etc /var" RESTIC_KEEP_DAILY=30 # Rsync configuration RSYNC_SOURCES="/home" RSYNC_DEST="/mnt/backup/home" # Cpdup configuration CPDUP_SOURCES="/home" CPDUP_DEST="/mnt/backup/home"
Set up email notifications for backup success/failure:
#!/bin/sh
# Backup with email notifications
LOG_FILE="/var/log/backup.log"
EMAIL="admin@example.com"
# Run backup
rsync -av /home /backup/home >> "$LOG_FILE" 2>&1
# Check if backup succeeded
if [ $? -eq 0 ]; then
SUBJECT="Backup Success: $(date +%Y%m%d)"
echo "Backup completed successfully" | mail -s "$SUBJECT" "$EMAIL"
else
SUBJECT="Backup Failed: $(date +%Y%m%d)"
echo "Backup failed. See $LOG_FILE for details" | mail -s "$SUBJECT" "$EMAIL"
cat "$LOG_FILE" | mail -s "Backup Log: $(date +%Y%m%d)" "$EMAIL"
fi
Regular verification and testing of backups is crucial to ensure data can be restored when needed.
Most backup tools provide commands to verify repository integrity:
# Restic repository check restic -r /backup/restic-repo check # Borg repository check borg check /backup/borg-repo # Duopoly repository check duopoly check --repository /backup/duopoly-repo # Tarsnap integrity check (list archives) tarsnap --list-archives
Regularly test restoring files to ensure your backups work:
# Test restore with restic restic -r /backup/restic-repo restore latest --target /tmp/test-restore ls -la /tmp/test-restore rm -rf /tmp/test-restore # Test restore with borg borg extract /backup/borg-repo::latest /tmp/test-restore ls -la /tmp/test-restore rm -rf /tmp/test-restore # Test restore with rsync (verify files exist) rsync -av --dry-run /backup/home/ /tmp/test-restore/
Automated verification script:
#!/bin/sh
# Backup verification script
VERIFY_LOG="/var/log/backup-verify.log"
TEST_DIR="/tmp/backup-verify-$(date +%Y%m%d)"
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$VERIFY_LOG"
}
mkdir -p "$TEST_DIR"
# Test restic restore
if command -v restic >/dev/null 2>&1; then
log "Testing restic restore"
if restic -r /backup/restic-repo restore latest --target "$TEST_DIR/restic" >> "$VERIFY_LOG" 2>&1; then
log "Restic restore: SUCCESS"
else
log "Restic restore: FAILED"
fi
fi
# Test borg restore
if command -v borg >/dev/null 2>&1; then
log "Testing borg restore"
if borg extract /backup/borg-repo::latest "$TEST_DIR/borg" >> "$VERIFY_LOG" 2>&1; then
log "Borg restore: SUCCESS"
else
log "Borg restore: FAILED"
fi
fi
# Clean up
rm -rf "$TEST_DIR"
log "Verification completed"
Regularly review backup logs to catch issues early:
# Check recent backup logs tail -n 50 /var/log/backup.log # Search for errors grep -i "error\|failed\|warning" /var/log/backup*.log # Check disk space usage du -sh /backup/* # Check when backups were last run ls -lt /var/log/backup*.log
Knowing how to restore data is just as important as creating backups. Here are recovery procedures for different scenarios:
Recovering individual files from various backup types:
# From restic restic -r /backup/restic-repo mount /mnt/restic # Browse /mnt/restic to find your file, then copy it cp /mnt/restic/latest/home/user/important-file /home/user/ umount /mnt/restic # From borg borg mount /backup/borg-repo /mnt/borg cp /mnt/borg/latest/home/user/document.txt /home/user/ umount /mnt/borg # From dump/restore # Find the archive containing the file ls -la /backup/dump/ # Mount dump archive and extract # This typically requires knowing which dump file contains the file # From rsync backups cp /backup/home/user/missing-file /home/user/
Recovering entire directories:
# Using restic restic -r /backup/restic-repo restore latest --target /restore --include /home/user # Using borg borg extract /backup/borg-repo::latest home/user /restore/ # Using rsync rsync -av /backup/home/user/ /restore/user/ # Using cpdup cpdup /backup/home/user /restore/user
For complete system recovery, you'll need a recovery environment:
Full system restore using restic:
# Boot from live media # Mount backup repository (adjust for your setup) mount /dev/da0s1 /mnt # Restore system files restic -r /mnt/restic-repo restore latest --target / --host midnightbsd # For ZFS systems, restore to alternate location first restic -r /mnt/restic-repo restore latest --target /mnt/restored # Then manually handle ZFS pool imports and data restoration
Full system restore using dump:
# Boot from live media # Mount backup drive mount /dev/da0s1 /mnt # Restore each file system cd /mnt restore -r -f /mnt/dump/root.dump restore -r -f /mnt/dump/var.dump restore -r -f /mnt/dump/usr.dump restore -r -f /mnt/dump/home.dump
Disaster Recovery Checklist:
Recovery Documentation Template:
# System Recovery Documentation # System: midnightbsd-server # Last Updated: $(date) ## Backup Locations - Local: /backup - Remote: backup-server:/backups/midnightbsd - Cloud: s3://my-backups/midnightbsd ## Backup Tools Used - Primary: restic - Secondary: rsync - Configuration: /etc/backup.conf ## Recovery Procedures 1. Boot from MidnightBSD installation USB 2. Mount backup repository: mount /dev/da0s1 /mnt 3. Run: restic -r /mnt/restic-repo restore latest --target /restore 4. Copy critical files: cp -R /restore/etc /etc 5. Reboot and verify ## Important Files to Restore First - /etc/rc.conf - /etc/fstab - /etc/ssh/sshd_config - /etc/passwd - /etc/group - /home/*/.ssh/authorized_keys ## Contacts for Recovery Assistance - Admin: admin@example.com (555-1234) - Backup Provider: support@backup-service.com