basil/scripts/README-POSTGRES-BACKUP.md

PostgreSQL Backup Scripts

Comprehensive backup and restore scripts for PostgreSQL databases.

Scripts Overview

1. backup-all-postgres-databases.sh

Backs up all databases on a PostgreSQL server (excluding system databases).

Features:

• ✅ Backs up all user databases automatically
• ✅ Includes global objects (roles, tablespaces)
• ✅ Optional gzip compression
• ✅ Automatic retention management
• ✅ Integrity verification
• ✅ Detailed logging with color output
• ✅ Backup summary reporting
• ✅ Email/Slack notification support (optional)

2. restore-postgres-database.sh

Restores a single database from backup.

Features:

• ✅ Safety backup before restore
• ✅ Interactive confirmation
• ✅ Automatic database name detection
• ✅ Compressed file support
• ✅ Integrity verification
• ✅ Post-restore verification

---

Quick Start

Backup All Databases

# Basic usage
./backup-all-postgres-databases.sh

# With compression (recommended)
./backup-all-postgres-databases.sh -c

# Custom configuration
./backup-all-postgres-databases.sh \
    -h db.example.com \
    -U postgres \
    -d /mnt/backups \
    -r 60 \
    -c

Restore a Database

# Interactive restore (with confirmation)
./restore-postgres-database.sh /var/backups/postgresql/20260120/mydb_20260120_020001.sql.gz

# Force restore (skip confirmation)
./restore-postgres-database.sh backup.sql.gz -d mydb -f

---

Detailed Usage

Backup Script Options

./backup-all-postgres-databases.sh [options]

Options:
  -h HOST       Database host (default: localhost)
  -p PORT       Database port (default: 5432)
  -U USER       Database user (default: postgres)
  -d BACKUP_DIR Backup directory (default: /var/backups/postgresql)
  -r DAYS       Retention days (default: 30)
  -c            Enable compression (gzip)
  -v            Verbose output
  -H            Show help

Restore Script Options

./restore-postgres-database.sh <backup_file> [options]

Options:
  -h HOST       Database host (default: localhost)
  -p PORT       Database port (default: 5432)
  -U USER       Database user (default: postgres)
  -d DBNAME     Target database name (default: from filename)
  -f            Force restore (skip confirmation)
  -v            Verbose output
  -H            Show help

---

Automated Backups with Cron

Daily Backups (Recommended)

# Edit crontab
sudo crontab -e

# Add daily backup at 2 AM with compression
0 2 * * * /path/to/backup-all-postgres-databases.sh -c >> /var/log/postgres-backup.log 2>&1

Alternative Schedules

# Every 6 hours
0 */6 * * * /path/to/backup-all-postgres-databases.sh -c

# Twice daily (2 AM and 2 PM)
0 2,14 * * * /path/to/backup-all-postgres-databases.sh -c

# Weekly on Sundays at 3 AM
0 3 * * 0 /path/to/backup-all-postgres-databases.sh -c -r 90

---

Backup Directory Structure

/var/backups/postgresql/
├── 20260120/                           # Date-based subdirectory
│   ├── globals_20260120_020001.sql.gz  # Global objects backup
│   ├── basil_20260120_020001.sql.gz    # Database backup
│   ├── myapp_20260120_020001.sql.gz    # Database backup
│   └── wiki_20260120_020001.sql.gz     # Database backup
├── 20260121/
│   ├── globals_20260121_020001.sql.gz
│   └── ...
└── 20260122/
    └── ...

---

Configuration Examples

Local PostgreSQL Server

./backup-all-postgres-databases.sh \
    -h localhost \
    -U postgres \
    -c

Remote PostgreSQL Server

./backup-all-postgres-databases.sh \
    -h db.example.com \
    -p 5432 \
    -U backup_user \
    -d /mnt/network/backups \
    -r 60 \
    -c \
    -v

High-Frequency Backups

# Short retention for frequent backups
./backup-all-postgres-databases.sh \
    -r 7 \
    -c

---

Authentication Setup

Option 1: .pgpass File (Recommended)

Create ~/.pgpass with connection credentials:

echo "localhost:5432:*:postgres:your-password" >> ~/.pgpass
chmod 600 ~/.pgpass

Format: hostname:port:database:username:password

Option 2: Environment Variables

export PGPASSWORD="your-password"
./backup-all-postgres-databases.sh

Option 3: Peer Authentication (Local Only)

Run as the postgres system user:

sudo -u postgres ./backup-all-postgres-databases.sh

---

Monitoring and Notifications

Email Notifications

Edit the scripts and uncomment the email notification section:

# In backup-all-postgres-databases.sh, uncomment:
if command -v mail &> /dev/null; then
    echo "$summary" | mail -s "PostgreSQL Backup $status - $(hostname)" admin@example.com
fi

Slack Notifications

Set webhook URL and uncomment:

export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"

# In script, uncomment:
if [ -n "$SLACK_WEBHOOK_URL" ]; then
    curl -X POST "$SLACK_WEBHOOK_URL" \
        -H 'Content-Type: application/json' \
        -d "{\"text\":\"PostgreSQL Backup $status\n$summary\"}"
fi

Log Rotation

Create /etc/logrotate.d/postgres-backup:

/var/log/postgres-backup.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
}

---

Backup Verification

Manual Verification

# List backups
ls -lh /var/backups/postgresql/$(date +%Y%m%d)/

# Verify compressed file integrity
gzip -t /var/backups/postgresql/20260120/basil_20260120_020001.sql.gz

# Preview backup contents
gunzip -c backup.sql.gz | head -50

Test Restore (Recommended Monthly)

# Restore to a test database
./restore-postgres-database.sh backup.sql.gz -d test_restore -f

# Verify
psql -d test_restore -c "\dt"

# Cleanup
dropdb test_restore

---

Disaster Recovery

Full Server Restore

1. Install PostgreSQL on new server
2. Restore global objects first:

   gunzip -c globals_YYYYMMDD_HHMMSS.sql.gz | psql -U postgres -d postgres
   

3. Restore each database:

   ./restore-postgres-database.sh basil_20260120_020001.sql.gz
   ./restore-postgres-database.sh myapp_20260120_020001.sql.gz
   

Point-in-Time Recovery

For PITR capabilities, enable WAL archiving in postgresql.conf:

wal_level = replica
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/wal_archive/%f'
max_wal_senders = 3

Then use pg_basebackup and WAL replay for PITR.

---

Troubleshooting

Permission Denied

# Fix backup directory permissions
sudo chown -R postgres:postgres /var/backups/postgresql
sudo chmod 755 /var/backups/postgresql

# Fix script permissions
chmod +x backup-all-postgres-databases.sh

Connection Failed

# Test connection manually
psql -h localhost -U postgres -c "SELECT version();"

# Check pg_hba.conf
sudo cat /etc/postgresql/*/main/pg_hba.conf

# Ensure proper authentication line exists:
# local   all   postgres   peer
# host    all   all        127.0.0.1/32   scram-sha-256

Out of Disk Space

# Check disk usage
df -h /var/backups

# Clean old backups manually
find /var/backups/postgresql -type d -name "????????" -mtime +30 -exec rm -rf {} \;

# Reduce retention period
./backup-all-postgres-databases.sh -r 7

Backup File Corrupted

# Verify integrity
gzip -t backup.sql.gz

# If corrupted, use previous backup
ls -lt /var/backups/postgresql/*/basil_*.sql.gz | head

---

Performance Optimization

Large Databases

For very large databases, consider:

# Parallel dump (PostgreSQL 9.3+)
pg_dump -Fd -j 4 -f backup_directory mydb

# Custom format (smaller, faster restore)
pg_dump -Fc mydb > backup.custom

# Restore from custom format
pg_restore -d mydb backup.custom

Network Backups

# Direct SSH backup (no local storage)
pg_dump mydb | gzip | ssh backup-server "cat > /backups/mydb.sql.gz"

---

Best Practices

1. Always test restores - Backups are worthless if you can't restore
2. Monitor backup completion - Set up alerts for failed backups
3. Use compression - Saves 80-90% of disk space
4. Multiple backup locations - Local + remote/cloud storage
5. Verify backup integrity - Run gzip -t on compressed backups
6. Document procedures - Keep runbooks for disaster recovery
7. Encrypt sensitive backups - Use gpg for encryption if needed
8. Regular retention review - Adjust based on compliance requirements

---

Security Considerations

Encryption at Rest

# Encrypt backup with GPG
pg_dump mydb | gzip | gpg --encrypt --recipient admin@example.com > backup.sql.gz.gpg

# Decrypt for restore
gpg --decrypt backup.sql.gz.gpg | gunzip | psql mydb

Secure Transfer

# Use SCP with key authentication
scp -i ~/.ssh/backup_key backup.sql.gz backup-server:/secure/backups/

# Or rsync over SSH
rsync -av -e "ssh -i ~/.ssh/backup_key" \
    /var/backups/postgresql/ \
    backup-server:/secure/backups/

Access Control

# Restrict backup directory permissions
chmod 700 /var/backups/postgresql
chown postgres:postgres /var/backups/postgresql

# Restrict script permissions
chmod 750 backup-all-postgres-databases.sh
chown root:postgres backup-all-postgres-databases.sh

---

Additional Resources

• PostgreSQL Backup Documentation
• pg_dump Manual
• pg_restore Manual
• Continuous Archiving and PITR

---

Support

For issues or questions:

• Check script help: ./backup-all-postgres-databases.sh -H
• Review logs: tail -f /var/log/postgres-backup.log
• Test connection: psql -h localhost -U postgres