pkartch/basil
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 6 Wiki Activity

feat: add CI/CD pipeline, backup system, and deployment automation
Some checks failed
CI/CD Pipeline / Run Tests (pull_request) Has been cancelled
Details
CI/CD Pipeline / Code Quality (pull_request) Has been cancelled
Details
CI Pipeline / Lint Code (pull_request) Has been cancelled
Details
CI Pipeline / Test API Package (pull_request) Has been cancelled
Details
CI Pipeline / Test Web Package (pull_request) Has been cancelled
Details
CI Pipeline / Test Shared Package (pull_request) Has been cancelled
Details
Docker Build & Deploy / Build Docker Images (pull_request) Has been cancelled
Details
E2E Tests / End-to-End Tests (pull_request) Has been cancelled
Details
E2E Tests / E2E Tests (Mobile) (pull_request) Has been cancelled
Details
Security Scanning / NPM Audit (pull_request) Has been cancelled
Details
Security Scanning / Dependency License Check (pull_request) Has been cancelled
Details
Security Scanning / Code Quality Scan (pull_request) Has been cancelled
Details
Security Scanning / Docker Image Security (pull_request) Has been cancelled
Details
CI/CD Pipeline / Build and Push Docker Images (pull_request) Has been cancelled
Details
CI Pipeline / Build All Packages (pull_request) Has been cancelled
Details
CI Pipeline / Generate Coverage Report (pull_request) Has been cancelled
Details
Docker Build & Deploy / Push Docker Images (pull_request) Has been cancelled
Details
Docker Build & Deploy / Deploy to Staging (pull_request) Has been cancelled
Details
Docker Build & Deploy / Deploy to Production (pull_request) Has been cancelled
Details
Security Scanning / Security Summary (pull_request) Has been cancelled
Details

Browse Source 
## Summary
- Add complete CI/CD pipeline with Gitea Actions for automated testing, building, and deployment
- Implement backup and restore system with full database and file backup to ZIP
- Add deployment automation with webhook receiver and systemd service
- Enhance recipe editing UI with improved ingredient parsing and cooking mode features
- Add comprehensive documentation for CI/CD, deployment, and backup features

## CI/CD Pipeline
- New workflow in .gitea/workflows/ci-cd.yml with test, build, and deploy stages
- Automated Docker image building and pushing to registry
- Webhook-triggered deployments to production servers

## Backup & Restore
- New backup service with ZIP creation including database dump and uploads
- REST API endpoints for create, list, download, restore, and delete operations
- Configurable backup path via BACKUP_PATH environment variable

## Deployment
- Automated deployment scripts (deploy.sh, manual-deploy.sh)
- Webhook receiver with systemd service for deployment triggers
- Environment configuration template (.env.deploy.example)

## Documentation
- docs/CI-CD-SETUP.md - Complete CI/CD pipeline setup guide
- docs/DEPLOYMENT-QUICK-START.md - Quick deployment reference
- docs/BACKUP.md - Backup and restore documentation
- docs/REMOTE_DATABASE.md - Remote database configuration guide
- scripts/README.md - Deployment scripts documentation

## Web Improvements
- Enhanced ingredient parser with better unit and quantity detection
- Improved recipe editing interface with unified edit experience
- Better cooking mode functionality
- Updated dependencies in package.json

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Paul R Kartchner
2025-12-08 05:04:39 +00:00
parent 35a3088c30
commit d1156833a2
24 changed files with 3543 additions and 229 deletions
Download Patch File Download Diff File Expand all files Collapse all files

503
docs/CI-CD-SETUP.md Normal file
View File

@@ -0,0 +1,503 @@
# CI/CD Setup Guide for Basil
This document describes the complete CI/CD pipeline for the Basil recipe manager, including Gitea Actions workflows, Docker image building, and automated deployments.
## Table of Contents
1. [Overview](#overview)
2. [Prerequisites](#prerequisites)
3. [Gitea Actions Workflow](#gitea-actions-workflow)
4. [Docker Registry Setup](#docker-registry-setup)
5. [Deployment Methods](#deployment-methods)
6. [Configuration](#configuration)
7. [Troubleshooting](#troubleshooting)
## Overview
The CI/CD pipeline consists of three main stages:
1. **Testing**: Runs unit tests (Vitest) and E2E tests (Playwright)
2. **Build & Push**: Builds Docker images and pushes to registry (on main branch only)
3. **Deploy**: Pulls new images and restarts containers on the production server
```
┌─────────────┐ ┌──────────────┐ ┌────────────────┐
│ Git Push │────▶│ Gitea Actions│────▶│ Docker Registry│
│ (main) │ │ - Test │ │ - API image │
└─────────────┘ │ - Build │ │ - Web image │
│ - Push │ └────────────────┘
└──────────────┘ │
│ │
▼ ▼
┌──────────────┐ ┌────────────────┐
│ Webhook │────▶│ Production │
│ Trigger │ │ Server │
└──────────────┘ └────────────────┘
```
## Prerequisites
### For CI/CD (Gitea)
- Gitea instance with Actions enabled
- Docker Hub account (or other registry)
- Node.js 20+ for testing
### For Deployment Server
- Docker and Docker Compose installed
- Bash shell
- `webhook` package (for automatic deployments)
- Network access to pull from Docker registry
## Gitea Actions Workflow
The workflow is defined in `.gitea/workflows/ci-cd.yml` and runs on:
- Push to `main` or `develop` branches
- Pull requests targeting `main` or `develop`
### Jobs
#### 1. Test Job
Runs all tests with a PostgreSQL service container:
- **Unit Tests**: API, Web, and Shared packages using Vitest
- **E2E Tests**: Full application tests using Playwright
- **Database**: Temporary PostgreSQL instance for testing
**Test Commands:**
```bash
# Run all tests locally
npm test
# Run E2E tests
npm run test:e2e
# Run with coverage
npm run test:coverage
```
#### 2. Lint Job
Runs ESLint on all packages to ensure code quality:
```bash
npm run lint
```
#### 3. Build and Push Job
Only runs on push to `main` branch:
1. Builds Docker images for API and Web
2. Tags with multiple tags (latest, SHA, semver)
3. Pushes to Docker registry
4. Triggers deployment webhook
**Image Names:**
- API: `{registry}/{username}/basil-api:{tag}`
- Web: `{registry}/{username}/basil-web:{tag}`
## Docker Registry Setup
### 1. Create Docker Hub Account
If using Docker Hub:
1. Sign up at https://hub.docker.com
2. Create an access token in Account Settings → Security
### 2. Configure Gitea Secrets
Add the following secrets to your Gitea repository:
**Settings → Secrets → Actions**
| Secret Name | Description | Example |
|-------------|-------------|---------|
| `DOCKER_USERNAME` | Docker Hub username | `myusername` |
| `DOCKER_PASSWORD` | Docker Hub access token | `dckr_pat_xxxxx...` |
| `DEPLOY_WEBHOOK_URL` | Webhook endpoint URL | `http://server.com:9000/hooks/basil-deploy` |
### 3. Alternative Registries
To use a different registry (e.g., GitHub Container Registry, GitLab):
1. Update `DOCKER_REGISTRY` in `.gitea/workflows/ci-cd.yml`:
```yaml
env:
DOCKER_REGISTRY: ghcr.io # or registry.gitlab.com
```
2. Update login credentials accordingly
## Deployment Methods
### Method 1: Automatic Webhook Deployment (Recommended)
Uses a webhook server to automatically deploy when images are pushed.
#### Setup Steps
1. **Copy environment template:**
```bash
cp .env.deploy.example .env.deploy
```
2. **Edit `.env.deploy`:**
```bash
DOCKER_USERNAME=your-docker-username
DOCKER_REGISTRY=docker.io
IMAGE_TAG=latest
WEBHOOK_PORT=9000
WEBHOOK_SECRET=your-random-secret-here
```
3. **Install webhook package:**
```bash
# Ubuntu/Debian
sudo apt-get install webhook
# RHEL/CentOS
sudo yum install webhook
```
4. **Install systemd service:**
```bash
# Copy service file
sudo cp scripts/basil-webhook.service /etc/systemd/system/
# Edit service file with your settings
sudo nano /etc/systemd/system/basil-webhook.service
# Enable and start service
sudo systemctl enable basil-webhook
sudo systemctl start basil-webhook
# Check status
sudo systemctl status basil-webhook
```
5. **Configure firewall (if needed):**
```bash
sudo ufw allow 9000/tcp
```
6. **Add webhook URL to Gitea secrets:**
```
DEPLOY_WEBHOOK_URL=http://your-server.com:9000/hooks/basil-deploy
```
Add this header when calling the webhook:
```
X-Webhook-Secret: your-random-secret-here
```
#### Manual Webhook Trigger
Test webhook manually:
```bash
curl -X POST http://localhost:9000/hooks/basil-deploy \
-H "Content-Type: application/json" \
-H "X-Webhook-Secret: your-secret" \
-d '{"branch": "main", "commit": "abc123"}'
```
### Method 2: Manual Deployment
For manual deployments without webhooks:
```bash
# Interactive deployment
./scripts/manual-deploy.sh
# Or with environment variables
DOCKER_USERNAME=myuser \
DOCKER_REGISTRY=docker.io \
IMAGE_TAG=latest \
./scripts/deploy.sh
```
The deployment script will:
1. Check Docker is running
2. Create a pre-deployment backup
3. Pull latest images from registry
4. Update docker-compose configuration
5. Restart containers
6. Perform health checks
7. Clean up old images
### Method 3: Cron-based Deployment
Set up a cron job for scheduled deployments:
```bash
# Edit crontab
crontab -e
# Add line to deploy every night at 2 AM
0 2 * * * cd /srv/docker-compose/basil && DOCKER_USERNAME=myuser ./scripts/deploy.sh >> /var/log/basil-deploy.log 2>&1
```
## Configuration
### Environment Variables
**For Deployment Scripts:**
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `DOCKER_USERNAME` | Yes | - | Docker registry username |
| `DOCKER_REGISTRY` | No | `docker.io` | Docker registry URL |
| `IMAGE_TAG` | No | `latest` | Image tag to pull |
| `WEBHOOK_PORT` | No | `9000` | Port for webhook server |
| `WEBHOOK_SECRET` | No | `changeme` | Secret for webhook authentication |
**For Application:**
See `packages/api/.env.example` for application configuration.
### Docker Compose Override
The deployment script automatically creates `docker-compose.override.yml` to use registry images instead of building from source:
```yaml
services:
api:
image: docker.io/username/basil-api:latest
web:
image: docker.io/username/basil-web:latest
```
This file is in `.gitignore` and is regenerated on each deployment.
## Monitoring and Logs
### View Deployment Logs
```bash
# Deployment log
tail -f deploy.log
# Webhook log
tail -f webhook.log
# Container logs
docker-compose logs -f api
docker-compose logs -f web
```
### Check Deployment Status
```bash
# Check running containers
docker-compose ps
# Check API health
curl http://localhost:3001/health
# View recent deployments
grep "Deployment completed" deploy.log
```
### Systemd Service Logs
```bash
# View webhook service logs
sudo journalctl -u basil-webhook -f
# View recent errors
sudo journalctl -u basil-webhook --since "1 hour ago" -p err
```
## Backup and Rollback
### Automatic Backups
The deployment script automatically creates a backup before deploying:
```bash
backups/pre-deploy-YYYYMMDD-HHMMSS.zip
```
### Manual Backup
```bash
# Via API
curl -X POST http://localhost:3001/api/backup \
-o backup-$(date +%Y%m%d).zip
# Via Docker
docker exec basil-api npm run backup
```
### Rollback to Previous Version
```bash
# Pull specific tag
DOCKER_USERNAME=myuser IMAGE_TAG=main-abc123 ./scripts/deploy.sh
# Or restore from backup
curl -X POST http://localhost:3001/api/backup/restore \
-F "file=@backups/pre-deploy-20250101-020000.zip"
```
## Troubleshooting
### Tests Failing in CI
**Check test logs in Gitea:**
1. Go to Actions tab in repository
2. Click on failed workflow run
3. Expand failed job to see detailed logs
**Common issues:**
- Database connection: Ensure PostgreSQL service is healthy
- Missing dependencies: Check `npm install` step
- Environment variables: Verify test environment configuration
### Images Not Pushing
**Check Docker credentials:**
```bash
# Test Docker login
docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
# Verify secrets in Gitea
# Settings → Secrets → Actions
```
**Check registry permissions:**
- Ensure token has write permissions
- Verify repository exists on Docker Hub
### Webhook Not Triggering
**Check webhook service:**
```bash
# Service status
sudo systemctl status basil-webhook
# Check if port is listening
sudo netstat -tlnp | grep 9000
# Test webhook endpoint
curl -I http://localhost:9000/hooks/basil-deploy
```
**Check firewall:**
```bash
# Ubuntu/Debian
sudo ufw status
# RHEL/CentOS
sudo firewall-cmd --list-all
```
**Verify secret header:**
```bash
# Wrong secret returns 403
curl -X POST http://localhost:9000/hooks/basil-deploy
# Should return 403 Forbidden
# Correct secret triggers deployment
curl -X POST http://localhost:9000/hooks/basil-deploy \
-H "X-Webhook-Secret: your-secret"
# Should return "Deployment triggered successfully"
```
### Deployment Fails
**Check Docker:**
```bash
# Docker running?
docker info
# Disk space?
df -h
# View deployment log
tail -100 deploy.log
```
**Check images:**
```bash
# Can we pull images?
docker pull $DOCKER_REGISTRY/$DOCKER_USERNAME/basil-api:latest
# Check image tags
docker images | grep basil
```
**Health check failures:**
```bash
# Check API logs
docker-compose logs api
# Check database connection
docker-compose exec api npx prisma studio
# Test API manually
curl http://localhost:3001/health
```
### Container Won't Start
**Check logs:**
```bash
docker-compose logs api
docker-compose logs web
```
**Common issues:**
- Database migrations: Check Prisma migration logs
- Environment variables: Verify `.env` files
- Port conflicts: Check if ports 3001/5173 are available
- Volume permissions: Check uploads/backups directory permissions
### Rollback Failed
**Manual rollback:**
```bash
# Stop containers
docker-compose down
# Remove override file
rm docker-compose.override.yml
# Restore from backup
unzip backups/pre-deploy-YYYYMMDD-HHMMSS.zip -d restore-temp/
# Manually restore database and files
# (See backup documentation)
# Start containers
docker-compose up -d
```
## Security Considerations
1. **Webhook Secret**: Use a strong, random secret (32+ characters)
2. **Firewall**: Restrict webhook port to known IPs if possible
3. **HTTPS**: Use HTTPS for webhook endpoint in production
4. **Secrets**: Never commit secrets to git
5. **Backups**: Store backups securely with encryption
6. **Docker Registry**: Use private registries for sensitive applications
## Best Practices
1. **Test Locally**: Always test changes locally before pushing
2. **Review PRs**: Use pull requests for code review
3. **Monitor Logs**: Regularly check deployment and application logs
4. **Backup First**: Always backup before major deployments
5. **Tag Releases**: Use semantic versioning for releases
6. **Health Checks**: Monitor application health after deployment
7. **Rollback Plan**: Know how to rollback quickly if needed
## Additional Resources
- [Gitea Actions Documentation](https://docs.gitea.io/en-us/actions/)
- [Docker Documentation](https://docs.docker.com/)
- [Webhook Documentation](https://github.com/adnanh/webhook)
- [Basil Project Documentation](../CLAUDE.md)