Rollback Procedure

How to rollback Chatty AI to a previous version after a failed upgrade.

Overview

If an upgrade fails or causes issues, you can rollback to the previous working version. This procedure restores:

Previous Docker images
Previous configuration files
Previous database state (if needed)

Quick Rollback Script

Download Rollback Script

Save this script as rollback-chatty.sh:

#!/bin/bash
#=================================================================
# Chatty AI Rollback Script
# Rolls back to previous version after failed upgrade
#=================================================================

set -e  # Exit on error

# Configuration
BACKUP_PATH="${1}"
COMPOSE_DIR="${COMPOSE_DIR:-/root/chatty-app-deploy/deploy}"

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Check if backup path provided
if [ -z "${BACKUP_PATH}" ]; then
    echo -e "${RED}Error: Backup path not provided${NC}"
    echo "Usage: $0 /path/to/pre-upgrade-backup"
    echo "Example: $0 /root/chatty-backups/pre-upgrade-20260409_140000"
    exit 1
fi

# Check if backup exists
if [ ! -d "${BACKUP_PATH}" ]; then
    echo -e "${RED}Error: Backup directory not found: ${BACKUP_PATH}${NC}"
    exit 1
fi

echo -e "${GREEN}==================================================================${NC}"
echo -e "${GREEN}Chatty AI Rollback Script${NC}"
echo -e "${GREEN}==================================================================${NC}"
echo ""
echo "Backup source: ${BACKUP_PATH}"
echo "Rollback target: ${COMPOSE_DIR}"
echo ""

# Confirmation
echo -e "${YELLOW}⚠️  WARNING: This will rollback to previous version!${NC}"
echo -e "${YELLOW}Current data since upgrade will be lost.${NC}"
echo ""
read -p "Are you sure you want to rollback? (yes/no): " CONFIRM

if [ "${CONFIRM}" != "yes" ]; then
    echo "Rollback cancelled"
    exit 0
fi

echo ""
echo -e "${YELLOW}Starting rollback process...${NC}"
echo ""

# Navigate to compose directory
cd "${COMPOSE_DIR}"

# Step 1: Stop services
echo -e "${YELLOW}[1/5] Stopping services...${NC}"
docker compose down
echo -e "${GREEN}✓ Services stopped${NC}"

# Step 2: Restore configuration
echo -e "${YELLOW}[2/5] Restoring configuration files...${NC}"
if [ -f "${BACKUP_PATH}/.env" ]; then
    cp "${BACKUP_PATH}/.env" .env
    echo "✓ .env restored"
fi

if [ -f "${BACKUP_PATH}/docker-compose.yaml" ]; then
    cp "${BACKUP_PATH}/docker-compose.yaml" docker-compose.yaml
    echo "✓ docker-compose.yaml restored"
fi
echo -e "${GREEN}✓ Configuration restored${NC}"

# Step 3: Restore database (optional)
echo -e "${YELLOW}[3/5] Restoring database...${NC}"
read -p "Restore database from backup? (yes/no): " RESTORE_DB

if [ "${RESTORE_DB}" == "yes" ]; then
    # Start only database
    docker compose up -d db
    sleep 5
    
    # Restore database
    if [ -f "${BACKUP_PATH}"/postgres-*.sql.gz ]; then
        gunzip -c "${BACKUP_PATH}"/postgres-*.sql.gz | docker compose exec -T db psql -U chattyAdmin -d chattydb
        echo "✓ Database restored"
    else
        echo "⚠ No database backup found, skipping"
    fi
    
    docker compose down
fi
echo -e "${GREEN}✓ Database restore complete${NC}"

# Step 4: Pull previous images
echo -e "${YELLOW}[4/5] Pulling previous Docker images...${NC}"
docker compose pull
echo -e "${GREEN}✓ Previous images pulled${NC}"

# Step 5: Start services
echo -e "${YELLOW}[5/5] Starting services with previous version...${NC}"
docker compose up -d
echo -e "${GREEN}✓ Services started${NC}"

# Wait for services
sleep 30

# Check status
echo ""
echo -e "${YELLOW}Service status:${NC}"
docker compose ps

echo ""
echo -e "${GREEN}==================================================================${NC}"
echo -e "${GREEN}Rollback Complete!${NC}"
echo -e "${GREEN}==================================================================${NC}"
echo ""
echo -e "${YELLOW}Next steps:${NC}"
echo "1. Verify services are running: docker compose ps"
echo "2. Check logs: docker compose logs"
echo "3. Test login to Chatty AI"
echo "4. Verify functionality"
echo ""

Make Script Executable

chmod +x rollback-chatty.sh

Run Rollback

# Rollback using pre-upgrade backup
./rollback-chatty.sh /root/chatty-backups/pre-upgrade-20260409_140000

# With custom compose directory
COMPOSE_DIR=/path/to/deploy ./rollback-chatty.sh /root/chatty-backups/pre-upgrade-20260409_140000

Manual Rollback Steps

Step 1: Stop Services

cd /root/chatty-app-deploy/deploy

# Stop all services
docker compose down

# Verify stopped
docker ps -a | grep chatty-app

Step 2: Restore Configuration Files

# Restore .env file
cp /root/chatty-backups/pre-upgrade-20260409/.env .env

# Restore docker-compose.yaml
cp /root/chatty-backups/pre-upgrade-20260409/docker-compose.yaml docker-compose.yaml

# Verify
diff .env /root/chatty-backups/pre-upgrade-20260409/.env

Step 3: Restore Database (If Needed)

Only restore database if upgrade modified data:

# Start only database
docker compose up -d db
sleep 5

# Drop current database
docker compose exec db psql -U chattyAdmin -d postgres -c "DROP DATABASE chattydb;"

# Create new database
docker compose exec db psql -U chattyAdmin -d postgres -c "CREATE DATABASE chattydb OWNER chattyAdmin;"

# Restore from backup
gunzip -c /root/chatty-backups/pre-upgrade-20260409/postgres.sql.gz | \
  docker compose exec -T db psql -U chattyAdmin -d chattydb

# Verify
docker compose exec db psql -U chattyAdmin -d chattydb -c "\dt"

# Stop database
docker compose down

Step 4: Pull Previous Images

# Pull images specified in docker-compose.yaml
docker compose pull

# Or manually pull specific versions
docker pull ghcr.io/chattyai-org/chatty-app:arm64-v3.5.0
docker pull ghcr.io/chattyai-org/chattyai-pipelines:amd64-v3.5.0

Step 5: Start Services

# Start all services
docker compose up -d

# Watch logs
docker compose logs -f

# Check status
docker compose ps

Step 6: Verify Rollback

# Check versions
docker images | grep chattyai-org

# Test web access
curl -k https://$(grep CHATTYAI_DOMAIN .env | cut -d= -f2)

# Test login
# Access web UI and login

Rollback Scenarios

Scenario 1: Configuration Issue Only

If only configuration is wrong (database is fine):

# Stop services
docker compose down

# Restore .env only
cp /root/chatty-backups/pre-upgrade-*/.env .env

# Start services
docker compose up -d

Scenario 2: Image Issue Only

If new images are problematic but data is fine:

# Stop services
docker compose down

# Restore docker-compose.yaml with old image versions
cp /root/chatty-backups/pre-upgrade-*/docker-compose.yaml docker-compose.yaml

# Pull old images
docker compose pull

# Start services
docker compose up -d

Scenario 3: Database Migration Failed

If database migration failed:

# Stop services
docker compose down

# Restore entire database
docker compose up -d db
sleep 5

gunzip -c /root/chatty-backups/pre-upgrade-*/postgres.sql.gz | \
  docker compose exec -T db psql -U chattyAdmin -d chattydb

docker compose down

# Restore configuration
cp /root/chatty-backups/pre-upgrade-*/.env .env
cp /root/chatty-backups/pre-upgrade-*/docker-compose.yaml docker-compose.yaml

# Start with old version
docker compose up -d

Scenario 4: Complete Rollback

Full rollback including all data:

# Use restore script
/path/to/restore-chatty.sh /root/chatty-backups/pre-upgrade-20260409

# Or manual full restore
# See Restore Procedure documentation

Portainer Rollback

For Portainer deployments (vendor-managed):

1. Contact Vendor

Immediately contact Chatty AI support:

Describe the issue
Provide error logs
Request rollback

2. Vendor Performs Rollback

Vendor will:

Stop current deployment
Deploy previous version images
Restore configuration if needed
Verify services working
Confirm with customer

3. Customer Verification

After vendor rollback:

Services accessible
Login working
Data intact
Functionality restored

Post-Rollback Verification

1. Service Health

# All services running
docker compose ps

# No errors in logs
docker compose logs | grep -i error

# Resource usage normal
docker stats --no-stream

2. Web Access

# Test all services
curl -k https://$(grep CHATTYAI_DOMAIN .env | cut -d= -f2)
curl -k https://$(grep N8N_DOMAIN .env | cut -d= -f2)
curl -k https://$(grep DATABASES_DOMAIN .env | cut -d= -f2)

3. Functionality Test

4. Data Integrity

# Check user count
docker compose exec db psql -U chattyAdmin -d chattydb -c "SELECT COUNT(*) FROM users;"

# Check conversations
docker compose exec db psql -U chattyAdmin -d chattydb -c "SELECT COUNT(*) FROM chats;"

# Check Qdrant collections
curl http://localhost:6333/collections

Troubleshooting Rollback

Rollback Fails - Database Restore Error

# Check database is running
docker compose ps db

# Check database logs
docker compose logs db

# Try manual restore
docker compose exec db psql -U chattyAdmin -d chattydb < backup.sql

Services Won't Start After Rollback

# Check logs
docker compose logs

# Verify configuration
docker compose config

# Recreate containers
docker compose down
docker compose up -d --force-recreate

Old Images Not Available

# Check available images
docker images | grep chattyai-org

# List all tags in registry
# Contact vendor for specific version

# Use backup images if available
docker load < /path/to/image-backup.tar

Data Loss After Rollback

# Check if backup is complete
ls -la /root/chatty-backups/pre-upgrade-*/

# Try alternative backup
ls -la /root/chatty-backups/

# Contact support if data critical

Preventing Rollback Needs

1. Always Test in Staging

# Test upgrade in staging first
# Verify all functionality
# Document any issues
# Then upgrade production

2. Create Complete Backups

# Full backup before upgrade
./backup-chatty.sh

# Verify backup
ls -la /root/chatty-backups/latest/

3. Review Release Notes

# Check for breaking changes
cat deploy/CHANGELOG.md

# Review upgrade requirements
cat deploy/UPGRADE.md

4. Schedule Adequate Time

Allow 2-3x expected upgrade time
Schedule during off-peak hours
Have support available
Plan for rollback if needed

Rollback Checklist

Pre-Rollback

During Rollback

Post-Rollback

After Successful Rollback

1. Document the Issue

# Create incident report
cat > /root/upgrade-incident-$(date +%Y%m%d).txt <<EOF
Upgrade Incident Report
Date: $(date)
Version attempted: [version]
Issue: [description]
Rollback performed: Yes
Current version: [version]
Data loss: [yes/no]
Lessons learned: [notes]
EOF

2. Analyze Root Cause

What went wrong?
Why did it happen?
How to prevent next time?
What warnings were missed?

3. Plan Next Upgrade

Review failed upgrade steps
Identify additional testing needed
Update upgrade procedure
Schedule retry with fixes

4. Communicate

Notify stakeholders
Update users
Share lessons learned
Document improvements

Manual Upgrade (Docker Compose) - Upgrade procedure
Upgrade via Portainer - Portainer upgrades
Backup Procedure - Creating backups
Restore Procedure - Full restore process

Overview​

Quick Rollback Script​

Download Rollback Script​

Make Script Executable​

Run Rollback​

Manual Rollback Steps​

Step 1: Stop Services​

Step 2: Restore Configuration Files​

Step 3: Restore Database (If Needed)​

Step 4: Pull Previous Images​

Step 5: Start Services​

Step 6: Verify Rollback​

Rollback Scenarios​

Scenario 1: Configuration Issue Only​

Scenario 2: Image Issue Only​

Scenario 3: Database Migration Failed​

Scenario 4: Complete Rollback​

Portainer Rollback​

1. Contact Vendor​

2. Vendor Performs Rollback​

3. Customer Verification​

Post-Rollback Verification​

1. Service Health​

2. Web Access​

3. Functionality Test​

4. Data Integrity​

Troubleshooting Rollback​

Rollback Fails - Database Restore Error​

Services Won't Start After Rollback​

Old Images Not Available​

Data Loss After Rollback​

Preventing Rollback Needs​

1. Always Test in Staging​

2. Create Complete Backups​

3. Review Release Notes​

4. Schedule Adequate Time​

Rollback Checklist​

Pre-Rollback​

During Rollback​

Post-Rollback​

After Successful Rollback​

1. Document the Issue​

2. Analyze Root Cause​

3. Plan Next Upgrade​

4. Communicate​

Related Documentation​