Restore Procedure

Minimize downtime and recover from data loss or system failure by mastering the ServiceOps application and database restoration procedures.

The ServiceOps Application and Database Restore Procedure is a critical process for ensuring business continuity in case of data loss, system failure, or migration scenarios. This guide provides comprehensive procedures for restoring both the ServiceOps application components and the PostgreSQL database from previously created backups.

The restoration process involves recovering application files, configurations, and database records from the most recent backup to minimize downtime and maintain operational efficiency. Proper restoration procedures ensure that all ServiceOps functionality is restored to its previous state.

Prerequisites

Before performing restore procedures, ensure you have:

• Administrative access to the ServiceOps server
• Valid backup files from a previous backup procedure
• Sufficient disk space for restoration (typically equal to or greater than backup size)
• Service status knowledge - understand which services need to be stopped
• Database credentials for PostgreSQL access
• Backup verification - confirm backup integrity before restoration

Pre-Restore Preparation

View Pre-Restore Preparation Steps

Step 1: Verify Backup Availability

1. Check backup files exist:

   ls -la /home/motadata/backupApp_DD-MM-YYYY/
   ls -la /home/motadata/backupDB_DD-MM-YYYY/

2. Verify backup integrity:

   # Check application backup structure
   find /home/motadata/backupApp_DD-MM-YYYY/ -type f | wc -l
   
   # Check database backup file
   ls -la /home/motadata/backupDB_DD-MM-YYYY/flotoitsmdb_dump

Step 2: Document Current State

1. Record current service status:

   systemctl status ft-main-server.service
   systemctl status ft-analytics-server.service
   systemctl status ft-file-server.service
   systemctl status postgresql.service

2. Document current configuration:

   # Backup current configuration before restore
   cp -r /opt/flotomate/main-server/config /tmp/current_config_backup

Application Restore

View Application Restore Steps

The application restore process involves stopping services and restoring all ServiceOps components from backup files.

Step 1: Stop All Services

Stop all running ServiceOps services to prevent conflicts during restoration:

# Check service status first
systemctl status ft-main-server.service
systemctl status ft-plugin-server.service
systemctl status ft-discovery-service.service
systemctl status ft-discovery-go.service
systemctl status ft-file-server.service
systemctl status elasticsearch.service
systemctl status nginx.service
systemctl status postgresql.service
systemctl status file-sync.service

Stop all services:

systemctl stop ft-main-server.service
systemctl stop ft-plugin-server.service
systemctl stop ft-discovery-service.service
systemctl stop ft-discovery-go.service
systemctl stop ft-file-server.service
systemctl stop elasticsearch.service
systemctl stop nginx.service
systemctl stop postgresql.service
systemctl stop file-sync.service

Step 2: Prepare Restore Environment

1. Navigate to home directory:

   cd /home/motadata

2. Access backup directory:

   Syntax: cd backupApp_DD-MM-YYYY/

   Example: cd backupApp_12-03-2025

Step 3: Restore Main Server

The main server contains core application logic and configurations:

1. Navigate to main server backup directory:

   cd main-server

2. Restore main server components:

   cp -pr lib /opt/flotomate/main-server/
   cp -pr dbs /opt/flotomate/main-server/
   cp -pr config /opt/flotomate/main-server/
   cp -pr filedb /opt/flotomate/main-server/

Step 4: Restore Analytics Server

1. Navigate to analytics server backup directory:

   cd ../analytics-server

2. Restore analytics components:

   cp -pr lib /opt/flotomate/cm-analytics/
   cp -pr config /opt/flotomate/cm-analytics/
   cp -pr filedb /opt/flotomate/cm-analytics/

Step 5: Restore File Server

1. Navigate to file server backup directory:

   cd ../fileserver

2. Restore file server components:

   cp -pr lib /opt/flotomate/fileserver/
   cp -pr config /opt/flotomate/fileserver/

Step 6: Restore Chat Server

1. Navigate to chat server backup directory:

   cd ../chat_server

2. Restore chat server files:

   cp -pr *.* /opt/flotomate/chat_server/

Step 7: Restore Plugin Server

1. Navigate to plugin server backup directory:

   cd ../plugin-server

2. Restore plugin server:

   cp -pr plugin-server /opt/flotomate/

Step 8: Restore Discovery Service

1. Navigate to discovery service backup directory:

   cd ../discovery-service

2. Restore discovery service components:

   cp -pr discovery /opt/flotomate/discovery/
   cp -pr pattern-executor /opt/flotomate/discovery/
   cp -pr app.config /opt/flotomate/discovery

Step 9: Restore System Files

1. Navigate to other files backup directory:

   cd ../otherfiles

2. Restore frontend application:

   cp -pr frontend /opt/flotomate/
   chown -R fmtuser:fmtusergroup /opt/flotomate/

3. Restore Nginx configuration:

   cp -pr nginx /etc/

4. Restore SSL certificates and keys:

   # Copy only certificate and key files
   cp -pr www.example.cert /etc/ssl/
   cp -pr www.example.key /etc/ssl/
   note

   SSL Configuration: After copying SSL files, verify the SSL certificate configuration in the fmt_nginx.conf file located in /etc/nginx/conf.d/ folder.

5. Restore PostgreSQL configuration:

   cp -pr postgresql /etc/
   chown -R postgres:postgres /etc/postgresql/

6. Restore PostgreSQL data files (Optional):

   # Rename existing main folder (optional)
   mv /var/lib/postgresql/16/main /var/lib/postgresql/16/main_DD-MM-YYYY
   
   # Restore data files
   cp -pr main /var/lib/postgresql/16/
   chown -R postgres:postgres /var/lib/postgresql/16/

Database Restore

View Database Restore Steps

The database restore process involves recreating the PostgreSQL database and restoring data from backup dumps.

Step 1: Prepare Database Backup

Download and move the latest database backup file to the /tmp folder using the below command::

Syntax:

mv DB_DATE_BACKUP.zip /tmp/
cd /tmp
unzip DB_DATE_BACKUP.zip

Example:

mv DB_22-03-2024_11-00-00_pm.zip /tmp
cd /tmp
unzip DB_22-03-2024_11-00-00_pm.zip

Step 2: Stop Application Services

Stop the main server and analytics server to prevent database conflicts:

systemctl stop ft-main-server
systemctl stop ft-analytics-server

Step 3: Recreate Database

1. Access PostgreSQL console:

   su postgres
   psql

2. Drop existing database:

   drop database flotoitsmdb;

3. Create new database:

   create database flotoitsmdb;

4. Verify database creation:

   \l

5. Exit PostgreSQL console:

   \q

Step 4: Restore Database Data

Restore database from dump file:

Syntax:

pg_restore -U postgres -d flotoitsmdb {DB_DATE_BACKUP}

Example:

pg_restore -U postgres -d flotoitsmdb DB_10-6-2021_11-00-00_pm

Post-Restore Procedures

View Post-Restore Procedures

Step 1: Restart Application Services

Start all ServiceOps services in the correct order:

# Start database first
systemctl start postgresql

# Start core services
systemctl start elasticsearch
systemctl start ft-file-server
systemctl start ft-discovery-service
systemctl start ft-discovery-go
systemctl start ft-plugin-server

# Start main application services
systemctl start ft-main-server
systemctl start ft-analytics-server

# Start web server
systemctl start nginx

Step 2: Verify Service Status

Check that all services are running correctly:

systemctl status ft-main-server
systemctl status ft-analytics-server
systemctl status ft-file-server
systemctl status postgresql
systemctl status nginx

Step 3: Test Application Functionality

1. Access the application through web browser
2. Verify login functionality
3. Check critical modules:
   • Incident Management
   • Service Request Management
   • Asset Management
   • Change Management
4. Verify data integrity by checking recent records
5. Test user permissions and role assignments

Troubleshooting

Common Restore Issues

Permission Errors:

# Fix ownership issues
chown -R fmtuser:fmtusergroup /opt/flotomate/
chown -R postgres:postgres /etc/postgresql/
chown -R postgres:postgres /var/lib/postgresql/16/

Service Start Failures:

# Check service logs
journalctl -u ft-main-server.service -f
journalctl -u postgresql.service -f

Database Connection Issues:

# Verify PostgreSQL is running
systemctl status postgresql
sudo -u postgres psql -c "\l"

Nginx Configuration Issues:

# Test nginx configuration
nginx -t
# Check nginx logs
tail -f /var/log/nginx/error.log

Recovery Verification

1. Check application logs for errors:

   tail -f /opt/flotomate/main-server/logs/application.log

2. Verify database connectivity:

   sudo -u postgres psql -d flotoitsmdb -c "SELECT COUNT(*) FROM apolo.request;"

3. Test file server functionality:

   curl -I http://localhost:8080/fileserver/health

Best Practices

View Best Practices for Restore Procedures

Before Restore

• Verify backup integrity before starting restoration
• Document current system state for rollback purposes
• Ensure sufficient disk space for restoration
• Plan maintenance window to minimize user impact

During Restore

• Follow service stop/start order to prevent conflicts
• Monitor system resources during restoration
• Keep detailed logs of restoration process
• Test each component after restoration

After Restore

• Verify all services are running correctly
• Test application functionality thoroughly
• Check data integrity across all modules -Update documentation with restoration details

Related Topics

• ServiceOps Application and Database Backup Procedure
• High Availability Deployment Setup
• DC-DR Setup

Next Steps

After completing the restore procedure:

1. Document restoration details: Record restore date, time, and any issues encountered
2. Update backup procedures: Incorporate lessons learned into backup processes
3. Test backup/restore cycle: Verify the complete backup and restore workflow
4. Review disaster recovery plan: Update procedures based on restoration experience
5. Schedule regular testing: Implement periodic restore testing in non-production environment