Upgrade Guide for Distributed Deployment

A comprehensive guide for upgrading ServiceOps servers across different deployment architectures with minimal downtime and maximum reliability.

In a distributed ServiceOps environment, the application and database are hosted on distinct servers. This separation requires specific upgrade procedures for each component to ensure a smooth transition and maintain system integrity. This guide details the steps for upgrading the application server and the database server independently.

ServiceOps server upgrades can be performed on distributed servers with specific considerations for the environment. From version v8.2 onwards, incremental upgrades are not required - you can upgrade directly from v8.2 to the latest version.

Key Upgrade Features

• Single Installer: From v8.6.0, use a single common installer across all supported operating systems
• Direct Upgrades: Skip incremental versions (e.g., v8.2 to latest available release version directly)
• Cross-Platform Support: Ubuntu 24 and RedHat 9.4
• Minimal Downtime: Structured upgrade procedures to minimize service interruption

Pre-Upgrade Requirements

System Requirements

Supported Operating Systems:

• Ubuntu: 22, 24
• RedHat: 9.2, 9.4

Pre-Upgrade Checklist

Before initiating any upgrade, complete these essential steps for both the Application and Database servers:

For Both Servers:

• VM Snapshot: Take a snapshot of the virtual machine for recovery
• Verify Disk Space: Ensure sufficient space for upgrade files and backups
• Review Release Notes: Check for any breaking changes or new requirements
• Schedule Maintenance Window: Plan upgrade during low-usage periods

For Application Server:

• Application Backup: Backup the application and filedb folder

For Database Server:

• Database Backup: Take a complete database backup
• Check Disk I/O Speed: Ensure that the I/O speed is greater than 150 MB/s:

  dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
• Check Available Disk Space:

  df -h
• Check Memory Availability:

  free -h
• Clear Cache:

  sudo sync; sudo sysctl -w vm.drop_caches=3

Backup Requirements

Backup Requirements: Always perform complete backups before upgrading. Refer to the Backup Procedure for detailed steps.

Application Upgrade Steps

In case of distributed server setup ( App and DB Separate ), follow the below steps in the Application server.

Step 1: Download the Release Build

1. Download the latest Common Upgrade Build from the Download Link

Step 2: Prepare the Installer

1. Copy the MotadataServiceOpsCommonUpgrade installer to the target machine

2. Login to the terminal server with root user rights:

   sudo su

3. Stop the main server and analytics server services:

   systemctl stop ft-main-server.service ft-analytics-server.service

   

4. Check the status of the services:

   systemctl status ft-main-server.service ft-analytics-server.service

   

5. Grant execute permissions for the installer file:

   chmod 777 MotadataServiceOpsCommonUpgrade_V860

   

Step 3: Run the Installer

1. Execute the installer:

   ./MotadataServiceOpsCommonUpgrade_V860

2. The upgrade procedure will begin automatically

3. Once you see the completion screen, the upgrade is successful

   

Database Upgrade Steps (Optional)

In case of distributed server setup ( App and DB Separate ), follow the below steps in the Database server (PostgreSQL).

Step 1: Stop All Services (on Database Server)

Before upgrading the PostgreSQL database, stop the PostgreSQL service to avoid conflicts during the installation:

1. Stop PostgreSQL Service: Login to the database server terminal and stop the postgresql service.

   sudo systemctl stop postgresql.service

2. Verify Service Status: Verify that the service is stopped:

   sudo systemctl status postgresql.service

   If the service is still running, repeat the stop command.

Step 2: Prepare for PostgreSQL Upgrade (on Database Server)

After ensuring that all relevant services are stopped, follow these steps to upgrade the PostgreSQL:

1. Download Upgrade Tool: Download the MotadataPGDBUpgrade file

2. Login to Server: Login to the database server terminal

3. Switch to Root: Switch to the root user

4. Set Permissions: Provide the necessary execution permissions:

   chmod 777 MotadataPGDatabaseUpgrade

   

5. Run Installer: Execute the PostgreSQL upgrade installer:

   ./MotadataPGDatabaseUpgrade

   

6. Respond to Upgrade Prompts: The script will prompt for upgrade options:

   Upgrade PostgreSQL 16 → 17? (y/n):
   Upgrade PostgreSQL 11 → 17? (y/n):

   

   Enter y for the relevant version upgrade.

7. Wait for Completion: The upgrade may take some time. Do not interrupt the process.

Step 3: Verify PostgreSQL Upgrade (on Database Server)

Once the PostgreSQL upgrade is installed, check the PostgreSQL version to confirm the upgrade was successful:

1. Switch to Postgres User:

   sudo su
   su postgres

2. Open PostgreSQL Console:

   psql

3. Check Cluster Status:

   pg_lsclusters

4. Check Version:

   select version();

   Ensure that the version displayed matches the expected upgraded version.

Step 4: Restart Services (on Database Server)

Once the PostgreSQL version is upgraded successfully, restart the postgresql service:

sudo systemctl start postgresql.service

Step 5: Monitor Logs & Services (on Database Server)

Observe the logs and services to ensure everything is running smoothly. Pay special attention to the postgresql service.

Post-Upgrade Verification

System Health Checks (Application Server)

1. Service Status: Check the status of all services on the application server:

   systemctl status ft-main-server.service ft-analytics-server.service ft-chat-server.service ft-ai-server.service ft-file-server.service ft-plugin-server.service meshcentral.service
2. Application Access: Login to the ServiceOps portal and verify the application version from Admin > Organization > Account > License Details.
3. Database Connectivity: Confirm the Application Server can connect to the upgraded database.
4. File Operations: Test file upload/download functionality.
5. Agent Communication: Verify agent connectivity and discovery.
6. Wait for Services: Wait for at least 5 minutes to allow the restart of all Motadata services.
7. Sanity Checks: Perform sanity checks and verify functionality.

System Health Checks (Database Server)

1. Service Status: Verify the postgresql service is running on the Database Server.
2. Database Connectivity: Confirm database health and accessibility.

Performance Monitoring

1. System Resources: Monitor CPU, memory, and disk usage on both Application and Database servers.
2. Response Times: Check application response times.
3. Error Rates: Monitor error logs for any issues on both servers.
4. User Feedback: Gather feedback from end users.

Troubleshooting Upgrade Issues

Common Upgrade Failures

1. Service Startup Failures: Check service status and logs
2. Database Connection Issues: Verify PostgreSQL service and connectivity
3. File Permission Errors: Ensure proper file permissions and ownership
4. Insufficient Disk Space: Check available disk space before upgrade

Log Collection for Troubleshooting

If upgrade fails, collect logs from the following locations:

Main Log Directory:

/opt/flotomate/main-server/logs/

Apollo Folder Logs (Default Tenant):

Log Category	Description	Purpose
Admin	Administrative operations	User management, permissions, system configuration
Discovery	Asset and service discovery	Network scanning, device detection, inventory updates
Error	Error tracking and debugging	Application errors, exceptions, failure logs
NSQ	Message queue operations	Inter-service communication, task processing
Shutdown	Service shutdown events	Graceful shutdown logs, cleanup operations
Agent	Agent communication logs	Agent connectivity, data collection, status updates
Elastic	Search and indexing logs	Elasticsearch operations, data indexing
Email	Email service operations	Email notifications, SMTP communication
Scheduler	Task scheduling logs	Automated job execution, cron operations
SMS	SMS service operations	Text message notifications, SMS gateway logs
CSV Import	Data import operations	CSV file processing, bulk data imports
API Logger	API request/response logs	REST API calls, authentication, rate limiting
Events	System event tracking	User actions, system events, audit trails
Service	Core service operations	Main application services, business logic
System	System-level operations	OS interactions, resource management
Audit	Security audit logs	Access attempts, security events, compliance

Common Folder Logs:

Log Category	Description	Purpose
API Logger	API request/response logs	REST API calls, authentication, rate limiting
Error	Error tracking and debugging	Application errors, exceptions, failure logs
NSQ	Message queue operations	Inter-service communication, task processing
Service	Core service operations	Main application services, business logic
Agent	Agent communication logs	Agent connectivity, data collection, status updates
Events	System event tracking	User actions, system events, audit trails
Integration	Third-party integrations	External system connections, API integrations
Shutdown	Service shutdown events	Graceful shutdown logs, cleanup operations

Recovery Procedures

1. Rollback to Snapshot: If upgrade fails, restore from VM snapshot
2. Database Restore: Restore from database backup if needed
3. Service Restart: Restart all services in proper order
4. Contact Support: Share collected logs with Motadata QA/Dev team

Related Topics

Environment-Specific Upgrades

For High Availability (HA) and Disaster Recovery (DC-DR) environment upgrades, refer to the dedicated guides:

• HA Upgrade Guide - Complete guide for HA environment upgrades
• DC-DR Upgrade Guide - Complete guide for DC-DR environment upgrades

• Backward Compatibility - Understand compatibility aspects with previous versions of ServiceOps.
• PostgreSQL Upgrade Guide - Detailed instructions for upgrading your PostgreSQL database.
• HA Upgrade Guide - Comprehensive guide for upgrading ServiceOps in High Availability environments.
• DC-DR Upgrade Guide - Step-by-step instructions for upgrading Data Center-Disaster Recovery setups.