Skip to main content

Standalone HA Deployment with DC-DR (4 Machines)

Extend 3-machine standalone HA with a DR node at a separate site and protect ServiceOps against a full DC failure with a manual switchover to the DR site.

This layout extends the 3-machine standalone HA setup with a single DR machine that acts as a Patroni replica at a separate site. Use it when you need geographic redundancy for disaster recovery alongside local high availability on the DC side.

Prerequisites​

Before starting, confirm the following:

  • Operating System: Ubuntu and RHEL
  • PostgreSQL Version: 16 or 17
  • 4 machines provisioned and reachable over the network
  • ServiceOps installed on DC Master, DC Slave, and DR Machine (follow the Standalone Installation Guide)
  • Back up your database before starting any HA configuration step if you are working on an existing production environment. See the ServiceOps Application and Database Backup Procedure.
  • Root or sudo access on all 4 machines
  • A common OS user with sudo access on all nodes
  • Setup scripts extracted on their respective machines:
    • MotadataETCDSetupU24 on the DC Observer and DR machine
    • MotadataPatroniSetupU24 on DC Master, DC Slave, and DR machine
    • MotadataAppHASetup on the DC Observer
    • MotadataHAFileSync on DC Master, DC Slave, and DR machine
    • Motadata_DCDR_Setup.sh on DC Master, DC Slave, and DR machine
  • Firewall ports open on each machine:
MachineOpen Ports
DC Observer / HAProxy / ETCD80 (Optional), 443, 2379, 2380, 5000, 7000
DC Master (APP + DB)80 (Optional), 443, 5432, 8008
DC Slave (APP + DB)80 (Optional), 443, 5432, 8008
DR Machine (APP + DB + ETCD)80 (Optional), 443, 5432, 8008 2379, 2380, 5000, 7000
Take a Database Backup First
  • Back up your database before starting any configuration step on an existing production environment. See the ServiceOps Application and Database Backup Procedure.
  • Back up your HAProxy configuration on the Observer machine from /etc/haproxy before making any changes.

Architecture Overview​

HA with DC-DR architecture showing the DC Site with an Observer Node running HA Proxy, ETCD, and HA Observer connected to a Master Server and Slave Server each running ServiceOps App and DB, and the DR Site with an Observer Node and a Master Server running ServiceOps App and DB, with File DB Sync and DB Sync connections between DC Slave and DR Master

The DC-DR layout spans two sites. The DC site runs a full 3-machine standalone HA setup. The DR site runs a single machine that handles both observer functions and the ServiceOps application with database.

SiteRoleMachineExample IP
DCObserver / HAProxy / ETCD / HA ObserverDC Observer172.16.13.163
DCAPP + DB Node 1 (Master)DC Master172.16.13.153
DCAPP + DB Node 2 (Slave)DC Slave172.16.13.159
DRETCD + APP + DB (Node 3)DR Machine172.16.13.169

Components and Roles​

  • DC Observer (HA Proxy / ETCD / HA Observer): Single entry point for all Agents, Technicians, and Requesters on the DC side. HAProxy load-balances application traffic on ports 80 and 443 and routes database connections to the Patroni Leader through port 5000 and 7000. ETCD on ports 2379 and 2380 stores cluster state and coordinates Patroni leader election across all nodes, including the DR site. The HA Observer monitors both DC servers and triggers failover scripts when the Master becomes unreachable.
  • DC Master Server (APP + DB): Active server handling all live application requests on the DC side. ServiceOps and the Patroni Leader database run on the same machine as Node 1. Ports 5432 and 8008 handle database and Patroni health checks. File DB Sync keeps non-database files in sync with the DC Slave.
  • DC Slave Server (APP + DB): Passive standby server on the DC side, kept in continuous sync with the DC Master through Data Replication and File DB Sync. Patroni automatically promotes it to Leader when the DC Master fails. This is Node 2 in the Patroni cluster. Ports 5432 and 8008 handle database and Patroni traffic.
  • DR Master Server (ETCD + APP + DB): Single disaster recovery machine at a separate site. Runs ETCD for local cluster participation, ServiceOps, and the Patroni Replica database as Node 3. DB Sync and File DB Sync connections from the DC Slave keep the DR machine current. The DR node carries nofailover: true so Patroni does not promote it automatically during DC-side failover. Traffic switches to the DR site only when you run the switchover script manually.
  • Patroni: Manages PostgreSQL replication from the DC Master across the DC Slave and the DR machine. DC-side failover promotes the DC Slave automatically. DR promotion requires a manual switchover script.
  • File DB Sync: Synchronizes non-database files between DC Master and DC Slave bidirectionally, and from the DC side to the DR machine, keeping attachments, patches, packages, and logs consistent across all nodes.

Part 1: DC Side Configuration​

Configure the DC Observer, DC Master, and DC Slave in this order before configuring the DR machine.

Step 1: Download Setup Files​

Download the zip file containing all setup files from the Download Links page and extract it on all four machines before proceeding.

Step 2: Set Up the DC Observer (ETCD and HAProxy)​

Run the following steps on the DC Observer machine (172.16.13.163).

  1. Give MotadataETCDSetupU24 execute permissions and run it:

    chmod 777 MotadataETCDSetupU24
    ./MotadataETCDSetupU24

    Terminal showing MotadataETCDSetupU24 installer starting on the DC Observer machine

  2. Type yes when prompted to install and configure ETCD.

    Terminal showing yes entered at the ETCD install and configure prompt on the DC Observer

  3. Enter 1 when prompted for the ETCD node number.

    Single ETCD Node for DC Side

    The DC side uses a single ETCD node. Leave all additional ETCD node IP prompts blank and press Enter to skip them.

    Terminal showing 1 entered at the ETCD node number prompt on the DC Observer

  4. Leave all additional ETCD node prompts blank and press Enter.

    Terminal showing blank entries at the additional ETCD node IP prompts on the DC Observer

  5. Type yes when prompted to install and configure HAProxy.

    Terminal showing yes entered at the HAProxy install and configure prompt on the DC Observer

  6. Enter the DB and APP IP addresses when prompted:

    PromptValue
    DB Node 1 IP172.16.13.153 (DC Master)
    DB Node 2 IP172.16.13.159 (DC Slave)
    APP Node 1 IP172.16.13.153 (DC Master)
    APP Node 2 IP172.16.13.159 (DC Slave)

ETCD and HAProxy setup on the DC Observer is complete.

Verify the HAProxy Configuration​

Open /etc/haproxy/haproxy.cfg and confirm it matches this structure:

global
    maxconn 100

defaults
    log global
    mode tcp
    retries 2
    timeout client 30m
    timeout connect 4s
    timeout server 30m
    timeout check 5s

listen stats
    mode http
    bind *:7000
    stats enable
    stats uri /

listen postgres
    bind *:5000
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server etcd1 172.16.13.153:5432 maxconn 100 check port 8008
    server etcd2 172.16.13.159:5432 maxconn 100 check port 8008

listen backend
    bind *:80
    balance roundrobin
    mode tcp
    option tcp-check
    server ubuntu1 172.16.13.153:80 check port 80
    server ubuntu2 172.16.13.159:80 check port 80

Verify ETCD Service Status​

Check that the ETCD service is running before continuing:

systemctl status motadata_etcd

Terminal showing DB and APP IP address entries completed during HAProxy configuration on the DC Observer

If the service is inactive, start it:

systemctl start motadata_etcd

Step 3: Configure the DC Master Database​

Run the following steps on DC Master (172.16.13.153).

  1. Give MotadataPatroniSetupU24 execute permissions and run it:

    chmod 777 MotadataPatroniSetupU24
    ./MotadataPatroniSetupU24

    Terminal showing MotadataPatroniSetupU24 starting on the DC Master machine

  2. Enter 1 when asked for the node number. This machine is Node 1 (Master).

  3. Enter the DC Slave IP address when prompted for Node 2: 172.16.13.159.

    Terminal showing Node 1 selected with DC Slave and DR machine IP addresses entered on the DC Master

  4. Enter the DR machine IP address when prompted for Node 3: 172.16.13.169.

    Terminal showing the DC ETCD IP address entered and Patroni setup completing on the DC Master

    Terminal showing Node 2 selected with DC Master and DR machine IP addresses entered on the DC Slave

  5. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

    Terminal showing the DC ETCD IP entered and Patroni service showing as inactive on the DC Master

Patroni setup on the DC Master is complete.

Terminal showing the DC Observer IP address entered during Master application configuration on the DC Master

Verify Patroni Is Inactive Before Continuing

After setup, confirm the Patroni service is inactive on the DC Master. If it shows active, stop it:

systemctl status patroni
systemctl stop patroni

Step 4: Configure the DC Slave Database​

Run the following steps on DC Slave (172.16.13.159).

  1. Give MotadataPatroniSetupU24 execute permissions and run it:

    chmod 777 MotadataPatroniSetupU24
    ./MotadataPatroniSetupU24

  2. Enter 2 when asked for the node number. This machine is Node 2 (Slave).

    Terminal showing master entered at the machine role prompt during DC Master application configuration

  3. Enter the DC Master IP address when prompted for Node 1: 172.16.13.153.

  4. Enter the DR machine IP address when prompted for Node 3: 172.16.13.169.

Terminal showing DC Master application configuration completed successfully

  1. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

    Terminal showing slave entered at the machine role prompt during DC Slave application configuration

Verify Patroni Is Inactive on the Slave Before Continuing

After setup, confirm the Patroni service on the DC Slave is inactive. If it shows active, stop it:

systemctl status patroni
systemctl stop patroni

Do not continue to Step 5 until the DC Slave Patroni service is inactive.

Terminal showing the DC Master DB password entered during DC Slave application configuration

Step 5: Configure the DC Master Application​

Run the following steps on DC Master (172.16.13.153).

  1. Copy MotadataPatroniHAMasterSlaveAppConfig from /opt/HA to the home directory.
  2. Give it execute permissions and run it:
    chmod 777 MotadataPatroniHAMasterSlaveAppConfig
    ./MotadataPatroniHAMasterSlaveAppConfig

Terminal showing DC Slave application configuration completed successfully

  1. Enter the DC Observer IP address when prompted: 172.16.13.163.

Terminal showing DC Master application configuration completed successfully

  1. Enter master when asked for the machine role.

Terminal showing DC Master application configuration completed successfully

Terminal showing DC Master application configuration completed successfully

DC Master application configuration is complete.

Step 6: Configure the DC Slave Application​

Run the following steps on DC Slave (172.16.13.159).

  1. Copy MotadataPatroniHAMasterSlaveAppConfig from the DC Master /opt/HA folder to the DC Slave.
  2. Give it execute permissions and run it:
    chmod 777 MotadataPatroniHAMasterSlaveAppConfig
    ./MotadataPatroniHAMasterSlaveAppConfig
  3. Enter the DC Observer IP address when prompted: 172.16.13.163.
  4. Enter slave when asked for the machine role.

Terminal showing DC Slave application configuration completed successfully

  1. Enter the DC Master application DB password when prompted.
Copy the DB Password from the DC Master
  • Retrieve the DB password from the DC Master configuration file located at the path, cat /opt/flotomate/main-server/lib/boot-hosted-exec.conf.
  • The password must be identical on all machines.

Terminal showing DC Slave application configuration completed successfully

DC Slave application configuration is complete.

Terminal showing DC Slave application configuration completed successfully

Step 7: Configure File Sync on the DC Master​

Run the following steps on DC Master (172.16.13.153).

  1. Change to the /opt/HA directory and run MotadataHAFileSync:
    cd /opt/HA
    ./MotadataHAFileSync

Terminal showing DC Slave application configuration completed successfully

  1. Enter the DC Slave IP address when prompted for Server 1: 172.16.13.159.

Terminal showing DC Slave application configuration completed successfully

  1. Enter the DR machine IP address when prompted for Server 2: 172.16.13.169.

Terminal showing DC Slave application configuration completed successfully

  1. Enter the common SSH username when prompted. Enter username that is available on all nodes and used for ssh between nodes.

Terminal showing DC Slave application configuration completed successfully

File sync configuration on the DC Master is complete.

Terminal showing DC Master file sync configuration completed successfully

Step 8: Configure File Sync on the DC Slave​

Run the following steps on DC Slave (172.16.13.159).

  1. Change to the /opt/HA directory and run MotadataHAFileSync:

    cd /opt/HA
    ./MotadataHAFileSync

  2. Enter the DC Master IP address when prompted for Server 1: 172.16.13.153.

  3. Enter the DR machine IP address when prompted for Server 2: 172.16.13.169.

  4. Enter the common SSH username when prompted. This username should be available on all nodes and used for ssh between nodes.

File sync configuration on the DC Slave is complete.

Step 9: Run DB Configuration on the DC Master​

Run the following steps on DC Master (172.16.13.153).

  1. Copy MotadataPatroniHADBConfig from /opt/HA to the home directory, give it execute permissions, and run it:
    chmod 777 MotadataPatroniHADBConfig
    ./MotadataPatroniHADBConfig

Terminal showing MotadataPatroniHADBConfig starting on the DC Master

Terminal showing MotadataPatroniHADBConfig starting on the DC Master

  1. Enter y when asked to reload and restart Patroni members. Perform this twice. Press Enter if prompted for a version.

Terminal showing the Patroni member reload and restart confirmation with y entered on the DC Master

Terminal showing DC Master DB configuration completed successfully

Terminal showing DC Master DB configuration completed successfully

DC Master DB configuration is complete.

Step 10: Run DB Configuration on the DC Slave​

Run the following steps on DC Slave (172.16.13.159).

  1. Run MotadataPatroniHADBConfig:
    chmod 777 MotadataPatroniHADBConfig
    ./MotadataPatroniHADBConfig

Terminal showing MotadataPatroniHADBConfig starting on the DC Slave

  1. Enter y when asked to reload and restart Patroni members. Perform this twice. Press Enter if prompted for a version.

Terminal showing MotadataPatroniHADBConfig starting on the DC Slave

DC Slave DB configuration is complete.

Terminal showing DC Slave DB configuration completed successfully

Remove Old HA Configuration

If an old HA configuration already exists, remove it after taking a backup:

rm -rf /opt/HA

Step 11: Configure the Application HA Observer​

Run the following steps on the DC Observer machine (172.16.13.163).

Run as a Standard User

Run this script as a normal user. Do not use root or sudo. The script fails when run as root.

  1. Give MotadataAppHASetup execute permissions and run it:
    chmod 777 MotadataAppHASetup
    ./MotadataAppHASetup

Terminal showing MotadataAppHASetup starting on the DC Observer as a normal user

  1. Enter the SSH username when prompted. Use the username that exists on all machines.
  2. Enter 22 for the SSH port.

Terminal showing SSH username and port 22 entered during Application HA Observer configuration

  1. Enter the DC Master IP address and password when prompted.

Terminal showing DC Master IP address and password entered during HA Observer configuration

  1. Enter the DC Slave IP address and sudo password when prompted.

Terminal showing DC Slave IP address and password entered during HA Observer configuration

  1. Press Enter at the key-pair generation prompts. Enter Y if prompted to overwrite an existing key pair.

Terminal showing SSH key pair generation complete and Application HA Observer setup done on the DC Observer

Application HA Observer configuration is complete. DC side configuration is now done. Proceed to Part 2 for DR side configuration.

Terminal showing Application HA Observer configuration is complete and DC side configuration is now done

Part 2: DR Side Configuration​

Configure the DR machine only after completing all 11 steps of Part 1. The DR machine runs ETCD, ServiceOps, and the Patroni database as Node 3 with no HAProxy.

Step 1: Set Up the DR Observer (ETCD Only)​

Run the following steps on the DR machine (172.16.13.169).

  1. Give MotadataETCDSetupU24 execute permissions and run it:
    chmod 777 MotadataETCDSetupU24
    ./MotadataETCDSetupU24

Terminal showing MotadataETCDSetupU24 starting on the DR machine

  1. Type yes when prompted to install and configure ETCD.

  2. Enter 1 when prompted for the ETCD node number.

Terminal showing ETCD IP address and node number 1 entered on the DR machine

  1. Leave all additional ETCD node IP prompts blank and press Enter. The DR side does not maintain ETCD clustering, so no additional node IPs are required.

Terminal showing all additional ETCD node IP prompts left blank and ETCD setup completing successfully on the DR machine

  1. Enter no when prompted to install and configure HAProxy. The DR machine is a single-node setup and does not require HAProxy.

Terminal showing no entered at the HAProxy install prompt on the DR machine

ETCD setup on the DR machine is complete.

Step 2: Configure the DR Database with Patroni​

Run the following steps on the DR machine (172.16.13.169).

  1. Give MotadataPatroniSetupU24 execute permissions and run it:
    chmod 777 MotadataPatroniSetupU24
    ./MotadataPatroniSetupU24

Terminal showing MotadataPatroniSetupU24 starting on the DR machine

  1. Enter 3 when asked for the node number. The DR machine is Node 3 in the Patroni cluster.

Terminal showing Node 3 selected with DC Master and DC Slave IP addresses entered on the DR machine

  1. Enter the DC Master IP address when prompted for Node 1: 172.16.13.153. Enter the DC Slave IP address when prompted for Node 2: 172.16.13.159.

Terminal showing Node 3 selected with DC Master and DC Slave IP addresses entered on the DR machine

  1. Press Enter when prompted for the PostgreSQL port to accept the default.

  2. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

    Use the DC Side ETCD IP on the DR Machine

    The DR machine connects to the DC side ETCD for cluster coordination. The DR side is down by default and stays in contact with DC side coordination. Enter the DC Observer IP here, not the DR machine's own IP.

Terminal showing Node 3 selected with DC Master and DC Slave IP addresses entered on the DR machine

Patroni setup on the DR machine is complete.

Step 3: Configure the DR Application​

Run the following steps on the DR machine (172.16.13.169).

  1. Copy MotadataPatroniHAMasterSlaveAppConfig from /opt/HA to the home directory.

  2. Give it execute permissions and run it:

    chmod 777 MotadataPatroniHAMasterSlaveAppConfig
    ./MotadataPatroniHAMasterSlaveAppConfig

  3. Enter the DR machine's own IP address when prompted for the Observer IP: 172.16.13.169.

    DR Observer IP Is the DR Machine Itself

    On the DR side, enter the DR machine's own IP as the Observer IP. Do not enter the DC Observer IP here.

Terminal showing the DR machine IP entered as the Observer IP during DR application configuration

  1. Enter slave when asked for the machine role.

Terminal showing slave selected and the DC Master DB password entered during DR application configuration

  1. Enter the DC Master application DB password when prompted.

Terminal showing slave selected and the DC Master DB password entered during DR application configuration

Use the DC Master DB Password

Retrieve the DB password from the DC Master before running this step:

cat /opt/flotomate/main-server/lib/boot-hosted-exec.conf

DR application configuration is complete.

Step 4: Configure File Sync on the DR Machine​

Run the following steps on the DR machine (172.16.13.169).

  1. Change to the /opt/HA directory and run MotadataFileSync:

    cd /opt/HA
    ./MotadataFileSync

  2. Enter the DC Master IP address when prompted for Server 1: 172.16.13.153.

Terminal showing MotadataFileSync starting on the DR machine

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Enter the DC Slave IP address when prompted for Server 2: 172.16.13.159.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Enter the common SSH username when prompted.

Terminal showing DC Master IP, DC Slave IP, and SSH username entered during file sync setup on the DR machine

File sync configuration on the DR machine is complete.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

Step 5: Run DB Configuration on the DR Machine​

Run the following steps on the DR machine (172.16.13.169).

  1. Change to the /opt/HA directory and run MotadataPatroniHADBConfig:

    cd /opt/HA
    chmod 777 MotadataPatroniHADBConfig
    ./MotadataPatroniHADBConfig

  2. Press Enter when prompted for the PostgreSQL port to accept the default.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Enter y whenever asked to reload and restart Patroni members.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Verify that three database nodes appear in the cluster output after configuration completes. The DR node shows nofailover: true.

DC-DR setup is complete. Proceed to Part 3 to configure passwordless SSH communication between all nodes.

Part 3: Configure Passwordless SSH Between All Nodes​

This step sets up SSH key-based authentication between the DC Master, DC Slave, and DR machine so that switchover scripts can execute across nodes without password prompts.

Copy the Script to the Home Directory Before Running

Copy Motadata_DCDR_Setup.sh from /opt/HA to the home directory of the normal user before running it. Run the script as a normal user, not root.

Step 1: Run on the DC Master​

Run the following steps on DC Master (172.16.13.153).

  1. Copy Motadata_DCDR_Setup.sh to the home directory and run it as a normal user:
    cp /opt/HA/Motadata_DCDR_Setup.sh ~/
    cd ~
    ./Motadata_DCDR_Setup.sh

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Enter the common SSH username when prompted.
  2. Enter 22 for the SSH port.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Enter the DC Master's own IP address when prompted for the current server IP.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Enter the password when prompted. The script generates an SSH key pair.

  2. Enter the DC Slave IP address when prompted for Server 1: 172.16.13.159.

  3. Enter the DR IP address when prompted for Server 2: 172.16.13.169.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Enter the DC Slave password when prompted. Enter it again if prompted a second time.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

Passwordless SSH setup on the DC Master is complete.

Step 2: Run on the DC Slave​

Run the following steps on DC Slave (172.16.13.159).

  1. Copy Motadata_DCDR_Setup.sh to the home directory and run it as a normal user:
    cp /opt/HA/Motadata_DCDR_Setup.sh ~/
    cd ~
    ./Motadata_DCDR_Setup.sh

Terminal showing Motadata_DCDR_Setup.sh running from the home directory on the DC Slave

  1. Enter the same SSH username and port 22.
  2. Enter the DC Slave's own IP address as the current server IP and enter the password.
  3. Enter the DC Master IP address when prompted for Server 1: 172.16.13.153.
  4. Enter the DR IP address when prompted for Server 2: 172.16.13.169
  5. Enter the DC Master password when prompted.

Passwordless SSH setup on the DC Slave is complete.

Step 3: Run on the DR Machine​

Run the following steps on the DR machine (172.16.13.169).

  1. Copy Motadata_DCDR_Setup.sh to the home directory and run it as a normal user:
    cp /opt/HA/Motadata_DCDR_Setup.sh ~/
    cd ~
    ./Motadata_DCDR_Setup.sh

Terminal showing Motadata_DCDR_Setup.sh running from the home directory on the DR machine

  1. Enter the same SSH username and port 22.

Terminal showing the SSH username and port entries on the DR machine during passwordless SSH setup

  1. Enter the DR machine's own IP address as the current server IP and enter the password.

Terminal showing the DR machine IP and password entered with SSH key pair generation in progress

  1. Enter the DC Master IP address when prompted for Server 1: 172.16.13.153.
  2. Enter the DC Master password when prompted.
  3. Enter the DC Slave IP address when prompted for Server 2: 172.16.13.159.
  4. Enter the DC Slave password when prompted.

Terminal showing the DC Master IP and password entries on the DR machine during passwordless SSH setup

Terminal showing the DC Slave IP and password entries on the DR machine during passwordless SSH setup

Passwordless SSH setup on the DR machine is complete. The DC-DR setup is now fully configured.

Switchover Procedures​

Two switchover scripts in /opt/HA control traffic direction between the DC and DR sites.

ScriptPurpose
dr2dc_switchoverActivates the DR machine as the primary when the DC site goes down
dc2dr_switchoverSyncs data from DR back to DC and prepares DC to resume as primary
Run Switchover Scripts on Both Master and Slave Machines

Run each switchover script on both the DC Master and DC Slave, and on the DR machine where indicated.

Switching Traffic from DC to DR (DC Site Down)​

Follow this procedure when the DC site is unavailable and you need the DR machine to serve as the live environment.

  1. On the DR machine, run dr2dc_switchover.sh:

    ./dr2dc_switchover.sh

  2. Enter the DR machine ETCD IP address when prompted: 172.16.13.169.

The DR machine is now the active primary site.

Switching Traffic Back from DR to DC (DC Site Restored)​

Follow this procedure when the DC site comes back online and you need to restore it as the primary.

  1. On the DC Master, run dc2dr_switchover.sh and enter the DR ETCD IP to sync data from DR back to DC:

    ./dc2dr_switchover.sh

  2. Enter the DR machine ETCD IP address when prompted: 172.16.13.169. Wait for data sync to complete before continuing.

  3. On the DC Slave, run dc2dr_switchover.sh and enter the DR ETCD IP address.

  4. Once data sync completes on both DC machines, run dr2dc_switchover.sh on the DC Master to restore DC as the active primary.

DC side is now the active primary site again.

Troubleshooting​

Use this section to diagnose common issues after completing the 4-machine DC-DR setup.

DR machine Patroni fails to join the DC cluster

Cause: The DR machine cannot reach the DC Observer ETCD on port 2379. The DR Patroni uses the DC ETCD IP, so any firewall blocking this port prevents the DR node from registering.

Fix: Confirm port 2379 is open from the DR machine to the DC Observer. Check both OS-level and network-level firewall rules:

telnet 172.16.13.163 2379
DR node does not show nofailover: true after DB configuration

Cause: The MotadataPatroniHADBConfig script did not complete successfully on the DR machine, or the reload step was skipped.

Fix: Re-run MotadataPatroniHADBConfig on the DR machine. Press Enter to accept the default PostgreSQL port and enter y at both reload prompts. Verify the cluster state after the script completes:

patronictl -c /etc/patroni/patroni.yml list
Switchover script fails with a permission error

Cause: The Motadata_DCDR_Setup.sh script was run directly from /opt/HA instead of the home directory, or it was run as root.

Fix: Copy the script to the home directory of the normal user and run it from there:

cp /opt/HA/Motadata_DCDR_Setup.sh ~/
cd ~
./Motadata_DCDR_Setup.sh
Machines cannot communicate after DC-DR setup

Cause: One or more required firewall ports are blocked between DC and DR machines.

Fix: Confirm the following ports are open between each machine pair:

SourceDestinationPorts
DC ObserverDC Master and DC Slave80 (optional), 443, 5432, 8008
DC Master and DC SlaveDC Observer2379, 2380, 5000, 7000
DR MachineDC Observer2379, 2380
DC Master and DC SlaveDR Machine5432, 8008

Check both OS-level and network-level firewall rules on each machine.