Skip to main content

Standalone HA Deployment with DC-DR (4 Machines)

Extend 3-machine standalone HA with a DR node at a separate site and protect ServiceOps against a full DC failure with a manual switchover to the DR site.

This layout extends the 3-machine standalone HA setup with a single DR machine that acts as a Patroni replica at a separate site. Use it when you need geographic redundancy for disaster recovery alongside local high availability on the DC side.

Prerequisites​

Before starting, confirm the following:

  • Operating System: Ubuntu and RHEL
  • PostgreSQL Version: 16 or 17
  • 4 machines provisioned and reachable over the network
  • ServiceOps installed on DC Master, DC Slave, and DR Machine (follow the Standalone Installation Guide)
  • Back up your database before starting any HA configuration step if you are working on an existing production environment. See the ServiceOps Application and Database Backup Procedure.
  • Root or sudo access on all 4 machines
  • A common OS user with sudo access on all nodes
  • Setup scripts extracted on their respective machines:
    • MotadataETCDSetupU24 on the DC Observer and DR machine
    • MotadataPatroniSetupU24 on DC Master, DC Slave, and DR machine
    • MotadataAppHASetup on the DC Observer
    • MotadataHAFileSync on DC Master, DC Slave, and DR machine
  • Firewall ports open on each machine:
MachineOpen Ports
DC Observer / HAProxy / ETCD80 (Optional), 443, 2379, 2380, 5000, 7000
DC Master (APP + DB)80 (Optional), 443, 5432, 8008
DC Slave (APP + DB)80 (Optional), 443, 5432, 8008
DR Machine (APP + DB + ETCD)80 (Optional), 443, 5432, 8008 2379, 2380, 5000, 7000
Take a Database Backup First
  • Back up your database before starting any configuration step on an existing production environment. See the ServiceOps Application and Database Backup Procedure.
  • Back up your HAProxy configuration on the Observer machine from /etc/haproxy before making any changes.

Architecture Overview​

HA with DC-DR architecture showing the DC Site with an Observer Node running HA Proxy, ETCD, and HA Observer connected to a Master Server and Slave Server each running ServiceOps App and DB, and the DR Site with an Observer Node and a Master Server running ServiceOps App and DB, with File DB Sync and DB Sync connections between DC Slave and DR Master

The DC-DR layout spans two sites. The DC site runs a full 3-machine standalone HA setup. The DR site runs a single machine that handles both observer functions and the ServiceOps application with database.

SiteRoleMachineExample IP
DCObserver / HAProxy / ETCD / HA ObserverDC Observer172.16.13.163
DCAPP + DB Node 1 (Master)DC Master172.16.13.153
DCAPP + DB Node 2 (Slave)DC Slave172.16.13.159
DRETCD + APP + DB (Node 3)DR Machine172.16.13.169

Components and Roles​

  • DC Observer (HA Proxy / ETCD / HA Observer): Single entry point for all Agents, Technicians, and Requesters on the DC side. HAProxy load-balances application traffic on ports 80 and 443 and routes database connections to the Patroni Leader through port 5000 and 7000. ETCD on ports 2379 and 2380 stores cluster state and coordinates Patroni leader election across all nodes, including the DR site. The HA Observer monitors both DC servers and triggers failover scripts when the Master becomes unreachable.
  • DC Master Server (APP + DB): Active server handling all live application requests on the DC side. ServiceOps and the Patroni Leader database run on the same machine as Node 1. Ports 5432 and 8008 handle database and Patroni health checks. File DB Sync keeps non-database files in sync with the DC Slave.
  • DC Slave Server (APP + DB): Passive standby server on the DC side, kept in continuous sync with the DC Master through Data Replication and File DB Sync. Patroni automatically promotes it to Leader when the DC Master fails. This is Node 2 in the Patroni cluster. Ports 5432 and 8008 handle database and Patroni traffic.
  • DR Master Server (ETCD + APP + DB): Single disaster recovery machine at a separate site. Runs ETCD for local cluster participation, ServiceOps, and the Patroni Replica database as Node 3. DB Sync and File DB Sync connections from the DC Slave keep the DR machine current. The DR node carries nofailover: true so Patroni does not promote it automatically during DC-side failover. Traffic switches to the DR site only when you run the switchover script manually.
  • Patroni: Manages PostgreSQL replication from the DC Master across the DC Slave and the DR machine. DC-side failover promotes the DC Slave automatically. DR promotion requires a manual switchover script.
  • File DB Sync: Synchronizes non-database files between DC Master and DC Slave bidirectionally, and from the DC side to the DR machine, keeping attachments, patches, packages, and logs consistent across all nodes.

Part 1: DC Side Configuration​

Configure the DC Observer, DC Master, and DC Slave in this order before configuring the DR machine.

Step 1: Download Setup Files​

Download the zip file containing all setup files from the Download Links page and extract it on all four machines before proceeding.

Step 2: Set Up the DC Observer (ETCD and HAProxy)​

Run the following steps on the DC Observer machine (172.16.13.163).

  1. Give MotadataETCDSetupU24 execute permissions and run it:

    chmod 777 MotadataETCDSetupU24
    ./MotadataETCDSetupU24

    Terminal showing MotadataETCDSetupU24 installer starting on the DC Observer machine

  2. Type yes when prompted to install and configure ETCD.

    Terminal showing yes entered at the ETCD install and configure prompt on the DC Observer

  3. Enter 1 when prompted for the ETCD node number.

    Single ETCD Node for DC Side

    The DC side uses a single ETCD node. Leave all additional ETCD node IP prompts blank and press Enter to skip them.

    Terminal showing 1 entered at the ETCD node number prompt on the DC Observer

  4. Leave all additional ETCD node prompts blank and press Enter.

    Terminal showing blank entries at the additional ETCD node IP prompts on the DC Observer

  5. Type yes when prompted to install and configure HAProxy.

    Terminal showing yes entered at the HAProxy install and configure prompt on the DC Observer

  6. Enter the DB and APP IP addresses when prompted:

    PromptValue
    DB Node 1 IP172.16.13.153 (DC Master)
    DB Node 2 IP172.16.13.159 (DC Slave)
    APP Node 1 IP172.16.13.153 (DC Master)
    APP Node 2 IP172.16.13.159 (DC Slave)

ETCD and HAProxy setup on the DC Observer is complete.

Verify the HAProxy Configuration​

Open /etc/haproxy/haproxy.cfg and confirm it matches this structure:

global
    maxconn 100

defaults
    log global
    mode tcp
    retries 2
    timeout client 30m
    timeout connect 4s
    timeout server 30m
    timeout check 5s

listen stats
    mode http
    bind *:7000
    stats enable
    stats uri /

listen postgres
    bind *:5000
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server etcd1 172.16.13.153:5432 maxconn 100 check port 8008
    server etcd2 172.16.13.159:5432 maxconn 100 check port 8008

listen backend
    bind *:80
    balance roundrobin
    mode tcp
    option tcp-check
    server ubuntu1 172.16.13.153:80 check port 80
    server ubuntu2 172.16.13.159:80 check port 80

Verify ETCD Service Status​

Check that the ETCD service is running before continuing:

systemctl status motadata_etcd

Terminal showing DB and APP IP address entries completed during HAProxy configuration on the DC Observer

If the service is inactive, start it:

systemctl start motadata_etcd

Step 3: Configure the DC Master Database​

Run the following steps on DC Master (172.16.13.153).

  1. Give MotadataPatroniSetupU24 execute permissions and run it:

    chmod 777 MotadataPatroniSetupU24
    ./MotadataPatroniSetupU24

    Terminal showing MotadataPatroniSetupU24 starting on the DC Master machine

  2. Enter 1 when asked for the node number. This machine is Node 1 (Master).

  3. Enter the DC Slave IP address when prompted for Node 2: 172.16.13.159.

    Terminal showing Node 1 selected with DC Slave and DR machine IP addresses entered on the DC Master

  4. Enter the DR machine IP address when prompted for Node 3: 172.16.13.169.

    Terminal showing the DC ETCD IP address entered and Patroni setup completing on the DC Master

    Terminal showing Node 2 selected with DC Master and DR machine IP addresses entered on the DC Slave

  5. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

    Terminal showing the DC ETCD IP entered and Patroni service showing as inactive on the DC Master

Patroni setup on the DC Master is complete.

Terminal showing the DC Observer IP address entered during Master application configuration on the DC Master

Verify Patroni Is Inactive Before Continuing

After setup, confirm the Patroni service is inactive on the DC Master. If it shows active, stop it:

systemctl status patroni
systemctl stop patroni

Step 4: Configure the DC Slave Database​

Run the following steps on DC Slave (172.16.13.159).

  1. Give MotadataPatroniSetupU24 execute permissions and run it:

    chmod 777 MotadataPatroniSetupU24
    ./MotadataPatroniSetupU24

  2. Enter 2 when asked for the node number. This machine is Node 2 (Slave).

    Terminal showing master entered at the machine role prompt during DC Master application configuration

  3. Enter the DC Master IP address when prompted for Node 1: 172.16.13.153.

  4. Enter the DR machine IP address when prompted for Node 3: 172.16.13.169.

Terminal showing DC Master application configuration completed successfully

  1. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

    Terminal showing slave entered at the machine role prompt during DC Slave application configuration

Verify Patroni Is Inactive on the Slave Before Continuing

After setup, confirm the Patroni service on the DC Slave is inactive. If it shows active, stop it:

systemctl status patroni
systemctl stop patroni

Do not continue to Step 5 until the DC Slave Patroni service is inactive.

Terminal showing the DC Master DB password entered during DC Slave application configuration

Step 5: Configure the DC Master Application​

Run the following steps on DC Master (172.16.13.153).

  1. Copy MotadataPatroniHAMasterSlaveAppConfig from /opt/HA to the home directory.
  2. Give it execute permissions and run it:
    chmod 777 MotadataPatroniHAMasterSlaveAppConfig
    ./MotadataPatroniHAMasterSlaveAppConfig

Terminal showing DC Slave application configuration completed successfully

  1. Enter the DC Observer IP address when prompted: 172.16.13.163.

Terminal showing DC Master application configuration completed successfully

  1. Enter master when asked for the machine role.

Terminal showing DC Master application configuration completed successfully

Terminal showing DC Master application configuration completed successfully

DC Master application configuration is complete.

Step 6: Configure the DC Slave Application​

Run the following steps on DC Slave (172.16.13.159).

  1. Copy MotadataPatroniHAMasterSlaveAppConfig from the DC Master /opt/HA folder to the DC Slave.
  2. Give it execute permissions and run it:
    chmod 777 MotadataPatroniHAMasterSlaveAppConfig
    ./MotadataPatroniHAMasterSlaveAppConfig
  3. Enter the DC Observer IP address when prompted: 172.16.13.163.
  4. Enter slave when asked for the machine role.

Terminal showing DC Slave application configuration completed successfully

  1. Enter the DC Master application DB password when prompted.
Copy the DB Password from the DC Master
  • Retrieve the DB password from the DC Master configuration file located at the path, cat /opt/flotomate/main-server/lib/boot-hosted-exec.conf.
  • The password must be identical on all machines.

Terminal showing DC Slave application configuration completed successfully

DC Slave application configuration is complete.

Terminal showing DC Slave application configuration completed successfully

Step 7: Move to DC side master application machine​

  1. Execute the MotadataHAFileSync setup.

setup

  1. Upon executing it will ask for Current Server IP address.

setup

Enter IP address of the current machine, in this scenario 172.16.13.153

setup

Next it will ask for server 1 IP address

setup

Enter slave application machine IP address in this scenario 172.16.13.159

setup

Next it will ask for Server 2 IP address

ip address

Here Enter DR side application machine IP address in this scenario 172.16.13.169

dr side ip address

Next, it will ask for username, enter username which is common on all machines and have sudo access.

dr side ip address

Next it will ask for password for that user

dr side ip address

Enter password here

note

Password will not be visible at this step as it is suppressed due to security.

Next it will ask for port number for SSH

dr side ip address

Enter 22 for default

dr side ip address

Next it will ask for DB node of the current machine

dr side ip address

Enter IP address for the DB machine of the current node, here it is in same machine so enter current machine IP address => 172.16.13.153

dr side ip address

Upon entering DB node ip address your file sync setup is completed

dr side ip address

note

After completing this setup check for the below mentioned service:

Command: systemctl status lsyncd

If found inactive, start it using the below command:

Command: systemctl start lsyncd

dr side ip address

Check for cronjob entry using crontab --l

dr side ip address

Step 8: Move to DC side slave application machine​

Execute the MotadataHAFileSync setup using the below command:

./MotadataHAFileSync

run setup

Similarly it will ask for current machine IP address.

run setup

Enter Current machine IP address here 172.16.13.159.

run setup

Next it will ask for server 1 IP address.

run setup

Enter master application machine IP address in this scenario 172.16.13.153

run setup

Next it will ask for Server 2 IP address.

run setup

Here, enter DR side application machine IP address in this scenario 172.16.13.169.

run setup

Next, it will ask for username, Enter username which is common on all machines and have sudo access.

run setup

Next it will ask for password for that user

run setup

Enter password here.

note

Password will not be visible at this step as it is suppressed due to security.

Next it will ask for port number for SSH

run setup

Enter 22 for default.

run setup

Next it will ask for DB node of the current machine.

run setup

Enter IP addrress for the DB machine of the current node, Here it is in same machine so enter 172.16.13.159

run setup

Upon entering DB node ip address your file sync setup is completed.

run setup

note

After completing this setup check for below mentioned service:

Command: systemctl status lsyncd

If found active, stop it using the below command:

systemctl stop lsyncd

Check for cronjob entry using crontab --l

run setup

Step 9: Run DB Configuration on the DC Master​

Run the following steps on DC Master (172.16.13.153).

  1. Copy MotadataPatroniHADBConfig from /opt/HA to the home directory, give it execute permissions, and run it:
    chmod 777 MotadataPatroniHADBConfig
    ./MotadataPatroniHADBConfig

Terminal showing MotadataPatroniHADBConfig starting on the DC Master

Terminal showing MotadataPatroniHADBConfig starting on the DC Master

  1. Enter y when asked to reload and restart Patroni members. Perform this twice. Press Enter if prompted for a version.

Terminal showing the Patroni member reload and restart confirmation with y entered on the DC Master

Terminal showing DC Master DB configuration completed successfully

Terminal showing DC Master DB configuration completed successfully

DC Master DB configuration is complete.

Step 10: Run DB Configuration on the DC Slave​

Run the following steps on DC Slave (172.16.13.159).

  1. Run MotadataPatroniHADBConfig:
    chmod 777 MotadataPatroniHADBConfig
    ./MotadataPatroniHADBConfig

Terminal showing MotadataPatroniHADBConfig starting on the DC Slave

  1. Enter y when asked to reload and restart Patroni members. Perform this twice. Press Enter if prompted for a version.

Terminal showing MotadataPatroniHADBConfig starting on the DC Slave

DC Slave DB configuration is complete.

Terminal showing DC Slave DB configuration completed successfully

Remove Old HA Configuration

If an old HA configuration already exists, remove it after taking a backup:

rm -rf /opt/HA

Step 11: Configure the Application HA Observer​

Run the following steps on the DC Observer machine (172.16.13.163).

Run as a Standard User

Run this script as a normal user. Do not use root or sudo. The script fails when run as root.

  1. Give MotadataAppHASetup execute permissions and run it:
    chmod 777 MotadataAppHASetup
    ./MotadataAppHASetup

Terminal showing MotadataAppHASetup starting on the DC Observer as a normal user

  1. Enter the SSH username when prompted. Use the username that exists on all machines.
  2. Enter 22 for the SSH port.

Terminal showing SSH username and port 22 entered during Application HA Observer configuration

  1. Enter the DC Master IP address and password when prompted.

Terminal showing DC Master IP address and password entered during HA Observer configuration

  1. Enter the DC Slave IP address and sudo password when prompted.

Terminal showing DC Slave IP address and password entered during HA Observer configuration

  1. Press Enter at the key-pair generation prompts. Enter Y if prompted to overwrite an existing key pair.

Terminal showing SSH key pair generation complete and Application HA Observer setup done on the DC Observer

Application HA Observer configuration is complete. DC side configuration is now done. Proceed to Part 2 for DR side configuration.

Terminal showing Application HA Observer configuration is complete and DC side configuration is now done

Part 2: DR Side Configuration​

Configure the DR machine only after completing all 11 steps of Part 1. The DR machine runs ETCD, ServiceOps, and the Patroni database as Node 3 with no HAProxy.

Step 1: Set Up the DR Observer (ETCD Only)​

Run the following steps on the DR machine (172.16.13.169).

  1. Give MotadataETCDSetupU24 execute permissions and run it:
    chmod 777 MotadataETCDSetupU24
    ./MotadataETCDSetupU24

Terminal showing MotadataETCDSetupU24 starting on the DR machine

  1. Type yes when prompted to install and configure ETCD.

  2. Enter 1 when prompted for the ETCD node number.

Terminal showing ETCD IP address and node number 1 entered on the DR machine

  1. Leave all additional ETCD node IP prompts blank and press Enter. The DR side does not maintain ETCD clustering, so no additional node IPs are required.

Terminal showing all additional ETCD node IP prompts left blank and ETCD setup completing successfully on the DR machine

  1. Enter no when prompted to install and configure HAProxy. The DR machine is a single-node setup and does not require HAProxy.

Terminal showing no entered at the HAProxy install prompt on the DR machine

ETCD setup on the DR machine is complete.

Step 2: Configure the DR Database with Patroni​

Run the following steps on the DR machine (172.16.13.169).

  1. Give MotadataPatroniSetupU24 execute permissions and run it:
    chmod 777 MotadataPatroniSetupU24
    ./MotadataPatroniSetupU24

Terminal showing MotadataPatroniSetupU24 starting on the DR machine

  1. Enter 3 when asked for the node number. The DR machine is Node 3 in the Patroni cluster.

Terminal showing Node 3 selected with DC Master and DC Slave IP addresses entered on the DR machine

  1. Enter the DC Master IP address when prompted for Node 1: 172.16.13.153. Enter the DC Slave IP address when prompted for Node 2: 172.16.13.159.

Terminal showing Node 3 selected with DC Master and DC Slave IP addresses entered on the DR machine

  1. Press Enter when prompted for the PostgreSQL port to accept the default.

  2. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

    Use the DC Side ETCD IP on the DR Machine

    The DR machine connects to the DC side ETCD for cluster coordination. The DR side is down by default and stays in contact with DC side coordination. Enter the DC Observer IP here, not the DR machine's own IP.

Terminal showing Node 3 selected with DC Master and DC Slave IP addresses entered on the DR machine

Patroni setup on the DR machine is complete.

Step 3: Configure the DR Application​

Run the following steps on the DR machine (172.16.13.169).

  1. Copy MotadataPatroniHAMasterSlaveAppConfig from /opt/HA to the home directory.

  2. Give it execute permissions and run it:

    chmod 777 MotadataPatroniHAMasterSlaveAppConfig
    ./MotadataPatroniHAMasterSlaveAppConfig

  3. Enter the DR machine's own IP address when prompted for the Observer IP: 172.16.13.169.

    DR Observer IP Is the DR Machine Itself

    On the DR side, enter the DR machine's own IP as the Observer IP. Do not enter the DC Observer IP here.

Terminal showing the DR machine IP entered as the Observer IP during DR application configuration

  1. Enter slave when asked for the machine role.

Terminal showing slave selected and the DC Master DB password entered during DR application configuration

  1. Enter the DC Master application DB password when prompted.

Terminal showing slave selected and the DC Master DB password entered during DR application configuration

Use the DC Master DB Password

Retrieve the DB password from the DC Master before running this step:

cat /opt/flotomate/main-server/lib/boot-hosted-exec.conf

DR application configuration is complete.

Step 4: Move to DC side master application machine​

Execute the MotadataHAFileSync setup file.

run setup

Upon executing it will ask for Current Server IP address.

run setup

Enter IP address of the current machine, in this scenario 172.16.13.169

run setup

Next, it will ask for server 1 IP address.

run setup

Enter DC side master machine IP here 172.16.13.153.

run setup

Next, it will ask for server 2 IP address.

run setup

Enter DC side slave machine IP here 172.16.13.159.

run setup

Next, it will ask for username.

run setup

Enter username which is common on all the machines and does have sudo rights. Here flotomate is used.

run setup

Upon entering username it will ask for password of that user.

run setup

Enter password here.

note

Password will not be visible at this step as it is suppressed due to security.

Next it will ask for port number for SSH.

run setup

Enter 22 for default.

run setup

Next it will ask for DB node of the current machine.

run setup

Enter IP address for the DB machine of the current node, here it is in same machine so enter current machine IP address, 172.16.13.169.

run setup

Upon entering DB node ip address your file sync setup is completed.

run setup

note

After completing this setup check for below mentioned service

Command: systemctl status lsyncd

If found active, stop it using the below command:

systemctl stop lsyncd

Check for cronjob entry using crontab -l

Step 5: Run DB Configuration on the DR Machine​

Run the following steps on the DR machine (172.16.13.169).

  1. Change to the /opt/HA directory and run MotadataPatroniHADBConfig:

    cd /opt/HA
    chmod 777 MotadataPatroniHADBConfig
    ./MotadataPatroniHADBConfig

  2. Press Enter when prompted for the PostgreSQL port to accept the default.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Enter y whenever asked to reload and restart Patroni members.

Terminal showing DC Master IP entered as the Server 1 during file sync setup on the DR machine

  1. Verify that three database nodes appear in the cluster output after configuration completes. The DR node shows nofailover: true.

DC-DR setup is complete. Proceed to Part 3 to configure passwordless SSH communication between all nodes.

Switchover Procedures​

The DCDR_SwitchOver utility manages all reconfiguration for both directions of switchover. Run it on the appropriate machine for each phase of the procedure.

The utility presents two options when launched:

  • DC to DR (1): Reconfigures the local Patroni node to sync from the specified ETCD cluster (demotes the node to a replica).
  • DR to DC (2): Promotes the local Patroni node to the active primary role and connects it to the specified ETCD cluster.

Patroni Switchover Utility menu showing two options: DC to DR and DR to DC with a choice prompt

Take a Database Backup Before Proceeding

Each switchover phase requires downtime. Take a full database backup before starting Phase 3. Confirm there is no data discrepancy between sites before switching traffic back.

Phase 1: DC Site Is Down, Promote DR to Primary​

Run this phase on the DR machine (172.16.13.169) when the DC site is unresponsive.

  1. Run the switchover utility:

    cd /opt/HA
    ./DCDR_SwitchOver

  2. Enter 2 at the choice prompt to select DR to DC.

    Patroni Switchover Utility with option 2 entered to promote DR site to primary

  3. Enter the DR machine's ETCD IP address when prompted: 172.16.13.169.

    Switchover utility showing DR to DC selected and Enter YOUR ETCD IP address prompt

    Switchover utility showing DR ETCD IP 172.16.13.169 entered at the prompt

  4. The utility processes /etc/patroni/config.yml, creates a backup, and displays the current cluster state. When prompted to reload members, enter Y.

  5. Press Enter when asked for the restart time to accept now.

  6. Enter y to confirm the restart.

  7. Press Enter at the PG version prompt to accept the default.

    Terminal showing the full Phase 1 switchover process including member reload prompts, restart timing, restart confirmation, and completed cluster state

The DR site is now the active primary. The switchover utility only reconfigures the database. Start the ServiceOps application services on the DR machine separately after this step completes.

Phase 2: DC Site Recovers, Rejoin DC Nodes as Replicas​

Run this phase after the DC site comes back online. This syncs the DC master and DC slave to receive data from the DR site, which is now the active primary.

Phase 2.1: Rejoin the DC Master as a Replica​

Run the following steps on the DC Master (172.16.13.153).

  1. Run the switchover utility:

    cd /opt/HA
    ./DCDR_SwitchOver

  2. Enter 1 at the choice prompt to select DC to DR.

    Patroni Switchover Utility with option 1 entered on the original DC master machine

  3. Enter the DR site ETCD IP address when prompted: 172.16.13.169.

    Switchover utility showing DC to DR selected and Enter YOUR ETCD IP address prompt

    Switchover utility showing DR ETCD IP 172.16.13.169 entered on the DC master

  4. The utility processes the Patroni config and displays the current cluster state.

    Terminal showing Phase 2.1 config processing with DR ETCD IP, backup created, and cluster state output

  5. Follow the same reload and restart steps as in Phase 1: enter Y to reload, Enter for now, y to confirm, and Enter for the PG version prompt.

The DC master has rejoined the cluster as a replica syncing from the DR site.

Phase 2.2: Rejoin the DC Slave as a Replica​

Run the following steps on the DC Slave (172.16.13.159).

  1. Run the switchover utility:

    cd /opt/HA
    ./DCDR_SwitchOver

    Patroni Switchover Utility menu on the DC slave machine

  2. Enter 1 at the choice prompt to select DC to DR.

  3. Enter the DR site ETCD IP address when prompted: 172.16.13.169.

  4. Follow the same reload and restart steps as in Phase 2.1.

The DC slave has rejoined the cluster as a replica. DC is now acting as DR, and the DR site is handling all traffic.

Phase 3: Switch Traffic Back to Original DC​

Run this phase when you are ready to restore normal operations with the original DC site as primary. This phase requires downtime. Confirm there is no data discrepancy and take a database backup before starting.

Phase 3.1: Promote the DC Master Back to Primary​

Run the following steps on the DC Master (172.16.13.153).

  1. Run the switchover utility:

    cd /opt/HA
    ./DCDR_SwitchOver

    Patroni Switchover Utility menu on the original DC master machine for Phase 3

  2. Enter 2 at the choice prompt to select DR to DC.

    Switchover utility with option 2 entered on the DC master for Phase 3.1

  3. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

    Switchover utility showing DR to DC selected and Enter YOUR ETCD IP address prompt for Phase 3.1

    Switchover utility showing DC ETCD IP 172.16.13.163 entered at the prompt

  4. The utility processes the Patroni config with the DC ETCD IP.

    Terminal showing Phase 3.1 config processing with DC ETCD IP 172.16.13.163 and cluster state output

  5. Follow the same reload and restart steps as in earlier phases.

    Terminal showing Phase 3.1 switchover completed successfully with member restart confirmation

The DC master is now the active primary again.

Phase 3.2: Rejoin the DC Slave Under the DC Master​

Run the following steps on the DC Slave (172.16.13.159).

  1. Run the switchover utility:

    cd /opt/HA
    ./DCDR_SwitchOver

    Patroni Switchover Utility menu on the DC slave machine for Phase 3.2

  2. Enter 2 at the choice prompt to select DR to DC.

  3. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

  4. Follow the same reload and restart steps as in Phase 3.1.

The DC slave rejoins as a replica under the DC master.

Phase 3.3: Switch the DR Machine Back to a Replica​

Run the following steps on the DR machine (172.16.13.169).

  1. Run the switchover utility:

    cd /opt/HA
    ./DCDR_SwitchOver

    Patroni Switchover Utility menu on the DR machine for Phase 3.3

  2. Enter 1 at the choice prompt to select DC to DR.

    Switchover utility with option 1 entered on the DR machine for Phase 3.3

  3. Enter the DC Observer ETCD IP address when prompted: 172.16.13.163.

    Switchover utility showing DC to DR selected and Enter YOUR ETCD IP address prompt on the DR machine

    Switchover utility showing DC ETCD IP 172.16.13.163 entered on the DR machine

  4. The utility processes the Patroni config.

    Terminal showing Phase 3.3 config processing with DC ETCD IP 172.16.13.163 on the DR machine

  5. Follow the same reload and restart steps as in earlier phases.

    Terminal showing Phase 3.3 switchover completed successfully with Switchover Completed successfully message

The DR machine is now a replica syncing from the DC site. Normal operations are fully restored.

Troubleshooting​

Use this section to diagnose common issues after completing the 4-machine DC-DR setup.

DR machine Patroni fails to join the DC cluster

Cause: The DR machine cannot reach the DC Observer ETCD on port 2379. The DR Patroni uses the DC ETCD IP, so any firewall blocking this port prevents the DR node from registering.

Fix: Confirm port 2379 is open from the DR machine to the DC Observer. Check both OS-level and network-level firewall rules:

telnet 172.16.13.163 2379
DR node does not show nofailover: true after DB configuration

Cause: The MotadataPatroniHADBConfig script did not complete successfully on the DR machine, or the reload step was skipped.

Fix: Re-run MotadataPatroniHADBConfig on the DR machine. Press Enter to accept the default PostgreSQL port and enter y at both reload prompts. Verify the cluster state after the script completes:

patronictl -c /etc/patroni/patroni.yml list
Switchover script fails with a permission error

Cause: The Motadata_DCDR_Setup.sh script was run directly from /opt/HA instead of the home directory, or it was run as root.

Fix: Copy the script to the home directory of the normal user and run it from there:

cp /opt/HA/Motadata_DCDR_Setup.sh ~/
cd ~
./Motadata_DCDR_Setup.sh
Machines cannot communicate after DC-DR setup

Cause: One or more required firewall ports are blocked between DC and DR machines.

Fix: Confirm the following ports are open between each machine pair:

SourceDestinationPorts
DC ObserverDC Master and DC Slave80 (optional), 443, 5432, 8008
DC Master and DC SlaveDC Observer2379, 2380, 5000, 7000
DR MachineDC Observer2379, 2380
DC Master and DC SlaveDR Machine5432, 8008

Check both OS-level and network-level firewall rules on each machine.