Skip to main content

DC-DR Setup with Patroni (Standalone Deployment)

Configure a Disaster Recovery (DR) replica for ServiceOps using Patroni - the DC site acts as the Patroni leader and the DR site joins as a continuously replicating standby, without HAProxy or automatic failover.

This guide covers standalone deployments where the application and database run on the same machine at both sites. It walks through configuring ETCD, Patroni, application settings, and file sync on both the DC and DR servers.

Architecture Overview​

DC-DR Patroni Architecture

Architecture Scope

This architecture is for a standalone deployment where the application and database run on the same machine at each site.

Utilities Reference​

The following utilities are used in this procedure. Grant each downloaded file execute permissions (chmod 777) before running it.

UtilityRun OnPurpose
MotadataDCDRPatroniSetupDC and DR machinesInstalls and configures ETCD and Patroni
MotadataPatroniDCDRConfigDC and DR machinesApplication configuration (located in /opt/HA)
MotadataPatroniDBConfigDC and DR machinesPatroni database configuration; creates the replica on DR (located in /opt/HA)
DCDRFileSyncDC and DR machinesFile sync between DC and DR (located in /opt/HA)
DCDR_SwitchOverDB machinesManual DC-DR switchover procedure

Download MotadataDCDRPatroniSetup from the Download Links page.

Prerequisites​

Before proceeding, ensure the following requirements are met:

  • System Requirements and Pre-Installation Checklist are satisfied.
  • ServiceOps is already installed on both the DC and DR machines in standalone mode.
  • The same Linux user (for example, flotomate) exists on both DC and DR, with the same password.
  • Ports 22, 5432, and 8008 are open between the DC and DR machines.
  • Ports 2379, 2380, 8008, and 5432 are open on each machine.
Take Backups First

If this setup is being configured in an existing environment, take proper backups and snapshots of both machines before proceeding.

Supported Operating Systems​

Operating SystemSupported Versions
Red Hat9.6, 9.8, and 10.2
Ubuntu24 and 26

Server Information​

SiteRoleExample IP Address
DCStandalone (APP + DB), Patroni Node 1172.20.21.23
DRStandalone (APP + DB), Patroni Node 2172.20.21.19

DC Site Configuration​

Configure the DC site first, completing all four steps before moving to the DR site.

StepActivityUtility
1ETCD and Patroni installationMotadataDCDRPatroniSetup
2Application configurationMotadataPatroniDCDRConfig
3Patroni database configurationMotadataPatroniDBConfig
4File sync configurationDCDRFileSync

Step 1: ETCD and Patroni Installation​

  1. Download MotadataDCDRPatroniSetup, move it to the DC machine, assign execute permissions, and run it:

    chmod 777 MotadataDCDRPatroniSetup
    ./MotadataDCDRPatroniSetup

  2. When prompted to install and configure ETCD, enter yes.

  1. The ETCD installation starts. When prompted to install and configure Patroni, enter yes.

  1. Once the packages are installed, when prompted to configure this node as Node 1 or Node 2, enter 1 (this is the DC site).

  1. When prompted for the Node 2 IP address, enter the DR site IP address (for example, 172.20.21.19).

  1. When prompted for the Patroni port, press Enter to accept the default. When prompted for the ETCD IP address, enter the current machine IP (for example, 172.20.21.23).

Patroni is now installed and configured on the DC site.

Stop Patroni Before Proceeding

Before moving to Step 2, verify that the Patroni service is inactive. If it is active, stop it:

systemctl status patroni
systemctl stop patroni

Step 2: Application Configuration​

  1. Move to /opt/HA and execute MotadataPatroniDCDRConfig:

    cd /opt/HA
    ./MotadataPatroniDCDRConfig

  2. When prompted for the IP address of the database node, enter the current machine IP address.

App and DB on the Same Machine

In this architecture, the application and database run on the same machine. Enter the current machine IP: 172.20.21.23.

  1. When prompted whether this machine is DC or DR, enter dc.

The application configuration on the DC site is now complete.

Start ETCD Before Proceeding

Before moving to Step 3, ensure the motadata_etcd service is active and running:

systemctl status motadata_etcd
systemctl start motadata_etcd

Step 3: Patroni Database Configuration​

  1. Move to /opt/HA and execute MotadataPatroniDBConfig:

    cd /opt/HA
    ./MotadataPatroniDBConfig

  2. When prompted for the Patroni port, press Enter to accept the default.

  1. The script shows the initial state of the cluster and asks whether to reload the members. Enter y.

  1. When asked when the restart should take place, press Enter for now.

  1. When asked to confirm restarting members, enter Y.

  1. When asked whether to restart if the PostgreSQL version is less than the provided one, press Enter.

The Patroni setup on the DC site is now complete.

Step 4: File Sync Configuration​

  1. Move to /opt/HA and execute DCDRFileSync:

    cd /opt/HA
    ./DCDRFileSync

  2. When prompted for the current machine IP address, enter the DC machine IP (for example, 172.20.21.23).

  1. When prompted for the server 1 IP address, enter the DR site IP address (for example, 172.20.21.19).

  1. When prompted for the username, enter the Linux username available on both DC and DR (for example, flotomate).

  1. When prompted for the password, enter the password for that user.
Password Not Visible

The password is not displayed in the terminal for security.

  1. When prompted for the SSH port, enter 22 for the default port, or enter the port in use.

  1. When prompted for the DB node IP address, enter the current machine IP (app and DB are on the same machine: 172.20.21.23).

  1. When prompted whether any plugins are installed, enter yes if ServiceOps has a plugin integration; otherwise enter no.

The DC side file sync configuration is now complete. The full DC site configuration is complete.

DR Site Configuration​

Complete DC Setup First

Before starting the DR configuration, ensure all four DC site steps are fully complete and that Patroni on the DC site is running as the Leader.

Server Information​

SiteRoleIP Address
DCPatroni Node 1 (Leader)172.20.21.23
DRPatroni Node 2 (Replica)172.20.21.19

Key Differences from DC Configuration​

#Difference
1Configure this node as Node 2 (DC was Node 1).
2Patroni on DR points to the ETCD running on the DC site, so the DR database joins the same cluster as a replica.
3Enter dr when MotadataPatroniDCDRConfig asks whether the machine is DC or DR.
4MotadataPatroniDBConfig must also run on DR - this is what creates the replica.
5There is no HAProxy and no automatic failover. The DR node always remains a replica until a manual switchover is performed.

Pre-checks​

Before starting, verify the following:

  • The same Linux user (for example, flotomate) exists on both DC and DR, with the same password.
  • Ports 22, 5432, and 8008 are open between DC and DR.
  • Ports 2379, 2380, 8008, and 5432 are open on each node.
  • Backups and snapshots of the DR machine have been taken.
  • From the DR machine, verify connectivity to the DC machine on all required ports using nc -zv. All checks should show succeeded.

Step 1: ETCD and Patroni Installation​

  1. On the DR machine (172.20.21.19), download the same MotadataDCDRPatroniSetup file from the Download Links page, provide 777 permissions, and execute it:

    chmod 777 MotadataDCDRPatroniSetup
    ./MotadataDCDRPatroniSetup

  2. When prompted to install and configure ETCD, enter yes.

ETCD on DR

This ETCD instance is installed locally on the DR machine and is kept ready for switchover. It is not joined with the DC ETCD cluster. Patroni on DR will use the DC ETCD, which is specified in a later prompt.

  1. The ETCD installation starts. When prompted to install and configure Patroni, enter yes.

  1. Once the packages are installed, when prompted to configure this node as Node 1 or Node 2, enter 2 (this is the DR site).

  1. When prompted for the Node 1 IP address, enter the DC site IP address (for example, 172.20.21.23).

  1. When prompted for the PostgreSQL port, press Enter to accept the default (5432). When prompted for the ETCD IP address, enter the DC site IP address.
ETCD IP Address for DR

For DR, the ETCD IP address is the DC site ETCD, not the current machine. Enter the DC IP: 172.20.21.23.

Patroni is now installed and configured on the DR site. The setup summary shows Node 1 as DC, Node 2 as DR, and ETCD as DC.

Stop Patroni Before Proceeding

Before moving to Step 2, verify that the Patroni service is inactive. If it is active, stop it:

systemctl stop patroni

Step 2: Application Configuration​

  1. Move to /opt/HA on the DR machine and execute MotadataPatroniDCDRConfig:

    cd /opt/HA
    ./MotadataPatroniDCDRConfig

  2. When prompted for the database node IP, enter the current machine (DR) IP address.

App and DB on the Same Machine

In this architecture, the application and database run on the same machine. Enter the current machine IP: 172.20.21.19.

  1. When prompted whether this machine is DC or DR, enter dr.

The application configuration on the DR site is now complete. The PostgreSQL service is disabled here because Patroni manages PostgreSQL directly.

Start ETCD Before Proceeding

Before moving to Step 3, ensure the motadata_etcd service is active and running:

systemctl status motadata_etcd
systemctl start motadata_etcd

Step 3: Patroni Database Configuration (Replica Creation)​

Prerequisites for Replica Creation

Before executing, ensure:

  • The motadata_etcd service is active on the DR machine.
  • Patroni on the DC site is running as Leader.
How the Replica Is Created

As on the DC site, MotadataPatroniDBConfig must be executed on the DR site as well. This is what creates the replica on the DR server, by joining it to the cluster of the DC site.

  1. Move to /opt/HA and execute MotadataPatroniDBConfig:

    cd /opt/HA
    ./MotadataPatroniDBConfig

  2. When prompted for the Patroni port, press Enter to accept the default.

  1. The script shows the current state of the cluster and asks whether to reload the members. Enter y. When asked when the restart should take place, press Enter for now.

  1. When asked to confirm restarting members, enter y. When asked whether to restart if the PostgreSQL version is less than provided, press Enter.

The Patroni setup on the DR site is complete and replica creation starts.

Replica Creation Time

The DR node copies data from the DC leader. Depending on the database size, this can take some time. During this period, the node state may show as creating replica or starting.

  1. Ensure the patroni service is active. If it is inactive, start it:

    systemctl start patroni

Replication Verification​

After replica creation is complete, verify replication using the following check:

Cluster status - Run on the DR site. The DR node should show as Replica with state streaming and Lag in MB = 0:

patronictl -c /etc/patroni/config.yml list

The Patroni setup on the DR site is now complete.

Step 4: File Sync Configuration​

  1. Move to /opt/HA and execute DCDRFileSync:

    cd /opt/HA
    ./DCDRFileSync

  2. When prompted for the current machine IP, enter the DR machine IP (for example, 172.20.21.19).

  1. When prompted for the server 1 IP, enter the DC site IP address (172.20.21.23).

  1. When prompted for the username, enter the same Linux username used on the DC site (for example, flotomate).

  1. When prompted for the password, enter the password for that user.
Password Not Visible

The password is not displayed in the terminal for security.

  1. When prompted for the SSH port, enter 22 for the default port, or enter the port in use.

  2. When prompted for the DB node IP, enter the current machine IP (app and DB are on the same machine: 172.20.21.19).

  1. When prompted whether any plugins are installed, enter yes if ServiceOps has a plugin integration; otherwise enter no.

The DR side file sync configuration and the full DR site configuration are now complete.

Important Notes for the DR Site​

Operational Constraints

Keep the following in mind when operating the DR site:

  • No HAProxy: There is no HAProxy in this setup, as no routing is needed.
  • No automatic failover: The DR node is always a replica. Any role change must be performed manually using the DCDR_SwitchOver procedure. Refer to the Disaster Recovery Operations Guide.
  • PostgreSQL must stay inactive: The PostgreSQL service is controlled by Patroni on both sites. Stop it if found active: systemctl stop postgresql
  • Application services after switchover: Application services are not managed by Patroni and must be started manually on the DR site after a switchover.

Next Steps​

After completing the DC-DR setup, you may need to set up additional components depending on your requirements.