DataSync: Enable Sibling BMC Synchronization with Stunnel and Rsyncd #21

RameshIyyar · 2025-03-17T16:19:57Z

No description provided.

RameshIyyar · 2025-03-18T04:45:34Z

config/certs/bmc0.cert.pem

@@ -0,0 +1,33 @@
+-----BEGIN CERTIFICATE-----


I'm considering a script that dynamically generates certificates and keys while syncing the CA to a sibling BMC temporarily.

This script could integrate with the BMC boot process using the multi-user target, avoiding regeneration if the certificate and key files already exist.

So that, we no need to push these files into the data sync repo since its temporary arrangement.

src/external_data_ifaces.cpp

src/manager.hpp

service_files/meson.build

config/rsync/bmc0_rsyncd.conf.in

config/meson.build

meson.build

scripts/update_sibling_bmc_ip.sh

spinler · 2025-03-18T14:36:08Z

service_files/SyncBMCData_stunnel.service

+After=SyncBMCData_rsync.service
+
+[Service]
+ExecStartPre=/usr/libexec/phosphor-data-sync/update_sibling_bmc_ip.sh


What if the sibling BMC isn't available at this point, but shows up later after this BMC has already became active? (Static IPs would make this easier.)

I considered starting these services (stunnel and possibly rsyncd) within the data sync application once the sibling BMC becomes available by monitoring the networkd interface. Let me think more on this, as I initially thought of it to handle sibling BMC reboot scenarios.

Static IPs would make this easier.

I agree, but it's still unclear how the network team is planning this. There’s a possibility that the IP could change during a CM operation if it was configured during manufacturing. It’s uncertain whether they would prefer to keep the same IP. However, we could use a hostname for the sibling BMC, allowing applications to communicate with it over the network. The network daemon could then update the sibling bmc hostname’s IP address based on the configured IP.

RameshIyyar

Please hold off on the review for now—I'm planning to make a few more changes. I just pushed to my fork to avoid losing my local work.

service_files/meson.build

config/rsync/bmc0_rsyncd.conf.in

config/meson.build

meson.build

scripts/update_sibling_bmc_ip.sh

RameshIyyar · 2025-03-19T05:05:57Z

service_files/SyncBMCData_stunnel.service

+After=SyncBMCData_rsync.service
+
+[Service]
+ExecStartPre=/usr/libexec/phosphor-data-sync/update_sibling_bmc_ip.sh


I considered starting these services (stunnel and possibly rsyncd) within the data sync application once the sibling BMC becomes available by monitoring the networkd interface. Let me think more on this, as I initially thought of it to handle sibling BMC reboot scenarios.

Static IPs would make this easier.

I agree, but it's still unclear how the network team is planning this. There’s a possibility that the IP could change during a CM operation if it was configured during manufacturing. It’s uncertain whether they would prefer to keep the same IP. However, we could use a hostname for the sibling BMC, allowing applications to communicate with it over the network. The network daemon could then update the sibling bmc hostname’s IP address based on the configured IP.

scripts/update_sibling_bmc_ip.sh

src/external_data_ifaces.cpp

src/manager.hpp

- In this commit, the data sync JSON configuration is moved into the config Meson build file to consolidate all configuration file build rules in the same Meson build file. Change-Id: I6025eb60bae4d658371059a8e184dc8db3a495f1 Signed-off-by: Ramesh Iyyar <[email protected]>

- In this commit, the data sync service file is moved into the service file Meson build file to consolidate all service file build rules in the same Meson build file. - Additionally, the Meson "fs" module is used to install the service file, as the configure_file function has been deprecated. Change-Id: Ia778a7a44f93112b4d7066e367fbf4ce828ca086 Signed-off-by: Ramesh Iyyar <[email protected]>

- Rsync is the core mechanism for data synchronization between BMCs. For more details, refer to: https://rsync.samba.org/. - Rsync supports both pull and push methods for data transfer between local and remote machines. - The data sync application uses the push method, ensuring that any data modified by the application is synchronized with the sibling BMC. - Rsync operates in two modes: local copy and remote copy. - Local copy is used for unit test cases. - Remote copy is used for data synchronization between BMCs. - This can be achieved in two ways: remote SSH shell or rsync daemon. - The data sync application uses the rsync daemon, enabling secure synchronization between sibling BMCs via stunnel, as rsync does not natively support mTLS. - Two rsyncd configuration files have been added for both BMCs: - BMC0 rsyncd listens on port 1703. - BMC1 rsyncd listens on port 1704. - These ports can be modified as needed in the future. - The rsyncd configuration files currently define modules for "var" and "etc" directories, with the flexibility to extend them as required. - For more details on rsyncd configuration, refer to: https://download.samba.org/pub/rsync/rsyncd.conf.5. - The rsync daemon is bind to localhost, preventing access from external machines. In future commits, the rsync daemon port will be used to securely route sync requests via stunnel. - A new systemd service file has been added for the rsync daemon to ensure it starts before the data sync application. - The appropriate rsyncd configuration file will be selected based on the local BMC's position. Tested: - Verified that the rsync daemon starts with the correct ports. - BMC0 ``` systemd: Started Rsync daemon for Redundant BMC Data Synchronization. rsyncd: rsyncd version 3.4.1 starting, listening on port 1703 ``` - BMC1 ``` systemd: Started Rsync daemon for Redundant BMC Data Synchronization. rsyncd: rsyncd version 3.4.1 starting, listening on port 1704 ``` Change-Id: I6caeff01ff2ca72c1cb9da5bcedc69fd8c13e59b Signed-off-by: Ramesh Iyyar <[email protected]>

- To ensure secure data synchronization, both BMCs must verify each other. This can be achieved using an mTLS handshake. However, since rsync does not natively support TLS or mTLS, stunnel will be used to establish the mTLS handshake. - For the mTLS handshake, both BMCs require certificate and key files, which have been added in this commit. - Note: Ideally, these certificates and keys should be retrieved from SPDM, but since SPDM support is currently unavailable, temporary certificate and key files are maintained in the data sync repo to enable the basic mTLS handshake for sync requests. Change-Id: Ia962cbf5b7f91a385f3ecc8eb3d71d3f39192662 Signed-off-by: Ramesh Iyyar <[email protected]>

- To ensure secure data synchronization, both BMCs must verify each other using an mTLS handshake. Since rsync does not natively support TLS or mTLS, stunnel is used to establish the mTLS handshake. - For more details on stunnel, refer to: https://www.stunnel.org/. - Two stunnel configuration files have been added for both BMCs: - Stunnel supports service configurations, allowing different services to route requests to remote and local machines. - Two services have been added for both local and sibling BMCs to configure packet forwarding between them, enabling bidirectional synchronization. - Local BMC service (acts as the stunnel server for rsync) - BMC0: Listens on port 3071 to accept sync requests from the sibling BMC and forwards them to the local rsync daemon on port 1703. - BMC1: Listens on port 4071 to accept sync requests from the sibling BMC and forwards them to the local rsync daemon on port 1704. - Sibling BMC service (acts as the stunnel client for rsync) - BMC0: Listens on port 1704 (sibling BMC's rsync daemon port), sends requests over localhost, and sends them to the sibling BMC's stunnel server on port 4071. - BMC1: Listens on port 1703 (sibling BMC's rsync daemon port), sends requests over localhost, and sends them to the sibling BMC's stunnel server on port 3071. - These ports can be modified as needed in the future. - The respective BMC certificate and key, along with the CA certificate, are configured in both local and sibling BMC stunnel services. - The stunnel services use verify level 2, enabling mTLS handshake. - Once a secure tunnel is established after the mTLS handshake, the connection remains secure until it is closed or the service is restarted. - A new systemd service file has been added for the stunnel daemon to ensure it starts after rsync and before the data sync application. - The appropriate stunnel configuration file will be selected based on the local BMC's position. Tested: - Verified that configured stunnel services starts with the correct ports. - BMC0 ``` systemd: Started Rsync daemon for Redundant BMC Data Synchronization. systemd: Starting Stunnel for Redundant BMC Data Synchronization... rsyncd: rsyncd version 3.4.1 starting, listening on port 1703 systemd: Started Stunnel for Redundant BMC Data Synchronization. systemd: Starting Redundant BMC Data Synchronization... systemd: Started Redundant BMC Data Synchronization. ... stunnel: LOG5[ui]: Reading configuration from file /etc/phosphor-data-sync/stunnel/bmc0_stunnel.conf ... stunnel: LOG5[ui]: Configuration successful stunnel: LOG5[ui]: Binding service [local_bmc] to :::3071: Address already in use (98) ``` - BMC1 ``` systemd: Started Rsync daemon for Redundant BMC Data Synchronization. systemd: Starting Stunnel for Redundant BMC Data Synchronization... rsyncd: rsyncd version 3.4.1 starting, listening on port 1704 systemd: Started Stunnel for Redundant BMC Data Synchronization. systemd: Starting Redundant BMC Data Synchronization... systemd: Started Redundant BMC Data Synchronization. ... stunnel: LOG5[ui]: Reading configuration from file /etc/phosphor-data-sync/stunnel/bmc1_stunnel.conf ... stunnel: LOG5[ui]: Configuration successful stunnel: LOG5[ui]: Binding service [local_bmc] to :::4071: Address already in use (98) ``` Change-Id: I09d3e746bc200b736fdf56f1a10dba5472910006 Signed-off-by: Ramesh Iyyar <[email protected]>

- The sibling BMC IP address needs to be configured in the stunnel configuration file. However, it cannot be statically assigned because the BMC IP address is determined dynamically using NDP (Neighbor Discovery Protocol) via the xyz.openbmc_project.Network.Neighbor DBus interface, managed by the network daemon. - To handle this, a script has been added to dynamically update the sibling BMC IP address when the stunnel service starts. - This script runs before the stunnel service starts to ensure the correct IP is configured. - Currently, network daemon support is not yet available, so for now, the sibling BMC IP address is configured based on its position in the simulation environment. Tested: - Verified that the sibling BMC IP address is changed dynamically based on its position. - After the stunnel configuration installed during the build. - bmc0_stunnel.conf ``` [sibling_bmc] client = yes accept = 127.0.0.1:1704 connect = <sibling_bmc_ip>:4071 cert = /etc/phosphor-data-sync/certs/bmc0.cert.pem key = /etc/phosphor-data-sync/certs/bmc0.key.pem CAfile = /etc/phosphor-data-sync/certs/cacert.pem verify = 2 ``` - bmc1_stunnel.conf ``` [sibling_bmc] client = yes accept = 127.0.0.1:1703 connect = <sibling_bmc_ip>:3071 cert = /etc/phosphor-data-sync/certs/bmc1.cert.pem key = /etc/phosphor-data-sync/certs/bmc1.key.pem CAfile = /etc/phosphor-data-sync/certs/cacert.pem verify = 2 ``` - After the stunnel service starts - In BMC0, bmc0_stunnel.conf ``` [sibling_bmc] client = yes accept = 127.0.0.1:1704 connect = 10.2.2.100:4071 cert = /etc/phosphor-data-sync/certs/bmc0.cert.pem key = /etc/phosphor-data-sync/certs/bmc0.key.pem CAfile = /etc/phosphor-data-sync/certs/cacert.pem verify = 2 ``` - In BMC0, bmc1_stunnel.conf ``` [sibling_bmc] client = yes accept = 127.0.0.1:1703 connect = <sibling_bmc_ip>:3071 cert = /etc/phosphor-data-sync/certs/bmc1.cert.pem key = /etc/phosphor-data-sync/certs/bmc1.key.pem CAfile = /etc/phosphor-data-sync/certs/cacert.pem verify = 2 ``` - In BMC1, bmc0_stunnel.conf ``` [sibling_bmc] client = yes accept = 127.0.0.1:1704 connect = <sibling_bmc_ip>:4071 cert = /etc/phosphor-data-sync/certs/bmc0.cert.pem key = /etc/phosphor-data-sync/certs/bmc0.key.pem CAfile = /etc/phosphor-data-sync/certs/cacert.pem verify = 2 ``` - In BMC1, bmc1_stunnel.conf ``` [sibling_bmc] client = yes accept = 127.0.0.1:1703 connect = 10.0.2.100:3071 cert = /etc/phosphor-data-sync/certs/bmc1.cert.pem key = /etc/phosphor-data-sync/certs/bmc1.key.pem CAfile = /etc/phosphor-data-sync/certs/cacert.pem verify = 2 ``` Change-Id: Id30834a38744cf81d6fcc41ee243f0e347dedeb8 Signed-off-by: Ramesh Iyyar <[email protected]>

- Currently, an API exists to retrieve the sibling BMC IP address, but it is no longer needed. Since the stunnel configuration already includes the sibling BMC IP, the sync request only needs to use the sibling BMC rsync daemon port with localhost, as stunnel handles forwarding to the sibling BMC. - Therefore, the API is being modified to get the sibling BMC position. It could be used to get the sibling BMC rsyncd port. Change-Id: Iaae597fa4f041aaad91a2d9e518839fb4e5d8a1d Signed-off-by: Ramesh Iyyar <[email protected]>

- To ensure secure data synchronization, both BMCs authenticate each other using an mTLS handshake. Since rsync does not natively support TLS or mTLS, stunnel is used to establish the handshake. - The data sync application now relies on stunnel for mTLS authentication, eliminating the need for a sibling BMC username and password so, the RbmcCredentials API has been removed. Change-Id: I225c70d614166f533570a602642131ffd0be1e8f Signed-off-by: Ramesh Iyyar <[email protected]>

- The data sync application now utilizes stunnel for secure data synchronization. Sync requests can be sent to the sibling BMC rsync daemon port via localhost, stunnel handling the forwarding to the sibling BMC. Tested: - Verified that the data is synced into sibling BMC after the mTLS handshake. - Data sync configuration ``` { "Files": [ { "Path": "/var/lib/pds/test", "Description": "Test Sibling BMC Synchronization", "SyncDirection": "Active2Passive", "SyncType": "Immediate" } ] } ``` - BMCs Role: BMC0 (Active) and BMC1 (Passive) - FullSync - BMC0 ``` systemd: Starting Redundant BMC Data Synchronization... phosphor-rbmc-data-sync-mgr: Full Sync started stunnel: LOG5[3]: Service [sibling_bmc] accepted connection from 127.0.0.1:50390 stunnel: LOG5[3]: s_connect: connected 10.2.2.100:4071 stunnel: LOG5[3]: Service [sibling_bmc] connected remote server from 10.0.2.100:53920 systemd: Started Redundant BMC Data Synchronization. stunnel: LOG5[3]: Certificate accepted at depth=0: C=IN, ST=KA, O=IBM, OU=ISDL, CN=bmc1, [email protected] <=== BMC1 certificate verified stunnel: LOG3[3]: OCSP: No OCSP stapling response received phosphor-rbmc-data-sync-mgr: Full Sync completed successfully phosphor-rbmc-data-sync-mgr: Elapsed time for full sync: [0] seconds stunnel: LOG5[3]: Connection closed: 193 byte(s) sent to TLS, 109 byte(s) sent to socket ``` - BMC1 ``` systemd: Starting Redundant BMC Data Synchronization... phosphor-rbmc-data-sync-mgr: Full Sync started phosphor-rbmc-data-sync-mgr: Full Sync completed successfully phosphor-rbmc-data-sync-mgr: Elapsed time for full sync: [0] seconds systemd: Started Redundant BMC Data Synchronization. stunnel: LOG5[3]: Service [local_bmc] accepted connection from 10.0.2.100:53920 stunnel: LOG5[3]: Certificate accepted at depth=0: C=IN, ST=KA, O=IBM, OU=ISDL, CN=bmc0, [email protected] <=== BMC0 certificate verified stunnel: LOG5[3]: s_connect: connected 127.0.0.1:1704 stunnel: LOG5[3]: Service [local_bmc] connected remote server from 127.0.0.1:45472 rsyncd: connect from localhost (127.0.0.1) rsyncd: rsync allowed access on module var from localhost (127.0.0.1) rsyncd: rsync to var/lib/pds/test from localhost (127.0.0.1) rsyncd: receiving file list rsyncd: sent 40 bytes received 110 bytes total size 0 <=== Data received stunnel: LOG5[3]: Connection closed: 109 byte(s) sent to TLS, 193 byte(s) sent to socket ``` - After FullSync ``` BMC0: cat /var/lib/pds/test Data from Active BMC BMC1: cat /var/lib/pds/test Data from Active BMC ``` - Immediate Sync - Before Immediate Sync ``` BMC0: cat /var/lib/pds/test Data from Active BMC BMC1: cat /var/lib/pds/test Data from Active BMC ``` - BMC0 Trace ``` stunnel: LOG5[4]: Service [sibling_bmc] accepted connection from 127.0.0.1:52292 stunnel: LOG5[4]: s_connect: connected 10.2.2.100:4071 stunnel: LOG5[4]: Service [sibling_bmc] connected remote server from 10.1.2.100:42794 stunnel: LOG5[4]: Connection closed: 201 byte(s) sent to TLS, 109 byte(s) sent to socket ``` - BMC1 Trace ``` stunnel: LOG5[4]: Service [local_bmc] accepted connection from 10.1.2.100:42794 stunnel: LOG5[4]: s_connect: connected 127.0.0.1:1704 stunnel: LOG5[4]: Service [local_bmc] connected remote server from 127.0.0.1:60416 rsyncd: connect from localhost (127.0.0.1) rsyncd: rsync allowed access on module var from localhost (127.0.0.1) rsyncd: rsync to var/lib/pds/test from localhost (127.0.0.1) rsyncd: receiving file list rsyncd: sent 40 bytes received 118 bytes total size 5 <=== Data received stunnel: LOG5[4]: Connection closed: 109 byte(s) sent to TLS, 201 byte(s) sent to socket ``` - After Immediate Sync ``` BMC0: cat /var/lib/pds/test Data from Active BMC Another data from Active BMC BMC1: cat /var/lib/pds/test Data from Active BMC Another data from Active BMC ``` Change-Id: Ia668ee21e69a78f973aea86fc96302ca76f499ec Signed-off-by: Ramesh Iyyar <[email protected]>

RameshIyyar · 2025-04-11T14:35:51Z

Just fixed conflicts so that others could pull my PR if needed, meanwhile I will try to push further changes to start services based on the sibling status.

RameshIyyar · 2025-07-08T04:32:44Z

Recently, we decided to hard the code sibling BMC IP address based on its position so we could simplify a few things and I am working on it.

RameshIyyar requested review from aryakpadman, harsh-agarwal1, manishtwari and spinler and removed request for aryakpadman and harsh-agarwal1 March 17, 2025 16:20

RameshIyyar commented Mar 18, 2025

View reviewed changes

spinler reviewed Mar 18, 2025

View reviewed changes

RameshIyyar force-pushed the mtls branch from b52a30a to aa8773a Compare April 9, 2025 14:18

RameshIyyar commented Apr 9, 2025

View reviewed changes

RameshIyyar added 9 commits April 11, 2025 02:07

RameshIyyar force-pushed the mtls branch from aa8773a to d0dd9c7 Compare April 11, 2025 14:34

aryakpadman mentioned this pull request Jun 4, 2025

Enable the necessary rsync CLI options #26

Merged

DataSync: Enable Sibling BMC Synchronization with Stunnel and Rsyncd #21

Are you sure you want to change the base?

DataSync: Enable Sibling BMC Synchronization with Stunnel and Rsyncd #21

Uh oh!

Conversation

RameshIyyar commented Mar 17, 2025

Uh oh!

RameshIyyar Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spinler Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

RameshIyyar Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

RameshIyyar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RameshIyyar Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RameshIyyar commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RameshIyyar commented Jul 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

RameshIyyar commented Apr 11, 2025 •

edited

Loading