How to backup a sharded cluster?

[FeryET]

I have a sharded cluster with replicas, that is replicated like this:

clk-01: shard_1, shard_2
clk-02: shard_2, shard_3
clk-03: shard_3, shard_1

My shards are being populated by consuming from kafka engine using MVs, my tables are ReplicatedMergeTree, and I have a 3 node clickhouse-keeper setup that manages the metadata.

At first, I ran clickhouse-backup create <backup_name> and the tool created a backup for me, upon restoring the backup on a node, an error like this happened:

can't attach data parts for table 'shard_3.table_1d': shard_3.table_1d can't restore cause system.replicas entries already exists and replication in progress from another replica, log_pointer=53498, log_max_index=53497, absolute_delay=0, queue_size=0

I then decided to only create backups from the main shard of each node:

clk-01: shard_1
clk-02: shard_2
clk-03: shard_3

The same issue arises, the replicated tables are somehow being replicated upon creation on click house and I can't restore the data in them.

Based on the example of sharded cluster, I think I should do these steps sequentially:

1. Detach kafka engines so I no longer consume new messages.
2. Drop sharded tables from all replicas.
3. Restore sharded tables on their main replicas.
4. Restore sharded tables schemas on other replicas.

I wanted to make sure that I have identified the problem correctly, so I need your advice on this issue.

[Slach]

look https://github.com/Altinity/clickhouse-backup/blob/master/Examples.md#how-to-back-up--restore-a-sharded-cluster

[FeryET]

I've read that and tried to implement that before posting here. The problem is that the cluster is somehow replicating the data while my restore is underway. I've put the error there.

The process is as follows currently:

1. I create a backup of the shard database on my first replica.
2. I detach the shard database and its replcias.
3. I restore the backup using --schema on all replicas.
4. I restore the backup on my first replica.

This is how I have defined a table on my sharded cluster:

CREATE TABLE shard_1.table_1d
(
    `time` DateTime,
    `uuid` String,
    `domain` String,
    `cache_hit` UInt8,
    `total_sent` UInt64,
    `total_received` UInt64,
    `total_requests` UInt64,
    `total_upstream` Float64
)
ENGINE = ReplicatedSummingMergeTree('/clickhouse/tables/shard_1/table_1d', 'replica-1', (total_sent, total_received, total_requests, total_upstream))
PARTITION BY toStartOfHour(time)
ORDER BY (time, uuid, domain, cache_hit)
TTL time + toIntervalWeek(1)
SETTINGS index_granularity = 8192

[Slach]

I create a backup of the shard database on my first replica.

if you have multi shard you need to create backup on first replica in each shard

I detach the shard database and its replcias.

This is not required, and not described in link above

= ReplicatedSummingMergeTree('/clickhouse/tables/shard_1/table_1d

on the second shard it shall be /clickhouse/tables/shard_2/table_1d
if you restore backup from shard_1 into shard_2 then you will replicated data between shard

ReplicatedTable know nothing about shards, this is just watch zookeeper path read about zoo_path in documentation
https://clickhouse.com/docs/engines/table-engines/mergetree-family/replication#creating-replicated-tables

use {shard} macro in table in definition