1
1
(integrations-mongo-cdc)=
2
2
# MongoDB CDC (Preview)
3
3
4
- CrateDB Cloud enables continuous data ingestion from MongoDB using Change Data
4
+ CrateDB Cloud enables continuous data ingestion from MongoDB using Change Data
5
5
Capture (CDC), providing seamless, real-time synchronization of your data.
6
6
7
7
:::{caution}
@@ -11,31 +11,31 @@ For more information, please [contact us](https://cratedb.com/contact).
11
11
12
12
## Key Concepts
13
13
14
- The MongoDB CDC integration in CrateDB Cloud allows you to keep your data
15
- synchronized between your MongoDB Atlas cluster and your CrateDB Cloud cluster
16
- in real-time.
14
+ The MongoDB CDC integration in CrateDB Cloud allows you to keep your data
15
+ synchronized between your MongoDB Atlas cluster and your CrateDB Cloud cluster
16
+ in real-time.
17
17
18
18
### How It Works
19
19
20
20
The integration functions in two main stages:
21
21
22
- 1 . ** Initial Sync:**
23
- The integration performs a complete scan of your MongoDB collections,
22
+ 1 . ** Initial Sync:**
23
+ The integration performs a complete scan of your MongoDB collections,
24
24
importing all existing data into your CrateDB Cloud cluster.
25
25
26
- 2 . ** Continuous Sync:**
27
- The integration uses MongoDB Change Streams to monitor changes in your
28
- MongoDB collections and syncs these updates to your CrateDB Cloud cluster
26
+ 2 . ** Continuous Sync:**
27
+ The integration uses MongoDB Change Streams to monitor changes in your
28
+ MongoDB collections and syncs these updates to your CrateDB Cloud cluster
29
29
in real-time, ensuring that your data remains current.
30
30
31
31
### Data Consistency and Mode
32
32
33
- For continuous sync, CrateDB Cloud uses MongoDB's ** full document mode** to
34
- ensure data consistency. This mode guarantees that MongoDB returns the latest
35
- majority-committed version of the updated document.
33
+ For continuous sync, CrateDB Cloud uses MongoDB's ** full document mode** to
34
+ ensure data consistency. This mode guarantees that MongoDB returns the latest
35
+ majority-committed version of the updated document.
36
36
37
- While receiving partial deltas is more efficient, full document mode provides
38
- robust functionality. Support for partial deltas may be added in the future to
37
+ While receiving partial deltas is more efficient, full document mode provides
38
+ robust functionality. Support for partial deltas may be added in the future to
39
39
enhance performance and flexibility.
40
40
41
41
---
@@ -55,81 +55,81 @@ Supported authentication methods:
55
55
The following steps should be performed in the MongoDB Atlas UI.
56
56
57
57
#### Step 1: Create a Custom Role
58
- 1 . ** Navigate to Database Access**
59
- Go to ** Database Access** in the MongoDB Atlas UI for the cluster you want to
58
+ 1 . ** Navigate to Database Access**
59
+ Go to ** Database Access** in the MongoDB Atlas UI for the cluster you want to
60
60
connect to CrateDB Cloud.
61
61
62
- 2 . ** Add a Custom Role**
62
+ 2 . ** Add a Custom Role**
63
63
Under ** Custom Roles** , click ** Add New Custom Role** .
64
64
65
- 3 . ** Set Up Read-Only Access**
65
+ 3 . ** Set Up Read-Only Access**
66
66
Assign the following actions or roles to the custom role:
67
67
- ` find `
68
68
- ` changeStream `
69
69
- ` collStats `
70
70
71
- Specify the databases and collections for these actions. You can update
71
+ Specify the databases and collections for these actions. You can update
72
72
access permissions in the MongoDB Atlas UI later if needed.
73
73
74
74
75
75
#### Step 2: Create a User
76
76
77
- Depending on whether you plan to use SCRAM or X.509 authentication, create a
77
+ Depending on whether you plan to use SCRAM or X.509 authentication, create a
78
78
database user with one of the following methods:
79
79
80
80
:::{tab} SCRAM Auhentication
81
81
82
- 1 . ** Navigate to Database Access**
83
- In the MongoDB Atlas UI, go to ** Database Access** and click ** Add New
82
+ 1 . ** Navigate to Database Access**
83
+ In the MongoDB Atlas UI, go to ** Database Access** and click ** Add New
84
84
Database User** .
85
85
86
- 2 . ** Set Authentication Method**
87
- Choose ** Password** as the authentication method and enter a username and
86
+ 2 . ** Set Authentication Method**
87
+ Choose ** Password** as the authentication method and enter a username and
88
88
password for the database user.
89
89
90
- 3 . ** Assign the Role**
90
+ 3 . ** Assign the Role**
91
91
Under ** Database User Privileges** , select the custom role created in Step 1.
92
92
93
- 4 . ** Copy User Credentials**
94
- Click ** Add User** , and make sure to record the username and password. These
93
+ 4 . ** Copy User Credentials**
94
+ Click ** Add User** , and make sure to record the username and password. These
95
95
credentials will be used later in the CrateDB Cloud Console.
96
96
:::
97
97
98
98
:::{tab} x.509 Authentication
99
99
100
- 1 . ** Navigate to Database Access**
101
- In the MongoDB Atlas UI, go to ** Database Access** and click ** Add New
100
+ 1 . ** Navigate to Database Access**
101
+ In the MongoDB Atlas UI, go to ** Database Access** and click ** Add New
102
102
Database User** .
103
103
104
- 2 . ** Set Authentication Method**
104
+ 2 . ** Set Authentication Method**
105
105
Choose ** Certificate** as the authentication method.
106
106
107
- 3 . ** Assign the Role**
107
+ 3 . ** Assign the Role**
108
108
Under ** Database User Privileges** , select the custom role created in Step 1.
109
109
110
- 4 . ** Save the Certificate**
111
- Click ** Add User** , and store the certificate securely. This will be required
110
+ 4 . ** Save the Certificate**
111
+ Click ** Add User** , and store the certificate securely. This will be required
112
112
later in the CrateDB Cloud Console.
113
113
:::
114
114
115
115
116
116
#### Step 3: Configure IP Access
117
117
118
- To allow CrateDB Cloud to access your MongoDB Atlas cluster, you must add the
118
+ To allow CrateDB Cloud to access your MongoDB Atlas cluster, you must add the
119
119
CrateDB Cloud IP addresses to the IP Access List in MongoDB Atlas.
120
120
121
- 1 . ** Navigate to Network Access**
121
+ 1 . ** Navigate to Network Access**
122
122
In the MongoDB Atlas UI, go to ** Network Access** from the left navigation.
123
123
124
- 2 . ** Add IP Address**
125
- Click ** Add IP Address** and choose an IP address or range to allow access.
126
- For testing purposes, you can select ** Allow Access from Anywhere** , but for
124
+ 2 . ** Add IP Address**
125
+ Click ** Add IP Address** and choose an IP address or range to allow access.
126
+ For testing purposes, you can select ** Allow Access from Anywhere** , but for
127
127
production, it is recommended to specify only the required IPs.
128
128
129
129
:::{note}
130
- The specific IP addresses depend on the region of your CrateDB Cloud cluster.
131
- These IP addresses can also be found in the ** Connection Details** section of the
132
- CrateDB Cloud Console, just before you click ** Test Connection** during the
130
+ The specific IP addresses depend on the region of your CrateDB Cloud cluster.
131
+ These IP addresses can also be found in the ** Connection Details** section of the
132
+ CrateDB Cloud Console, just before you click ** Test Connection** during the
133
133
setup process.
134
134
135
135
** Outbound IP Addresses** :
@@ -156,17 +156,17 @@ out to our support team.
156
156
You'll need to provide the connection string for your MongoDB Atlas cluster so
157
157
that CrateDB Cloud can connect to it.
158
158
159
- 1 . ** Navigate to Your Cluster**
159
+ 1 . ** Navigate to Your Cluster**
160
160
In the MongoDB Atlas UI, navigate to the cluster you want to connect to CrateDB Cloud.
161
161
162
- 2 . ** Click "Connect"**
162
+ 2 . ** Click "Connect"**
163
163
From the cluster view, click on ** Connect** .
164
164
165
- 3 . ** Select "Connect Your Application"**
165
+ 3 . ** Select "Connect Your Application"**
166
166
Choose ** Connect your application** as the connection method.
167
167
168
- 4 . ** Copy the Connection String**
169
- Copy the connection string provided in the MongoDB Atlas UI. It will look like this:
168
+ 4 . ** Copy the Connection String**
169
+ Copy the connection string provided in the MongoDB Atlas UI. It will look like this:
170
170
171
171
```
172
172
mongodb+srv://:@/?retryWrites=true&w=majority
@@ -193,55 +193,55 @@ in CrateDB Cloud.
193
193
194
194
Follow these steps in the CrateDB Cloud Console to set up the MongoDB CDC integration:
195
195
196
- #### Step 1: Create an Integration
197
- 1 . Navigate to the ** Import** section in the CrateDB Cloud Console.
196
+ #### Step 1: Create an Integration
197
+ 1 . Navigate to the ** Import** section in the CrateDB Cloud Console.
198
198
2 . Click ** Create Integration** and select ** MongoDB** as the source type.
199
199
200
- #### Step 2: Configure Connection
200
+ #### Step 2: Configure Connection
201
201
1 . Choose ** Create New Connection** or select an existing one.
202
202
2 . Fill in the following details:
203
203
:::{tab} SCRAM Auhentication
204
- - ** Connection Name** : Provide a unique name for the connection.
205
- - ** Connection String** : Paste the connection string from MongoDB Atlas.
206
- - ** Username** : Enter the database username (required for SCRAM).
204
+ - ** Connection Name** : Provide a unique name for the connection.
205
+ - ** Connection String** : Paste the connection string from MongoDB Atlas.
206
+ - ** Username** : Enter the database username (required for SCRAM).
207
207
- ** Password** : Enter the database password (required for SCRAM).
208
208
- ** Default Database** : Specify the default database to use for this connection.
209
209
:::
210
210
:::{tab} X.509 Auhentication
211
- - ** Connection Name** : Provide a unique name for the connection.
212
- - ** Connection String** : Paste the connection string from MongoDB Atlas.
211
+ - ** Connection Name** : Provide a unique name for the connection.
212
+ - ** Connection String** : Paste the connection string from MongoDB Atlas.
213
213
- ** Certificate** : Upload the X.509 certificate file.
214
214
- ** Default Database** : Specify the default database to use for this connection.
215
215
:::
216
216
217
- #### Step 3: Test the Connection
217
+ #### Step 3: Test the Connection
218
218
Click ** Test Connection** to verify CrateDB Cloud can connect to your MongoDB
219
219
Atlas cluster. Resolve any issues if the test fails.
220
220
221
- #### Step 4: Select Collection
221
+ #### Step 4: Select Collection
222
222
Enter the database and collection name from your MongoDB Atlas cluster, that you
223
223
want to sync with CrateDB Cloud.
224
224
225
- #### Step 5: Select Target Table
226
- 1 . Specify the target table in your CrateDB Cloud cluster where the data will be synced.
227
- 2 . MongoDB records will be inserted into an object column called ` document ` .
228
- 3 . Select the object type for the column:
229
- - ** ` dynamic ` ** : Allows indexing and columnar storage for faster querying.
230
- - ** ` ignored ` ** : Prevents type conflicts in CrateDB if your source data lacks a strict schema.
225
+ #### Step 5: Select Target Table
226
+ 1 . Specify the target table in your CrateDB Cloud cluster where the data will be synced.
227
+ 2 . MongoDB records will be inserted into an object column called ` document ` .
228
+ 3 . Select the object type for the column:
229
+ - ** ` dynamic ` ** : Allows indexing and columnar storage for faster querying.
230
+ - ** ` ignored ` ** : Prevents type conflicts in CrateDB if your source data lacks a strict schema.
231
231
232
232
:::{note}
233
- If your source data doesn't follow a strict schema, select ` ignored ` to avoid type conflicts.
234
- However, selecting ` dynamic ` provides faster query performance by utilizing indexes and columnar storage.
233
+ If your source data doesn't follow a strict schema, select ` ignored ` to avoid type conflicts.
234
+ However, selecting ` dynamic ` provides faster query performance by utilizing indexes and columnar storage.
235
235
:::
236
236
237
- #### Step 6: Configure Integration Settings
238
- 1 . Enter a name for the integration.
239
- 2 . Select the integration mode:
240
- - ** Full Load Only** : Imports the data once but doesn’t sync changes.
241
- - ** Full Load and CDC** : Imports the data and syncs changes in real-time.
237
+ #### Step 6: Configure Integration Settings
238
+ 1 . Enter a name for the integration.
239
+ 2 . Select the integration mode:
240
+ - ** Full Load Only** : Imports the data once but doesn’t sync changes.
241
+ - ** Full Load and CDC** : Imports the data and syncs changes in real-time.
242
242
- ** CDC Only** : Syncs only new changes in real-time without importing existing data.
243
243
244
- #### Step 7: Create the Integration
244
+ #### Step 7: Create the Integration
245
245
Click ** Create Integration** to finalize the setup. CrateDB Cloud will now sync
246
246
your MongoDB data based on the selected settings.
247
247
@@ -275,5 +275,5 @@ CDC integration:
275
275
value.
276
276
- ** Binary data types** other than UUIDs, which are converted to ` TEXT ` and
277
277
** vectors** , which are converted to ` ARRAY ` s of numbers.
278
- - The ` Decimal128 ` data type is not supported and is converted to a string, as
278
+ - The ` Decimal128 ` data type is not supported and is converted to a string, as
279
279
CrateDB does not support a decimal data type.
0 commit comments