Skip to content

Commit e8cbbf1

Browse files
[SDP] PipelinesHandler + DatasetManager
1 parent b8a2c89 commit e8cbbf1

File tree

7 files changed

+4
-2
lines changed

7 files changed

+4
-2
lines changed

docs/declarative-pipelines/DatasetManager.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# DatasetManager
22

3-
`DatasetManager` is a global manager to [materialize datasets](#materializeDatasets) (tables and persistent views).
3+
`DatasetManager` is a global manager to [materialize datasets](#materializeDatasets) (tables and persistent views) right after a [pipeline update](PipelineExecution.md#startPipeline).
4+
5+
![DatasetManager](./images/DatasetManager.png)
46

57
!!! note ""
68
**Materialization** is a process of publishing tables and persistent views to session [TableCatalog](../connector/catalog/TableCatalog.md) and [SessionCatalog](../SessionCatalog.md), for tables and persistent views, respectively.

docs/declarative-pipelines/PipelinesHandler.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ subtitle: Spark Connect Endpoint
88

99
`PipelinesHandler` acts as a bridge between Python execution environment of [Spark Declarative Pipelines](index.md) and [Spark Connect Server]({{ book.spark_connect }}/server/) (where pipeline execution happens).
1010

11-
![PipelinesHandler](./images/spark-sql-sdp-PipelinesHandler.png)
11+
![PipelinesHandler](./images/PipelinesHandler.png)
1212

1313
## Handle Pipelines Command { #handlePipelinesCommand }
1414

30 KB
Loading
18.6 KB
Loading
Binary file not shown.
235 KB
Binary file not shown.
19.3 KB
Binary file not shown.

0 commit comments

Comments
 (0)