[SDP] UnresolvedFlow and Python/SQL APIs

Author: jaceklaskowski

docs/declarative-pipelines/GraphRegistrationContext.md

@@ -35,6 +35,32 @@ toDataflowGraph: DataflowGraph
 
 * `PipelinesHandler` is requested to [start a pipeline run](PipelinesHandler.md#startRun)
 
+### assertNoDuplicates { #assertNoDuplicates }
+
+```scala
+assertNoDuplicates(
+  qualifiedTables: Seq[Table],
+  validatedViews: Seq[View],
+  qualifiedFlows: Seq[UnresolvedFlow]): Unit
+```
+
+`assertNoDuplicates`...FIXME
+
+### assertFlowIdentifierIsUnique { #assertFlowIdentifierIsUnique }
+
+```scala
+assertFlowIdentifierIsUnique(
+  flow: UnresolvedFlow,
+  datasetType: DatasetType,
+  flows: Seq[UnresolvedFlow]): Unit
+```
+
+`assertFlowIdentifierIsUnique` throws an `AnalysisException` if the given [UnresolvedFlow](UnresolvedFlow.md)'s identifier is used by multiple flows (among the given `flows`):
+
+```text
+Flow [flow_name] was found in multiple datasets: [dataset_names]
+```
+
 ## Tables { #tables }
 
 `GraphRegistrationContext` creates an empty registry of [Table](Table.md)s when [created](#creating-instance).
@@ -78,9 +104,9 @@ registerFlow(
 `registerFlow` is used when:
 
 * `PipelinesHandler` is requested to [define a flow](PipelinesHandler.md#defineFlow)
-* `SqlGraphRegistrationContext` is requested to [process the following SQL commands](SqlGraphRegistrationContext.md#processSqlQuery):
-    * [CreateFlowCommand](../logical-operators/CreateFlowCommand.md)
-    * [CreateMaterializedViewAsSelect](../logical-operators/CreateMaterializedViewAsSelect.md)
-    * [CreateView](../logical-operators/CreateView.md)
-    * [CreateStreamingTableAsSelect](../logical-operators/CreateStreamingTableAsSelect.md)
-    * [CreateViewCommand](../logical-operators/CreateViewCommand.md)
+* `SqlGraphRegistrationContext` is requested to [handle the following logical commands](SqlGraphRegistrationContext.md#processSqlQuery):
+    * [CreateFlowCommand](SqlGraphRegistrationContext.md#CreateFlowCommand)
+    * [CreateMaterializedViewAsSelect](SqlGraphRegistrationContext.md#CreateMaterializedViewAsSelect)
+    * [CreateView](SqlGraphRegistrationContext.md#CreateView)
+    * [CreateStreamingTableAsSelect](SqlGraphRegistrationContext.md#CreateStreamingTableAsSelect)
+    * [CreateViewCommand](SqlGraphRegistrationContext.md#CreateViewCommand)

docs/declarative-pipelines/PipelinesHandler.md

@@ -45,11 +45,15 @@ handlePipelinesCommand(
 
 ### CREATE_DATAFLOW_GRAPH { #CREATE_DATAFLOW_GRAPH }
 
-`handlePipelinesCommand` [creates a dataflow graph](#createDataflowGraph) and sends the graph ID back.
+[handlePipelinesCommand](#handlePipelinesCommand) creates a [dataflow graph](#createDataflowGraph) and sends the graph ID back.
+
+### DROP_DATAFLOW_GRAPH { #DROP_DATAFLOW_GRAPH }
+
+[handlePipelinesCommand](#handlePipelinesCommand)...FIXME
 
 ### DEFINE_DATASET { #DEFINE_DATASET }
 
-`handlePipelinesCommand` prints out the following INFO message to the logs:
+[handlePipelinesCommand](#handlePipelinesCommand) prints out the following INFO message to the logs:
 
 ```text
 Define pipelines dataset cmd received: [cmd]
@@ -59,7 +63,7 @@ Define pipelines dataset cmd received: [cmd]
 
 ### <span id="DefineFlow"> DEFINE_FLOW { #DEFINE_FLOW }
 
-`handlePipelinesCommand` prints out the following INFO message to the logs:
+[handlePipelinesCommand](#handlePipelinesCommand) prints out the following INFO message to the logs:
 
 ```text
 Define pipelines flow cmd received: [cmd]
@@ -69,14 +73,18 @@ Define pipelines flow cmd received: [cmd]
 
 ### START_RUN { #START_RUN }
 
-`handlePipelinesCommand` prints out the following INFO message to the logs:
+[handlePipelinesCommand](#handlePipelinesCommand) prints out the following INFO message to the logs:
 
 ```text
 Start pipeline cmd received: [cmd]
 ```
 
 `handlePipelinesCommand` [starts a pipeline run](#startRun).
 
+### DEFINE_SQL_GRAPH_ELEMENTS { #DEFINE_SQL_GRAPH_ELEMENTS }
+
+[handlePipelinesCommand](#handlePipelinesCommand)...FIXME
+
 ## Start Pipeline Run { #startRun }
 
 ```scala
@@ -153,6 +161,9 @@ defineFlow(
   sparkSession: SparkSession): Unit
 ```
 
+??? note "DEFINE_FLOW Pipeline Command"
+    `defineFlow` is used to handle [DEFINE_FLOW](#DEFINE_FLOW).
+
 `defineFlow` looks up the [GraphRegistrationContext](DataflowGraphRegistry.md#getDataflowGraphOrThrow) for the given `flow` (or throws a `SparkException` if not found).
 
 !!! note "Implicit Flows"
@@ -164,3 +175,4 @@ defineFlow(
     `defineFlow` reports an `AnalysisException` if the given `flow` is not an implicit flow, but is defined with a multi-part identifier.
 
 In the end, `defineFlow` [registers a flow](GraphRegistrationContext.md#registerFlow).
+

docs/declarative-pipelines/SqlGraphRegistrationContext.md

@@ -37,7 +37,7 @@ processSqlFile(
 
 * `PipelinesHandler` is requested to [defineSqlGraphElements](PipelinesHandler.md#defineSqlGraphElements)
 
-### Process Single SQL Query { #processSqlQuery }
+### Process Single Logical Command { #processSqlQuery }
 
 ```scala
 processSqlQuery(

docs/declarative-pipelines/UnresolvedFlow.md

@@ -1,6 +1,17 @@
+---
+hide:
+    - toc
+---
+
 # UnresolvedFlow
 
-`UnresolvedFlow` is a [Flow](Flow.md).
+`UnresolvedFlow` is a [Flow](Flow.md) that represents flows in the following Python and SQL transformations in [Spark Declarative Pipelines](index.md):
+
+* [register_flow](GraphElementRegistry.md#register_flow) in PySpark's decorators
+* [CREATE FLOW ... AS INSERT INTO ... BY NAME](../logical-operators/CreateFlowCommand.md)
+* [CREATE MATERIALIZED VIEW](../logical-operators/CreateMaterializedViewAsSelect.md)
+* [CREATE STREAMING TABLE ... AS](../logical-operators/CreateStreamingTableAsSelect.md)
+* [CREATE VIEW](../logical-operators/CreateView.md) and the other variants of [CREATE VIEW](../logical-operators/CreateViewCommand.md)
 
 `UnresolvedFlow` is registered to a [GraphRegistrationContext](GraphRegistrationContext.md) with [registerFlow](GraphRegistrationContext.md#registerFlow).
 
@@ -23,7 +34,7 @@
 `UnresolvedFlow` is created when:
 
 * `PipelinesHandler` is requested to [define a flow](PipelinesHandler.md#defineFlow)
-* `SqlGraphRegistrationContext` is requested to [handle the following pipeline commands](SqlGraphRegistrationContext.md#processSqlQuery):
+* `SqlGraphRegistrationContext` is requested to [handle the following logical commands](SqlGraphRegistrationContext.md#processSqlQuery):
     * [CreateFlowCommand](SqlGraphRegistrationContext.md#CreateFlowCommand)
     * [CreateMaterializedViewAsSelect](SqlGraphRegistrationContext.md#CreateMaterializedViewAsSelect)
     * [CreateView](SqlGraphRegistrationContext.md#CreateView)