[SDP] Processing SQL libraries + ONCE flows

Author: jaceklaskowski

docs/declarative-pipelines/Flow.md

@@ -1,3 +1,35 @@
 # Flow
 
-`Flow` is...FIXME
+`Flow` is an [extension](#contract) of the [GraphElement](GraphElement.md) abstraction for [flows](#implementations) in dataflow graphs.
+
+Flows must be successfully analyzed, thus resolved, in order to determine whether they are streaming or not.
+
+## Contract (Subset)
+
+### once { #once }
+
+```scala
+once: Boolean
+```
+
+Indicates whether this is a **ONCE flow** or not. ONCE flows can only be run once per full refresh.
+
+* ONCE flows are planned for execution as [AppendOnceFlow](AppendOnceFlow.md)s ([FlowResolver](FlowResolver.md#convertResolvedToTypedFlow))
+* ONCE flows are marked as IDLE when `TriggeredGraphExecution` is requested to start flows in [topologicalExecution](TriggeredGraphExecution.md#topologicalExecution).
+* [ONCE flows must be batch (not streaming)](GraphValidations.md#validateFlowStreamingness).
+* For ONCE flows or when the [logical plan for the flow](ResolvedFlow.md#df) is streaming, `GraphElementTypeUtils` considers a [ResolvedFlow](ResolvedFlow.md) as a `STREAMING_TABLE` (in [getDatasetTypeForMaterializedViewOrStreamingTable](GraphElementTypeUtils.md#getDatasetTypeForMaterializedViewOrStreamingTable)).
+
+Default: `false`
+
+Used when:
+
+* `TriggeredGraphExecution` is requested to [topologicalExecution](TriggeredGraphExecution.md#topologicalExecution)
+* `PipelinesErrors` is requested to [checkStreamingErrorsAndRetry](PipelinesErrors.md#checkStreamingErrorsAndRetry) (to skip ONCE flows with no exception)
+* `GraphValidations` is requested to [validateFlowStreamingness](GraphValidations.md#validateFlowStreamingness)
+* `GraphElementTypeUtils` is requested to [getDatasetTypeForMaterializedViewOrStreamingTable](GraphElementTypeUtils.md#getDatasetTypeForMaterializedViewOrStreamingTable)
+* `FlowResolver` is requested to [convertResolvedToTypedFlow](FlowResolver.md#convertResolvedToTypedFlow)
+
+## Implementations
+
+* [ResolutionCompletedFlow](ResolutionCompletedFlow.md)
+* [UnresolvedFlow](UnresolvedFlow.md)

docs/declarative-pipelines/GraphElement.md

@@ -0,0 +1,3 @@
+# GraphElement
+
+`GraphElement` is...FIXME

docs/declarative-pipelines/GraphElementTypeUtils.md

@@ -0,0 +1,17 @@
+# GraphElementTypeUtils
+
+## getDatasetTypeForMaterializedViewOrStreamingTable { #getDatasetTypeForMaterializedViewOrStreamingTable }
+
+```scala
+getDatasetTypeForMaterializedViewOrStreamingTable(
+  flowsToTable: Seq[ResolvedFlow]): DatasetType
+```
+
+`getDatasetTypeForMaterializedViewOrStreamingTable`...FIXME
+
+---
+
+`getDatasetTypeForMaterializedViewOrStreamingTable` is used when:
+
+* `GraphValidations` is requested to [validateUserSpecifiedSchemas](GraphValidations.md#validateUserSpecifiedSchemas)
+* `SchemaInferenceUtils` is requested to [inferSchemaFromFlows](SchemaInferenceUtils.md#inferSchemaFromFlows)

docs/declarative-pipelines/GraphValidations.md

@@ -0,0 +1,3 @@
+# GraphValidations
+
+`GraphValidations` is...FIXME

docs/declarative-pipelines/PipelinesErrors.md

@@ -0,0 +1,3 @@
+# PipelinesErrors
+
+`PipelinesErrors` is...FIXME

docs/declarative-pipelines/SchemaInferenceUtils.md

@@ -0,0 +1,3 @@
+# SchemaInferenceUtils
+
+`SchemaInferenceUtils` is...FIXME

docs/declarative-pipelines/SqlGraphRegistrationContext.md

@@ -9,13 +9,13 @@
 `SqlGraphRegistrationContext` is created when:
 
 * `PipelinesHandler` is requested to [handle DEFINE_SQL_GRAPH_ELEMENTS command](PipelinesHandler.md#handlePipelinesCommand) (and [defineSqlGraphElements](PipelinesHandler.md#defineSqlGraphElements))
-* `SqlGraphRegistrationContext` is requested to [processSqlFile](#processSqlFile)
+* `SqlGraphRegistrationContext` is requested to [process a SQL file](#processSqlFile)
 
 ## SqlGraphRegistrationContextState { #context }
 
 When [created](#creating-instance), `SqlGraphRegistrationContext` creates a [SqlGraphRegistrationContextState](SqlGraphRegistrationContextState.md) (with the [defaultCatalog](GraphRegistrationContext.md#defaultCatalog), the [defaultDatabase](GraphRegistrationContext.md#defaultDatabase) and the [defaultSqlConf](GraphRegistrationContext.md#defaultSqlConf)).
 
-## Process SQL Definition File { #processSqlFile }
+## Process SQL File { #processSqlFile }
 
 ```scala
 processSqlFile(

docs/declarative-pipelines/UnresolvedFlow.md

@@ -1,19 +1,14 @@
----
-hide:
-    - toc
----
-
 # UnresolvedFlow
 
-`UnresolvedFlow` is a [Flow](Flow.md) that represents flows in the following Python and SQL transformations in [Spark Declarative Pipelines](index.md):
+`UnresolvedFlow` is a [Flow](Flow.md) that represents a flow in the Python and SQL transformations in [Spark Declarative Pipelines](index.md):
 
 * [register_flow](GraphElementRegistry.md#register_flow) in PySpark's decorators
 * [CREATE FLOW ... AS INSERT INTO ... BY NAME](../logical-operators/CreateFlowCommand.md)
 * [CREATE MATERIALIZED VIEW](../logical-operators/CreateMaterializedViewAsSelect.md)
 * [CREATE STREAMING TABLE ... AS](../logical-operators/CreateStreamingTableAsSelect.md)
 * [CREATE VIEW](../logical-operators/CreateView.md) and the other variants of [CREATE VIEW](../logical-operators/CreateViewCommand.md)
 
-`UnresolvedFlow` is registered to a [GraphRegistrationContext](GraphRegistrationContext.md) with [registerFlow](GraphRegistrationContext.md#registerFlow).
+`UnresolvedFlow` is registered to a [GraphRegistrationContext](GraphRegistrationContext.md) with [register a flow](GraphRegistrationContext.md#registerFlow).
 
 `UnresolvedFlow` is analyzed and resolved to [ResolvedFlow](ResolvedFlow.md) (by [FlowResolver](FlowResolver.md#attemptResolveFlow) when [DataflowGraph](DataflowGraph.md) is requested to [resolve](DataflowGraph.md#resolve)).
 
@@ -28,7 +23,7 @@ hide:
 * <span id="func"> `FlowFunction`
 * <span id="queryContext"> `QueryContext`
 * <span id="sqlConf"> SQL Config
-* <span id="once"> `once` flag
+* [once](#once) flag
 * <span id="origin"> `QueryOrigin`
 
 `UnresolvedFlow` is created when:
@@ -40,3 +35,19 @@ hide:
     * [CreateView](SqlGraphRegistrationContext.md#CreateView)
     * [CreateStreamingTableAsSelect](SqlGraphRegistrationContext.md#CreateStreamingTableAsSelect)
     * [CreateViewCommand](SqlGraphRegistrationContext.md#CreateViewCommand)
+
+### once Flag { #once }
+
+`UnresolvedFlow` is given the [once](Flow.md#once) flag when [created](#creating-instance).
+
+`once` flag is disabled (`false`) explicitly for the following:
+
+* [CreateFlowHandler](SqlGraphRegistrationContext.md#CreateFlowHandler)
+* [CreateMaterializedViewAsSelectHandler](SqlGraphRegistrationContext.md#CreateMaterializedViewAsSelectHandler)
+* [CreatePersistedViewCommandHandler](SqlGraphRegistrationContext.md#CreatePersistedViewCommandHandler)
+* [CreateStreamingTableAsSelectHandler](SqlGraphRegistrationContext.md#CreateStreamingTableAsSelectHandler)
+* [CreateTemporaryViewHandler](SqlGraphRegistrationContext.md#CreateTemporaryViewHandler)
+* `PipelinesHandler` is requested to [define a flow](PipelinesHandler.md#defineFlow)
+
+!!! note "No ONCE UnresolvedFlows"
+    It turns out that all `UnresolvedFlow`s created are not [ONCE flows](Flow.md#once).