Skip to content

Commit e26f7bb

Browse files
jaceklaskowskijaceklaskowski
committed
[SDP] SQL statements
1 parent 84c979d commit e26f7bb

File tree

4 files changed

+48
-41
lines changed

4 files changed

+48
-41
lines changed

‎docs/declarative-pipelines/SqlGraphRegistrationContext.md‎

Lines changed: 39 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -47,15 +47,17 @@ processSqlQuery(
4747

4848
`processSqlQuery` handles (_processes_) the given [LogicalPlan](../logical-operators/LogicalPlan.md) logical commands:
4949

50-
* [CreateFlowCommand](#CreateFlowCommand)
51-
* [CreateMaterializedViewAsSelect](#CreateMaterializedViewAsSelect)
52-
* [CreateStreamingTable](#CreateStreamingTable)
53-
* [CreateStreamingTableAsSelect](#CreateStreamingTableAsSelect)
54-
* [CreateView](#CreateView)
55-
* [CreateViewCommand](#CreateViewCommand)
56-
* [SetCatalogCommand](#SetCatalogCommand)
57-
* [SetCommand](#SetCommand)
58-
* [SetNamespaceCommand](#SetNamespaceCommand)
50+
| Logical Command | Command Handler | Datasets |
51+
|-|-|-|
52+
| [CreateFlowCommand](../logical-operators/CreateFlowCommand.md) | [CreateFlowHandler](#CreateFlowCommand) | [UnresolvedFlow](GraphRegistrationContext.md#registerFlow) ([once](UnresolvedFlow.md#once) disabled) |
53+
| [CreateMaterializedViewAsSelect](../logical-operators/CreateMaterializedViewAsSelect.md) | [CreateMaterializedViewAsSelectHandler](#CreateMaterializedViewAsSelect) | [Table](GraphRegistrationContext.md#registerTable) ([isStreamingTable](Table.md#isStreamingTable) disabled)<br>[UnresolvedFlow](GraphRegistrationContext.md#registerFlow) ([once](UnresolvedFlow.md#once) disabled) |
54+
| [CreateStreamingTable](../logical-operators/CreateStreamingTable.md) | [CreateStreamingTableHandler](#CreateStreamingTable) | [Table](GraphRegistrationContext.md#registerTable) ([isStreamingTable](Table.md#isStreamingTable) enabled) |
55+
| [CreateStreamingTableAsSelect](../logical-operators/CreateStreamingTableAsSelect.md) | [CreateStreamingTableAsSelectHandler](#CreateStreamingTableAsSelect) | [Table](GraphRegistrationContext.md#registerTable) ([isStreamingTable](Table.md#isStreamingTable) enabled)<br>[UnresolvedFlow](GraphRegistrationContext.md#registerFlow) ([once](UnresolvedFlow.md#once) disabled) |
56+
| [CreateView](../logical-operators/CreateView.md) | [CreatePersistedViewCommandHandler](#CreateView) | [PersistedView](GraphRegistrationContext.md#registerView)<br>[UnresolvedFlow](GraphRegistrationContext.md#registerFlow) ([once](UnresolvedFlow.md#once) disabled) |
57+
| [CreateViewCommand](../logical-operators/CreateViewCommand.md) | [CreateTemporaryViewHandler](#CreateViewCommand) | [TemporaryView](GraphRegistrationContext.md#registerView)<br>[UnresolvedFlow](GraphRegistrationContext.md#registerFlow) ([once](UnresolvedFlow.md#once) disabled) |
58+
| [SetCatalogCommand](../logical-operators/SetCatalogCommand.md) | [SetCatalogCommandHandler](#SetCatalogCommand) | |
59+
| [SetCommand](../logical-operators/SetCommand.md) | [SetCommandHandler](#SetCommand) | |
60+
| [SetNamespaceCommand](../logical-operators/SetNamespaceCommand.md) | [SetNamespaceCommandHandler](#SetNamespaceCommand) | |
5961

6062
### splitSqlFileIntoQueries { #splitSqlFileIntoQueries }
6163

@@ -78,7 +80,7 @@ A flow name must be a single-part name (that is resolved against the current pip
7880

7981
The [flowOperation](../logical-operators/CreateFlowCommand.md#flowOperation) of a [CreateFlowCommand](../logical-operators/CreateFlowCommand.md) command must be [InsertIntoStatement](../logical-operators/InsertIntoStatement.md).
8082

81-
!!! note
83+
??? warning
8284
Only `INSERT INTO ... BY NAME` flows are supported in [Spark Declarative Pipelines](index.md).
8385

8486
`INSERT OVERWRITE` flows are not supported.
@@ -91,46 +93,49 @@ In the end, `CreateFlowHandler` requests this [GraphRegistrationContext](#graphR
9193

9294
### CreateMaterializedViewAsSelect { #CreateMaterializedViewAsSelect }
9395

94-
[CreateMaterializedViewAsSelect](../logical-operators/CreateMaterializedViewAsSelect.md) logical commands are handled by `CreateMaterializedViewAsSelectHandler`.
96+
[processSqlQuery](#processSqlQuery) handles [CreateMaterializedViewAsSelect](../logical-operators/CreateMaterializedViewAsSelect.md) logical commands using `CreateMaterializedViewAsSelectHandler`.
9597

96-
`CreateMaterializedViewAsSelectHandler` requests this [GraphRegistrationContext](#graphRegistrationContext) to register a [table](GraphRegistrationContext.md#registerTable) and a [flow](GraphRegistrationContext.md#registerFlow) (that backs the materialized view).
98+
`CreateMaterializedViewAsSelectHandler` requests this [GraphRegistrationContext](#graphRegistrationContext) to [register a table](GraphRegistrationContext.md#registerTable) and a [flow](GraphRegistrationContext.md#registerFlow) (that backs the materialized view).
9799

98100
### CreateStreamingTable { #CreateStreamingTable }
99101

100102
[processSqlQuery](#processSqlQuery) handles [CreateStreamingTable](../logical-operators/CreateStreamingTable.md) logical commands using `CreateStreamingTableHandler`.
101103

102-
```scala
103-
handle(
104-
cst: CreateStreamingTable,
105-
queryOrigin: QueryOrigin): Unit
106-
```
107-
108-
`handle` requests this [SqlGraphRegistrationContextState](#context) to [register a streaming table](GraphRegistrationContext.md#registerTable).
104+
`CreateStreamingTableHandler` requests this [SqlGraphRegistrationContextState](#context) to [register a streaming table](GraphRegistrationContext.md#registerTable).
109105

110106
### CreateStreamingTableAsSelect { #CreateStreamingTableAsSelect }
111107

112108
[processSqlQuery](#processSqlQuery) handles [CreateStreamingTableAsSelect](../logical-operators/CreateStreamingTableAsSelect.md) logical commands using `CreateStreamingTableAsSelectHandler`.
113109

114-
```scala
115-
handle(
116-
cst: CreateStreamingTableAsSelect,
117-
queryOrigin: QueryOrigin): Unit
118-
```
110+
`CreateStreamingTableAsSelectHandler` requests this [SqlGraphRegistrationContextState](#context) to [register a streaming table](GraphRegistrationContext.md#registerTable) and the accompanying [flow](GraphRegistrationContext.md#registerFlow) (for the streaming table).
111+
112+
### CreateView { #CreateView }
113+
114+
[processSqlQuery](#processSqlQuery) handles [CreateView](../logical-operators/CreateView.md) logical commands using `CreatePersistedViewCommandHandler`.
115+
116+
`CreatePersistedViewCommandHandler` requests this [GraphRegistrationContext](#graphRegistrationContext) to [register a PersistedView](GraphRegistrationContext.md#registerView) and the accompanying [flow](GraphRegistrationContext.md#registerFlow) (for the `PersistedView`).
117+
118+
### CreateViewCommand { #CreateViewCommand }
119119

120-
`handle` requests this [SqlGraphRegistrationContextState](#context) to [register a streaming table](GraphRegistrationContext.md#registerTable) and the accompanying [flow](GraphRegistrationContext.md#registerFlow) (for the streaming table).
120+
[processSqlQuery](#processSqlQuery) handles [CreateViewCommand](../logical-operators/CreateViewCommand.md) logical commands using `CreateTemporaryViewHandler`.
121+
122+
`CreateTemporaryViewHandler` requests this [GraphRegistrationContext](#graphRegistrationContext) to [register a TemporaryView](GraphRegistrationContext.md#registerView) and the accompanying [flow](GraphRegistrationContext.md#registerFlow) (for the `TemporaryView`).
123+
124+
### SetCatalogCommand { #SetCatalogCommand }
125+
126+
[processSqlQuery](#processSqlQuery) handles [SetCatalogCommand](../logical-operators/SetCatalogCommand.md) logical commands using `SetCatalogCommandHandler`.
127+
128+
`SetCatalogCommandHandler` requests this [SqlGraphRegistrationContextState](#context) to [setCurrentCatalog](SqlGraphRegistrationContextState.md#setCurrentCatalog) to the [catalogName](../logical-operators/SetCatalogCommand.md#catalogName) of the given [SetCatalogCommand](../logical-operators/SetCatalogCommand.md).
129+
130+
In the end, `SetCatalogCommandHandler` requests this [SqlGraphRegistrationContextState](#context) to [clearCurrentDatabase](SqlGraphRegistrationContextState.md#clearCurrentDatabase).
121131

122132
### SetCommand { #SetCommand }
123133

124134
[processSqlQuery](#processSqlQuery) handles [SetCommand](../logical-operators/SetCommand.md) logical commands using `SetCommandHandler`.
125135

126-
```scala
127-
handle(
128-
setCommand: SetCommand): Unit
129-
```
130-
131-
`handle` requests this [SqlGraphRegistrationContextState](#context) to [setSqlConf](#setSqlConf) with the key-value pair of the given [SetCommand](../logical-operators/SetCommand.md) logical command.
136+
`SetCommandHandler` requests this [SqlGraphRegistrationContextState](#context) to [setSqlConf](#setSqlConf) with the key-value pair of the given [SetCommand](../logical-operators/SetCommand.md) logical command.
132137

133-
??? note "RuntimeException"
138+
??? warning "RuntimeException"
134139

135140
`handle` makes sure that the given `SetCommand` comes with a `key = value` pair or throws a `RuntimeException`:
136141

@@ -146,17 +151,12 @@ handle(
146151

147152
[processSqlQuery](#processSqlQuery) handles [SetNamespaceCommand](../logical-operators/SetNamespaceCommand.md) logical commands using `SetNamespaceCommandHandler`.
148153

149-
```scala
150-
handle(
151-
setNamespaceCommand: SetNamespaceCommand): Unit
152-
```
153-
154-
`handle` requests this [SqlGraphRegistrationContextState](#context) for the following:
154+
`SetNamespaceCommandHandler` requests this [SqlGraphRegistrationContextState](#context) for the following:
155155

156156
* For a `database`-only, single-part namespace, [setCurrentDatabase](SqlGraphRegistrationContextState.md#setCurrentDatabase)
157157
* For a `catalog.database` two-part namespace, [setCurrentCatalog](SqlGraphRegistrationContextState.md#setCurrentCatalog) and [setCurrentDatabase](SqlGraphRegistrationContextState.md#setCurrentDatabase)
158158

159-
??? note "SparkException"
159+
??? warning "SparkException"
160160

161161
`handle` throws a `SparkException` for invalid namespaces:
162162

‎docs/declarative-pipelines/index.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,11 @@ Supported SQL statements:
181181
* [CREATE MATERIALIZED VIEW ... AS](../sql/SparkSqlAstBuilder.md#visitCreatePipelineDataset)
182182
* [CREATE STREAMING TABLE](../sql/SparkSqlAstBuilder.md#visitCreatePipelineDataset)
183183
* [CREATE STREAMING TABLE ... AS](../sql/SparkSqlAstBuilder.md#visitCreatePipelineDataset)
184-
* ...
184+
* [CREATE (PERSISTED) VIEW](../sql/SparkSqlAstBuilder.md#visitCreateView)
185+
* [CREATE TEMPORARY VIEW](../sql/SparkSqlAstBuilder.md#visitCreateView)
186+
* [SET](../logical-operators/SetCommand.md)
187+
* [SET CATALOG](../logical-operators/SetCatalogCommand.md)
188+
* [USE NAMESPACE](../logical-operators/SetNamespaceCommand.md)
185189

186190
A streaming table can be defined without a query, as streaming tables' data can be backed by standalone flows.
187191
During a pipeline execution, it is validated that a streaming table has at least one standalone flow writing to the table, if no query is specified in the create statement itself.

‎docs/logical-operators/SetCatalogCommand.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# SetCatalogCommand
2+
3+
`SetCatalogCommand` is...FIXME

‎docs/logical-operators/SetNamespaceCommand.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,4 +18,4 @@ USE [NAMESPACE | DATABASE | SCHEMA] identifier
1818

1919
`SetNamespaceCommand` is created when:
2020

21-
* `SparkSqlAstBuilder` is requested to parse [USE NAMESPACE](../sql/SparkSqlAstBuilder.md#visitUseNamespace) SQL statement
21+
* `SparkSqlAstBuilder` is requested to [parse USE NAMESPACE SQL statement](../sql/SparkSqlAstBuilder.md#visitUseNamespace)

0 commit comments

Comments
 (0)