Implement SeaDatabricksClient (Complete Execution Spec) (#590)

* [squash from exec-sea] bring over execution phase changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess test

Signed-off-by: varun-edachali-dbx <[email protected]>

* add docstring

Signed-off-by: varun-edachali-dbx <[email protected]>

* remvoe exec func in sea backend

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess files

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess models

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess sea backend tests

Signed-off-by: varun-edachali-dbx <[email protected]>

* cleanup

Signed-off-by: varun-edachali-dbx <[email protected]>

* re-introduce get_schema_desc

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove SeaResultSet

Signed-off-by: varun-edachali-dbx <[email protected]>

* clean imports and attributes

Signed-off-by: varun-edachali-dbx <[email protected]>

* pass CommandId to ExecResp

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove changes in types

Signed-off-by: varun-edachali-dbx <[email protected]>

* add back essential types (ExecResponse, from_sea_state)

Signed-off-by: varun-edachali-dbx <[email protected]>

* fix fetch types

Signed-off-by: varun-edachali-dbx <[email protected]>

* excess imports

Signed-off-by: varun-edachali-dbx <[email protected]>

* reduce diff by maintaining logs

Signed-off-by: varun-edachali-dbx <[email protected]>

* fix int test types

Signed-off-by: varun-edachali-dbx <[email protected]>

* [squashed from exec-sea] init execution func

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove irrelevant changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove ResultSetFilter functionality

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove more irrelevant changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove more irrelevant changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* even more irrelevant changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove sea response as init option

Signed-off-by: varun-edachali-dbx <[email protected]>

* exec test example scripts

Signed-off-by: varun-edachali-dbx <[email protected]>

* formatting (black)

Signed-off-by: varun-edachali-dbx <[email protected]>

* [squashed from sea-exec] merge sea stuffs

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess removed docstring

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess changes in backend

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess imports

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove accidentally removed _get_schema_desc

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove unnecessary init with sea_response tests

Signed-off-by: varun-edachali-dbx <[email protected]>

* rmeove unnecessary changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* formatting (black)

Signed-off-by: varun-edachali-dbx <[email protected]>

* improved models and filters from cloudfetch-sea branch

Signed-off-by: varun-edachali-dbx <[email protected]>

* filters stuff (align with JDBC)

Signed-off-by: varun-edachali-dbx <[email protected]>

* backend from cloudfetch-sea

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove filtering, metadata ops

Signed-off-by: varun-edachali-dbx <[email protected]>

* raise NotImplementedErrror for metadata ops

Signed-off-by: varun-edachali-dbx <[email protected]>

* change to valid table name

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove un-necessary changes

covered by #588

Signed-off-by: varun-edachali-dbx <[email protected]>

* simplify test module

Signed-off-by: varun-edachali-dbx <[email protected]>

* logging -> debug level

Signed-off-by: varun-edachali-dbx <[email protected]>

* change table name in log

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove un-necessary changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove un-necessary backend cahnges

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove un-needed GetChunksResponse

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove un-needed GetChunksResponse

only relevant in Fetch phase

Signed-off-by: varun-edachali-dbx <[email protected]>

* reduce code duplication in response parsing

Signed-off-by: varun-edachali-dbx <[email protected]>

* reduce code duplication

Signed-off-by: varun-edachali-dbx <[email protected]>

* more clear docstrings

Signed-off-by: varun-edachali-dbx <[email protected]>

* introduce strongly typed ChunkInfo

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove is_volume_operation from response

Signed-off-by: varun-edachali-dbx <[email protected]>

* add is_volume_op and more ResultData fields

Signed-off-by: varun-edachali-dbx <[email protected]>

* add test scripts

Signed-off-by: varun-edachali-dbx <[email protected]>

* Revert "Merge branch 'exec-models-sea' into exec-phase-sea"

This reverts commit be1997e, reversing
changes made to 37813ba.

* change logging level

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove un-necessary changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove excess changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove _get_schema_bytes (for now)

Signed-off-by: varun-edachali-dbx <[email protected]>

* redundant comments

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove fetch phase methods

Signed-off-by: varun-edachali-dbx <[email protected]>

* reduce code repetititon + introduce gaps after multi line pydocs

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove unused imports

Signed-off-by: varun-edachali-dbx <[email protected]>

* move description extraction to helper func

Signed-off-by: varun-edachali-dbx <[email protected]>

* formatting (black)

Signed-off-by: varun-edachali-dbx <[email protected]>

* add more unit tests

Signed-off-by: varun-edachali-dbx <[email protected]>

* streamline unit tests

Signed-off-by: varun-edachali-dbx <[email protected]>

* test getting the list of allowed configurations

Signed-off-by: varun-edachali-dbx <[email protected]>

* reduce diff

Signed-off-by: varun-edachali-dbx <[email protected]>

* reduce diff

Signed-off-by: varun-edachali-dbx <[email protected]>

* house constants in enums for readability and immutability

Signed-off-by: varun-edachali-dbx <[email protected]>

* add note on hybrid disposition

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove redundant note on arrow_schema_bytes

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove invalid import

Signed-off-by: varun-edachali-dbx <[email protected]>

* add strong typing for manifest in _extract_description

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove un-necessary column skipping

Signed-off-by: varun-edachali-dbx <[email protected]>

* remove parsing in backend

Signed-off-by: varun-edachali-dbx <[email protected]>

* fix: convert sea statement id to CommandId type

Signed-off-by: varun-edachali-dbx <[email protected]>

* make polling interval a separate constant

Signed-off-by: varun-edachali-dbx <[email protected]>

* align state checking with Thrift implementation

Signed-off-by: varun-edachali-dbx <[email protected]>

* update unit tests according to changes

Signed-off-by: varun-edachali-dbx <[email protected]>

* add unit tests for added methods

Signed-off-by: varun-edachali-dbx <[email protected]>

* add spec to description extraction docstring, add strong typing to params

Signed-off-by: varun-edachali-dbx <[email protected]>

* add strong typing for backend parameters arg

Signed-off-by: varun-edachali-dbx <[email protected]>

---------

Signed-off-by: varun-edachali-dbx <[email protected]>

Author: varun-edachali-dbx

src/databricks/sql/backend/databricks_client.py

@@ -82,7 +82,7 @@ def execute_command(
         lz4_compression: bool,
         cursor: "Cursor",
         use_cloud_fetch: bool,
-        parameters: List[ttypes.TSparkParameter],
+        parameters: List,
         async_op: bool,
         enforce_embedded_schema_correctness: bool,
     ) -> Union["ResultSet", None]: