Skip to content

Add functionality for export of latency logs via telemetry #608

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 69 commits into
base: telemetry
Choose a base branch
from

Conversation

saishreeeee
Copy link
Collaborator

@saishreeeee saishreeeee commented Jun 19, 2025

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Other

Description

Add functionality for export of latency logs via telemetry

Example latency log:
{
"frontend_log_event_id": "30565427-1299-4fbc-bec0-25378cfe912b",
"context": {
"client_context": {
"timestamp_millis": 1750592858108,
"user_agent": "PyDatabricksSqlConnector/4.0.4"
}
},
"entry": {
"sql_driver_log": {
"session_id": "01f04f5e-b1a8-1dd1-962a-e40efd092bdd",
"system_configuration": {
"driver_version": "4.0.4",
"os_name": "Darwin",
"os_version": "24.5.0",
"os_arch": "arm64",
"runtime_name": "Python 3.13.3",
"runtime_version": "3.13.3",
"runtime_vendor": "CPython",
"driver_name": "Databricks SQL Python Connector",
"char_set_encoding": "utf-8",
"locale_name": "en_US"
},
"driver_connection_params": {
"http_path": "/sql/1.0/warehouses/864004c1b3961382",
"mode": "THRIFT",
"host_info": {
"host_url": "e2-dogfood.staging.cloud.databricks.com",
"port": 443
},
"auth_mech": "PAT"
},
"sql_statement_id": "01f04f5e-b1d6-1a1e-afe4-e99e1ccb8805",
"sql_operation": {
"statement_type": "sql",
"is_compressed": true,
"execution_result": "inline_arrow",
"retry_count": 0
},
"operation_latency_ms": 518
}
}
}

How is this tested?

  • Unit tests (will add unit tests)
  • E2E Tests
  • Manually
  • N/A

Related Tickets & Documents

PECOBLR-554

Signed-off-by: Sai Shree Pradhan <[email protected]>
…ze and get telemetry client

Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
…pTelemetryClient class with NOOP_TELEMETRY_CLIENT singleton, updated tests accordingly

Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Signed-off-by: Sai Shree Pradhan <[email protected]>
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Copy link
Contributor

@vikrantpuppala vikrantpuppala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! similar to your previous PR can we add an example latency log to the description?

if hasattr(instance.thrift_backend, "_use_arrow_native_complex_types"):
return ExecutionResultFormat.INLINE_ARROW

return ExecutionResultFormat.FORMAT_UNSPECIFIED
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all of these functions are ifs for Cursor/Connection/ResultSet, can we do something here like have a base class and have implementations for each class?

def log_latency():
def decorator(func):
@functools.wraps(func)
def wrapper(self, *args, **kwargs):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if something fails in this decorator? we should be handling that gracefully

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both the functions: get_telemetry_client and export_latency_log have try/catch blocks in them.

Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: Sai Shree Pradhan <[email protected]>
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Signed-off-by: Sai Shree Pradhan <[email protected]>
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@saishreeeee saishreeeee marked this pull request as ready for review June 22, 2025 11:50
Copy link

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants