Skip to content

Ability to Sync rETL Logs to Warehouse #7399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bjackowitz-caring opened this issue Jan 23, 2025 · 2 comments
Open

Ability to Sync rETL Logs to Warehouse #7399

bjackowitz-caring opened this issue Jan 23, 2025 · 2 comments
Assignees

Comments

@bjackowitz-caring
Copy link

File: connections/reverse-etl/manage-retl.md

We want to be able to monitor the history of our rETL syncs more proactively and natively in the tools that we use every day (i.e. - not the Segment web app). If we were able to sync the logs of our rETL runs to the warehouse, it would give us a lot more visibility into the health of those syncs/models which we could address more proactively.

@pwseg pwseg self-assigned this Jan 23, 2025
@pwseg
Copy link
Contributor

pwseg commented Jan 24, 2025

@bjackowitz-caring Thanks for reaching out about this. Could you tell us a bit more about your use case and what specific data you're looking to monitor? For example, are you mostly looking for which syncs succeeded, failed, or partially succeeded? or are you looking for details on specific errors/sync times?

If you can give us more info about your needs, I’ll definitely pass it along to our team.

In the meantime, while it's not currently possible to sync rETL run logs directly to your warehouse, there are some potential workarounds that might be helpful:

  • Monitor emails; Segment sends out monitor emails for failed or partially successful rETL syncs. Those don't include a full log history, but they should give you a high-level overview of sync health and alert you to any issues.
  • Use our API; we have a couple of endpoints you can use to get details about rETL syncs: Get Reverse ETL Sync Status, which gets details for a specific sync (and requires a sync_id), and List Reverse ETL Sync Statuses, which will give you the sync history for a specific model or subscription.

Hopefully this helps, though let me know if you have any other questions.

@bjackowitz-caring
Copy link
Author

Looking primarily for:

  • Specifics on errors - which syncs failed, and was there a specific reason?
  • How many records succeeded, failed, etc for each sync
  • If we can get details on the failures (i.e. - if there is a specific record or value that failed, we'd want to know that)

If we can get record level metrics that'd be awesome, but I understand that could be challenging. Something like one entry for each record we tried to sync, the outcome, and any associated error messages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants