Skip to content

DVPL-12497: Fix CSC apps' tests #633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 19, 2025

Conversation

cecyliaborek
Copy link

@cecyliaborek cecyliaborek commented Aug 8, 2025

Fixed a flaky test tests/searchcommands/test_csc_apps.py::TestCSC::test_eventing_app caused by:

  • Unreliable data source – The test relied on events from Splunk’s _internal index, which is not guaranteed to contain the required data.
  • Malformed search query - missing status field caused eventingcsc status=200 to raise KeyError, which wasn’t caught by the test assertions.

The Fix

  • Created a helper method _create_test_data_search that creates synthetic events for the eventingcsc command
  • Made assertions stricter to check both result count and field values.

@cecyliaborek cecyliaborek force-pushed the dev/CB/DVPL-12497-fix-csc-apps-test branch from 5fc4ec1 to d514410 Compare August 11, 2025 09:33
@Ickerday
Copy link
Collaborator

For reasons yet unknown to me, we check for is_preview in results - here's how I dealt with this for Splunk 10. Maybe it'll be useful
c14f4f3

@cecyliaborek cecyliaborek force-pushed the dev/CB/DVPL-12497-fix-csc-apps-test branch from d514410 to 2b63865 Compare August 11, 2025 14:03
@Ickerday
Copy link
Collaborator

Ickerday commented Aug 11, 2025

To be honest, I don't understand what is going on here. Why are we testing for errors in Splunk? I thought what actually needs fixing is the query. Otherwise, the test that was already there looks fine to me - it tests the eventing app's behaviour, not Splunk's.

@cecyliaborek
Copy link
Author

To be honest, I don't understand what is going on here. Why are we testing for errors in Splunk? I thought what actually needs fixing is the query. Otherwise, the test that was already there looks fine to me - it tests the eventing app's behaviour, not Splunk's.

I don’t think the goal is to test the eventing app itself (it’s just a simple example) but rather the SDK components used to build it. My thinking was that it’s good to cover not just the happy path, but also cases where something is done incorrectly, and ensure the SDK handles that properly. That said, you’re right — maybe we should focus on testing cases where an app is created incorrectly, rather than cases where it’s just used incorrectly.

@cecyliaborek cecyliaborek force-pushed the dev/CB/DVPL-12497-fix-csc-apps-test branch 5 times, most recently from 12d213b to 14d70f5 Compare August 13, 2025 11:47
@cecyliaborek cecyliaborek marked this pull request as ready for review August 13, 2025 12:05
@cecyliaborek cecyliaborek force-pushed the dev/CB/DVPL-12497-fix-csc-apps-test branch from 14d70f5 to 8ee7e8a Compare August 13, 2025 14:11
@cecyliaborek cecyliaborek force-pushed the dev/CB/DVPL-12497-fix-csc-apps-test branch from 8ee7e8a to ccf4e8e Compare August 13, 2025 14:36
@Ickerday Ickerday changed the title Fix csc apps test DVPL-12497: Fix CSC apps' tests Aug 19, 2025
@Ickerday Ickerday merged commit a84c951 into develop Aug 19, 2025
21 of 22 checks passed
@Ickerday Ickerday deleted the dev/CB/DVPL-12497-fix-csc-apps-test branch August 19, 2025 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants