[Entity Analytics][Hypotheses] Prebuilt hypotheses initialisation and reconciliation

[CAWilson94]

Playground for Threat Hunting Hypothesis Definitions Initialisation and Reconciliation — testing Saved Object creation, schema setup, client wiring along with init and reconciliation of hypothesis definitions list to surface questions early.

What it does currently:

• On kibana startup, if feature flag enabled - call hypothesis definitions init
• pull in hard coded list of definitions, for each of these, store down into saved objects.
• on each startup - runs upsert and does deletion when version is bumped. ✅
• Logs single message with created, deleted, updated.
• Runs recon - renamed to updateThreatHuntingHypothesisDefinitions, moved to scheduleEntityAnalyticsMigration 👍
• FTR test - validates init saves hypotheses to saved objects.

🚧 In progress:

1. Fresh round of smoke testing - updated to use scheduleEntityAnalyticsMigration but have NOT tested here.
2. FTR Tests - recon, validate deletion and upgrade version numbers.

Notes:

• Space awareness: using agnostic / global namespace for hard coded hypotheses.
• AuditLogger/AuditService - previous from pluginSetup/Start deprecated so using Core AuditService usage ml plugin as reference.
  • EntityAnalyticsMigrationsParams uses old auditLogger type 🎏

How To Test Locally

These steps worked before move to call from scheduleEntityAnalyticsMigration, hopefully 🤞they still do:

1. Start kibana with entityThreatHuntingEnabled feature flag enabled.
2. On first run, expect 2 saved objects of type 'threat-hunting-hypothesis' to be created:

GET .kibana_security_solution*/_search
{
  "size": 50,
  "query": {
    "term": { "type": "threat-hunting-hypothesis" }
  }
}

3. Bump the HYPOTHESES_VERSION to 2, check dev tools again, version on all saved objects should bump to 2 and log message should specify 2 updated.
4. Delete any of the hypothesis definitions from getHypothesisDefinitions in x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/hypothesis_threat_hunting/lib/hypothesis_definitions.ts then bump the version again - expect 1 updated, 1 deleted, 0 created in log. Expect only one saved object of threat-hunting-hypothesis type should now exist.

Preview, dev tools from Saved Object:

[CAWilson94]

@tiansivive @machadoum I 100% copy pasta'd this logger from both your efforts (thank you 🙏) - QQ on the categories --> for saved_object_create (specified in docs here) is this good enough or is there something else I need to add?

[elasticmachine]

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 620dc29

Failed CI Steps

• Quick Checks
• Checks
• Check Types

History

• 💔 Build #359766 failed 0a4ae84
• 💔 Build #359746 failed 74a086d
• 💔 Build #359662 failed 4676813

[jbudz]

.buildkite/ftr_security_stateful_configs.yml LGTM