Introduce Insights API #1610

jackiehanyang · 2025-11-11T06:12:29Z

Description

Introducing a new Insights API
- POST /_plugins/_anomaly_detection/insights/_start - Start insights job
- GET /_plugins/_anomaly_detection/insights/_status - Get insights job status
- GET /_plugins/_anomaly_detection/insights/_results - Get latest insights results
- POST /_plugins/_anomaly_detection/insights/_stop - Stop insights job
Introducing ml-commons metrics correlation runtime dependency
- sending anomaly results to ml-commons metrics correlation algorithm to analyze
- write analyze results into insights-results index
- frontend will read from this index to display insights on dashboard

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Jackie <[email protected]>

…nsights

Signed-off-by: Jackie <[email protected]>

kaituo · 2025-11-13T01:37:43Z

CI failed due to jacoco changes in build.gradle. Not sure how to fix. One naive way is to add correlation request, response, and Action in AD to avoid ml-commons dependency.

* What went wrong:
Execution failed for task ':jacocoTestCoverageVerification'.
> A failure occurred while executing org.gradle.internal.jacoco.JacocoCoverageAction
   > Rule violated for class org.opensearch.ad.AnomalyDetectorRunner: branches covered ratio is 0.35, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.AnomalyDetectorRunner: lines covered ratio is 0.47, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.util.ModelUtil: branches covered ratio is 0.32, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.util.ModelUtil: lines covered ratio is 0.48, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.util.DataUtil: lines covered ratio is 0.72, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.feature.AbstractRetriever: branches covered ratio is 0.55, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.feature.AbstractRetriever: lines covered ratio is 0.63, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.feature.SearchFeatureDao: branches covered ratio is 0.28, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.feature.SearchFeatureDao: lines covered ratio is 0.59, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.ModelValidationActionHandler: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.ModelValidationActionHandler: lines covered ratio is 0.00, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.ConfigUpdateConfirmer: branches covered ratio is 0.06, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.ConfigUpdateConfirmer: lines covered ratio is 0.19, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.AggregationPrep: branches covered ratio is 0.36, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.AggregationPrep: lines covered ratio is 0.40, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.IntervalCalculation: branches covered ratio is 0.06, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.IntervalCalculation: lines covered ratio is 0.18, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.LatestTimeRetriever: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.LatestTimeRetriever: lines covered ratio is 0.00, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.handler.IntervalCalculation.IntervalRecommendationListener: branches covered ratio is 0.37, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.handler.IntervalCalculation.IntervalRecommendationListener: lines covered ratio is 0.55, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.ratelimit.ColdStartWorker: branches covered ratio is 0.45, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.ratelimit.ColdStartWorker: lines covered ratio is 0.73, but expected minimum is 0.75
     Rule violated for class org.opensearch.ad.transport.SuggestAnomalyDetectorParamTransportAction: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.transport.SuggestAnomalyDetectorParamTransportAction: lines covered ratio is 0.11, but expected minimum is 0.75
     Rule violated for class org.opensearch.ad.transport.ADSuggestName: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.transport.ADSuggestName: lines covered ratio is 0.57, but expected minimum is 0.75
     Rule violated for class org.opensearch.ad.transport.ADResultProcessor: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.transport.ADResultProcessor: lines covered ratio is 0.61, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.model.InitProgressProfile: branches covered ratio is 0.50, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.model.IntervalTimeConfiguration: branches covered ratio is 0.50, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.ml.MLCommonsClient: lines covered ratio is 0.62, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.RestValidateAction: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.RestValidateAction: lines covered ratio is 0.26, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.RestJobAction: branches covered ratio is 0.00, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.rest.RestJobAction: lines covered ratio is 0.25, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.rest.AbstractSearchAction: lines covered ratio is 0.60, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.transport.SuggestConfigParamResponse: branches covered ratio is 0.59, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.transport.SuggestConfigParamRequest: branches covered ratio is 0.50, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.transport.SuggestConfigParamRequest: lines covered ratio is 0.68, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.transport.SuggestConfigParamResponse.Builder: lines covered ratio is 0.57, but expected minimum is 0.75
     Rule violated for class org.opensearch.timeseries.transport.handler.IndexMemoryPressureAwareResultHandler: branches covered ratio is 0.54, but expected minimum is 0.60
     Rule violated for class org.opensearch.timeseries.transport.handler.IndexMemoryPressureAwareResultHandler: lines covered ratio is 0.68, but expected minimum is 0.75
     Rule violated for class org.opensearch.ad.ratelimit.ADSaveResultStrategy: branches covered ratio is 0.43, but expected minimum is 0.60
     Rule violated for class org.opensearch.ad.ratelimit.ADSaveResultStrategy: lines covered ratio is 0.53, but expected minimum is 0.75

kaituo

partial review

kaituo · 2025-11-13T01:50:41Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+
+        log.info("Built correlation input: {} metrics × {} buckets", input.getNumMetrics(), input.getNumBuckets());
+
+        log.info("Matrix contents: {}", input.getMatrix());


The matrix can be huge and may expose metric/entity values. Recommend DEBUG with truncation or only log dimensions.

added it for testing purpose, will clean up

kaituo · 2025-11-13T02:32:08Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+            executionStartTime = executionEndTime.minus(24, ChronoUnit.HOURS);
+            // executionStartTime = executionEndTime.minus(intervalAmount, intervalUnit);


You want to use the commented out line instead of hard coded 24 hours?

oops, changed it for testing purpose, will clean up

kaituo · 2025-11-13T02:50:01Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+
+        InjectSecurity injectSecurity = new InjectSecurity(jobParameter.getName(), settings, localClient.threadPool().getThreadContext());
+        try {
+            injectSecurity.inject(user, roles);


A normal user cannot query system index. Please add security tests.

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

kaituo · 2025-11-13T02:56:55Z

src/main/java/org/opensearch/ad/constant/ADCommonName.java

+    // Insights job
+    // ======================================
+    // The Insights job name
+    public static final String INSIGHTS_JOB_NAME = "insights_job";


how about changing to ad_insights_job in case we need forecasting job later?

kaituo

partial review

kaituo · 2025-11-14T19:17:33Z

src/main/java/org/opensearch/ad/rest/RestInsightsJobAction.java

+        return ImmutableList
+            .of(
+                // Start insights job
+                new ReplacedRoute(


You don't need ReplaceRoute as this is a new API. TimeSeriesAnalyticsPlugin.AD_BASE_URI alone is enough.

kaituo · 2025-11-14T19:51:31Z

src/main/java/org/opensearch/ad/ml/InsightsGenerator.java

+        builder.startObject();
+
+        // Task metadata
+        builder.field("task_id", "task_" + ADCommonName.INSIGHTS_JOB_NAME + "_" + UUID.randomUUID().toString());


Why do you need task id? AD task id is the doc id of state index.

kaituo · 2025-11-14T20:08:22Z

src/main/java/org/opensearch/ad/ml/InsightsGenerator.java

+
+                if (parts.length > 1) {
+                    String seriesKey = parts[1];
+                    seriesKeys.add(seriesKey);


Is the entities set redundant with seriesKeys set?

kaituo · 2025-11-14T20:51:31Z

src/main/java/org/opensearch/ad/ml/MLMetricsCorrelationInputBuilder.java

+                // Use MAX score if multiple anomalies in same bucket
+                double currentScore = bucketScores.getOrDefault(bucketIndex, 0.0);
+                double newScore = anomaly.getAnomalyScore();
+                bucketScores.put(bucketIndex, Math.max(currentScore, newScore));


Should we consider interval? Our anomalies are interval anomalies. We can put anomalies scores to all of the buckets interleaving current interval [data start, data end]. If you have already done it, can you point me the code? I cannot find it.

kaituo · 2025-11-14T20:54:28Z

src/main/resources/mappings/insights-results.json

+            "fields": {
+              "raw": {
+                "type": "keyword",
+                "ignore_above": 32766
+              }


do you need keyword?

Signed-off-by: Jackie <[email protected]>

kaituo

partial review

kaituo · 2025-11-21T20:25:06Z

src/main/java/org/opensearch/ad/transport/InsightsJobTransportAction.java

+            handleStartOperation(request, listener);
+        } else if (request.isStatusOperation()) {
+            handleStatusOperation(request, listener);
+        } else if (request.isStopOperation()) {
+            handleStopOperation(request, listener);


Do you need to stash context before accessing job index (system index)? Please add security tests.

kaituo · 2025-11-21T20:25:43Z

src/main/java/org/opensearch/ad/transport/InsightsJobTransportAction.java

+                        .sort("generated_at", SortOrder.DESC)
+                );
+
+            client.search(searchRequest, ActionListener.wrap(searchResponse -> {


Do you need to add backend role filtering before search? Please add security tests with backend role filtering on.

kaituo · 2025-11-21T22:51:49Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+
+    private static final Logger log = LogManager.getLogger(InsightsJobProcessor.class);
+
+    private static InsightsJobProcessor INSTANCE;


add volatile. read https://en.wikipedia.org/wiki/Double-checked_locking

kaituo · 2025-11-21T23:18:08Z

src/main/java/org/opensearch/ad/InsightsJobProcessor.java

+        try {
+            injectSecurity.inject(user, roles);
+
+            localClient


We should verify if mapping is changed by customer before writing. If yes, report error/stop job and stop writing.

kaituo · 2025-11-21T23:21:10Z

src/main/java/org/opensearch/ad/rest/handler/InsightsJobActionHandler.java

+                Instant.now(),
+                lockDurationSeconds,
+                user,
+                ADCommonName.INSIGHTS_RESULT_INDEX_ALIAS,


Can you add this index in ADIndex? This would be consistent with other indexes.

Signed-off-by: Jackie <[email protected]>

jackiehanyang added 3 commits November 6, 2025 10:48

initial commit

64021b5

Signed-off-by: Jackie <[email protected]>

Merge remote-tracking branch 'origin/main' into tmp/merge-main-into-i…

45e795f

…nsights

Introduce Insights API

b14da81

Signed-off-by: Jackie <[email protected]>

jackiehanyang requested review from VijayanB, amitgalitz, dbwiddis, jmazanec15, jngz-es, joshpalis, kaituo, ohltyler, owaiskazi19, saratvemulapalli, sean-zheng-amazon, vamshin and ylwu-amzn as code owners November 11, 2025 06:12

opensearch-trigger-bot bot added infra Changes to infrastructure, testing, CI/CD, pipelines, etc. backport 2.x labels Nov 11, 2025

jackiehanyang added 4 commits November 10, 2025 22:28

spotless apply

ed68a6d

Signed-off-by: Jackie <[email protected]>

add change log

200d461

Signed-off-by: Jackie <[email protected]>

fix

671e103

Signed-off-by: Jackie <[email protected]>

make timestamp more user friendly when reading

77fcda9

Signed-off-by: Jackie <[email protected]>

jackiehanyang force-pushed the insights branch from 4883d42 to 77fcda9 Compare November 11, 2025 20:00

jackiehanyang added 2 commits November 11, 2025 12:11

fix forbidden api violation by avoiding default-locale usage

3d50179

Signed-off-by: Jackie <[email protected]>

add more tests

49018e0

Signed-off-by: Jackie <[email protected]>

jackiehanyang force-pushed the insights branch from 0eba385 to 49018e0 Compare November 12, 2025 17:31

kaituo reviewed Nov 13, 2025

View reviewed changes

kaituo reviewed Nov 14, 2025

View reviewed changes

use stashed context when starting insights job

816d69b

Signed-off-by: Jackie <[email protected]>

kaituo reviewed Nov 22, 2025

View reviewed changes

add one time insights job run

1356920

Signed-off-by: Jackie <[email protected]>

jackiehanyang force-pushed the insights branch from 3076f15 to 1356920 Compare November 22, 2025 09:20


		log.info("Built correlation input: {} metrics × {} buckets", input.getNumMetrics(), input.getNumBuckets());

		log.info("Matrix contents: {}", input.getMatrix());

		executionStartTime = executionEndTime.minus(24, ChronoUnit.HOURS);
		// executionStartTime = executionEndTime.minus(intervalAmount, intervalUnit);


		private static final Logger log = LogManager.getLogger(InsightsJobProcessor.class);

		private static InsightsJobProcessor INSTANCE;

Introduce Insights API #1610

Are you sure you want to change the base?

Introduce Insights API #1610

Uh oh!

Conversation

jackiehanyang commented Nov 11, 2025

Description

Related Issues

Check List

Uh oh!

kaituo commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaituo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaituo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaituo Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaituo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kaituo commented Nov 13, 2025 •

edited

Loading

kaituo Nov 14, 2025 •

edited

Loading