Updates Docker and Kubernetes to use EDOT Collector instead of APM Server #452

codefromthecrypt · 2025-04-23T03:35:35Z

This drops apm-server from our public examples here, and replaces it with Elastic Distribution of OpenTelemetry (EDOT) Collector. This simplifies configuration as it uses expected ports, and encourages the distribution.

This configuration was kicked around internally the last couple weeks and brought here (to public) in tact.

After this, any of our EDOT SDK examples that reference the docker config should probably update. I'll update the couple example projects I use at conferences. My motivation is to have this available so that Cloud Native Summit next week, if I do a demo it can use EDOT collector, not APM server.

codefromthecrypt · 2025-04-23T03:36:12Z

Screen shots of chatbot-rag-app using this:

docker:

k8s:

…rver

codefromthecrypt · 2025-04-23T03:52:36Z

k8s/k8s-manifest-elastic.yml

          readinessProbe:
            tcpSocket:
-              port: 8200
+              port: 4317


I chose to only check the grpc port to keep things simple

codefromthecrypt · 2025-04-23T03:59:47Z

here are the readiness probes working:

$ kubectl get pods
NAME                              READY   STATUS      RESTARTS   AGE
chatbot-rag-app-8956bdd6f-7wnbq   1/1     Running     0          10m
elasticsearch-65b96f5bdb-6r52k    1/1     Running     0          11m
kibana-5ccdf5bcd9-7glkw           1/1     Running     0          11m
otel-collector-7bb456d445-5fxrh   1/1     Running     0          11m
setup-kibana-system-user-g57sb    0/1     Completed   0          15m

codefromthecrypt · 2025-04-23T04:23:44Z

looks like the elser smoke test is failing due to being slower than before.. will look into that now

Signed-off-by: Adrian Cole <[email protected]>

felixbarny · 2025-04-23T05:12:05Z

docker/docker-compose-elastic.yml

+          mapping:
+            mode: otel
+          logs_dynamic_index:
+            enabled: true
+          metrics_dynamic_index:
+            enabled: true
+          traces_dynamic_index:
+            enabled: true


This should be the default config and thus can probably be removed to simplify.

Once upon a time I thought tried this and it didn't work, but was likely something else. will try it and resolve if I can!

We changed the defaults since.

bad news.. I remember this from before. so maybe it isn't default yet for 9.0.0?

2025-04-23T07:26:53.308Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:54.306Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:54.306Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:54.306Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:54.307Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:54.307Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:54.307Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:54.311Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "index": ".ds-metrics-generic.otel-default-2025.04.23-000001", "error.type": "version_conflict_engine_exception", "error.reason": "[k-PIaMvtD8VS5En8AAABlmGLzBg][dLstI3i0U_2YySajCZHIT4H97y5Pg75paAm6yduLEF6l5hxci4O-aWjxg21cn4NT1cc8PhdJOCZ_Fh4bVNlcJO4CGLq-FsIGgM_qxe1jUUH7d4idRLDI76OR38GRJje-JBuPjK4LcPX44D6VJjTDPkq38IVp@2025-04-23T07:27:53.880Z]: version conflict, document already exists (current version [1])", "hint": "check the \"Known issues\" section of Elasticsearch Exporter docs"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:56.304Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:59.305Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213 2025-04-23T07:27:59.305Z error [email protected]/bulkindexer.go:354 failed to index document {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces", "index": "traces-generic-default", "error.type": "index_not_found_exception", "error.reason": "no such index [traces-generic-default] and the index creation request requires a data stream, but no matching index template with data stream template was found for it"} github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.flushBulkIndexer github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:354 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).flush github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:326 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*asyncBulkIndexerWorker).run github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:319 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newAsyncBulkIndexer.func1 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/bulkindexer.go:213

specifically this which I reverted after noticing, was the change that had that error in the collector logs 13814c8

Not in 9.0.0. It requires v0.122.0 es exporter which isn't in any stack versioned releases yet.

docker/docker-compose-elastic.yml

Signed-off-by: Adrian Cole <[email protected]>

codefromthecrypt · 2025-04-23T06:39:12Z

ok amd64 passes as expected I think the latest release of ES is a bit too much for the free arm64 github runner. I'll remove it from the CI matrix as it was really slow before anyway (due to how ELSER works it is running an unoptimized variant when not on x86)

Signed-off-by: Adrian Cole <[email protected]>

This reverts commit 13814c8.

codefromthecrypt · 2025-04-23T07:47:06Z

mental note: to save the extremely arduous iteration cycle... it might make sense to make something fake data into the collector and look for errors. what I noticed is that the errors happen in a way that doesn't crash the app (I guess good) just you need to look at the logs to notice if there was a problem due to collector config or not. @axw no rush but if you have any smoke tests from collector land like this that are simple and containable to drop into CI, I might add or make a new CI script for it. That way it would fail without having to wait for ELSER to boot etc or doing the same locally.

codefromthecrypt · 2025-04-24T00:18:50Z

going to merge this as improvements require changes made and released to EDOT collector and blocking until then doesn't add value.

codefromthecrypt · 2025-04-24T00:19:57Z

any follow-ups should be tiny and welcome changes in PRs. Just make sure if anyone changes the EDOT config they look at the collector logs to verify nothing broke (e.g. docker logs otel-collector after it has collected some signals)

axw · 2025-04-28T01:33:37Z

mental note: to save the extremely arduous iteration cycle... it might make sense to make something fake data into the collector and look for errors. what I noticed is that the errors happen in a way that doesn't crash the app (I guess good) just you need to look at the logs to notice if there was a problem due to collector config or not. @axw no rush but if you have any smoke tests from collector land like this that are simple and containable to drop into CI, I might add or make a new CI script for it. That way it would fail without having to wait for ELSER to boot etc or doing the same locally.

I'm not aware of any tests like that. The smoke tests that we run send data to the collector and then wait for it to end up in Elasticsearch. e.g. see https://github.com/elastic/hosted-otel-collector/blob/2e08c4100f15bf41d5a1669fa517206112c30505/testing/smoketest/otlp_test.go#L72-L101

codefromthecrypt requested review from xrmx, trentm, joemcelroy, estolfo, axw, bshetti, felixbarny, ygenkina, SrdjanLL, emma-raffenne and daniela-elastic April 23, 2025 03:35

codefromthecrypt requested a review from anuraaga as a code owner April 23, 2025 03:35

Updates Docker and Kubernetes to use EDOT Collector instead of APM Se…

c8af42b

…rver

codefromthecrypt force-pushed the edot-collector branch from f19b25d to c8af42b Compare April 23, 2025 03:47

codefromthecrypt commented Apr 23, 2025

View reviewed changes

good night

b3441a0

Signed-off-by: Adrian Cole <[email protected]>

felixbarny reviewed Apr 23, 2025

View reviewed changes

codefromthecrypt added 2 commits April 23, 2025 14:26

try ES 9 client (unlikely to fix the arm timeout, but anyway)

564105b

Signed-off-by: Adrian Cole <[email protected]>

don't fail fast to see if x86 passes

a5ecd9f

Signed-off-by: Adrian Cole <[email protected]>

codefromthecrypt added 3 commits April 23, 2025 14:41

don't test on arm64

f82c431

Signed-off-by: Adrian Cole <[email protected]>

less config

13814c8

Signed-off-by: Adrian Cole <[email protected]>

Revert "less config"

0491b96

This reverts commit 13814c8.

codefromthecrypt merged commit 0a1cf84 into main Apr 24, 2025
5 checks passed

codefromthecrypt deleted the edot-collector branch April 24, 2025 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates Docker and Kubernetes to use EDOT Collector instead of APM Server #452

Updates Docker and Kubernetes to use EDOT Collector instead of APM Server #452

codefromthecrypt commented Apr 23, 2025

codefromthecrypt commented Apr 23, 2025

codefromthecrypt Apr 23, 2025

codefromthecrypt commented Apr 23, 2025 •

edited

Loading

codefromthecrypt commented Apr 23, 2025

felixbarny Apr 23, 2025

codefromthecrypt Apr 23, 2025

felixbarny Apr 23, 2025

codefromthecrypt Apr 23, 2025

codefromthecrypt Apr 23, 2025

carsonip Apr 25, 2025

codefromthecrypt commented Apr 23, 2025 •

edited

Loading

codefromthecrypt commented Apr 23, 2025

codefromthecrypt commented Apr 24, 2025

codefromthecrypt commented Apr 24, 2025

axw commented Apr 28, 2025

Updates Docker and Kubernetes to use EDOT Collector instead of APM Server #452

Updates Docker and Kubernetes to use EDOT Collector instead of APM Server #452

Conversation

codefromthecrypt commented Apr 23, 2025

codefromthecrypt commented Apr 23, 2025

codefromthecrypt Apr 23, 2025

Choose a reason for hiding this comment

codefromthecrypt commented Apr 23, 2025 • edited Loading

codefromthecrypt commented Apr 23, 2025

felixbarny Apr 23, 2025

Choose a reason for hiding this comment

codefromthecrypt Apr 23, 2025

Choose a reason for hiding this comment

felixbarny Apr 23, 2025

Choose a reason for hiding this comment

codefromthecrypt Apr 23, 2025

Choose a reason for hiding this comment

codefromthecrypt Apr 23, 2025

Choose a reason for hiding this comment

carsonip Apr 25, 2025

Choose a reason for hiding this comment

codefromthecrypt commented Apr 23, 2025 • edited Loading

codefromthecrypt commented Apr 23, 2025

codefromthecrypt commented Apr 24, 2025

codefromthecrypt commented Apr 24, 2025

axw commented Apr 28, 2025

codefromthecrypt commented Apr 23, 2025 •

edited

Loading

codefromthecrypt commented Apr 23, 2025 •

edited

Loading