From 69d6602697f38f7fbf15570280274799631b8391 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Tue, 21 Oct 2025 10:43:43 +0100 Subject: [PATCH 1/5] adding example to otel sources data prepper Signed-off-by: Anton Rubin --- .../configuration/sources/otel-logs-source.md | 116 ++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md index 4efa7e8b3a8..9025201ce6d 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md @@ -53,6 +53,7 @@ To get started, create a `pipeline.yaml` file and add `otel_logs_source` as the source: - otel_logs_source: ``` +{% include copy.html %} To generate data in the OpenTelemetry format, set the `output_format` setting to `otel`, as shown in the following example: @@ -63,6 +64,121 @@ source: ``` {% include copy.html %} +## Example + +The following pipeline demonstrates Data Prepper receiving OTLP logs over HTTPS using a PEM cert and key with unframed HTTP at a custom path, accepting `gzip` payloads, preserving OTEL-shaped documents, and indexing them into OpenSearch. + +```yaml +otel-logs-otel-output: + source: + otel_logs_source: + port: 21894 + ssl: true + sslKeyFile: "/usr/share/data-prepper/certs/dp-key.pem" + sslKeyCertChainFile: "/usr/share/data-prepper/certs/dp-cert.pem" + unframed_requests: true + path: "/ingest/${pipelineName}/v1/logs" + compression: gzip + output_format: otel + request_timeout: 15000 + health_check_service: true + proto_reflection_service: true + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + index: "otel-logs-otel-output" + username: "admin" + password: "admin_pass" + insecure: true +``` +{% include copy.html %} + +You can test this pipeline using the following commands: + +```bash +cat > /tmp/otel-log3.json <<'JSON' +{ + "resourceLogs": [{ + "resource": {"attributes":[ + {"key":"service.name","value":{"stringValue":"checkout"}}, + {"key":"service.version","value":{"stringValue":"1.4.2"}} + ]}, + "scopeLogs": [{ + "scope": {"name":"manual-gzip"}, + "logRecords": [{ + "timeUnixNano": "1739999999000000000", + "severityText": "ERROR", + "body": {"stringValue":"payment gateway timeout"}, + "attributes":[ + {"key":"region","value":{"stringValue":"eu-west-1"}}, + {"key":"latency_ms","value":{"doubleValue":1234.5}} + ] + }] + }] + }] +} +JSON + +gzip -c /tmp/otel-log3.json > /tmp/otel-log3.json.gz + +curl -s -X POST "https://localhost:21894/ingest/otel-logs-otel-output/v1/logs" \ + -H 'Content-Type: application/json' \ + -H 'Content-Encoding: gzip' \ + --insecure \ + --data-binary @/tmp/otel-log3.json.gz +``` +{% include copy.html %} + +The document stored in OpenSearch contains the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "otel-logs-otel-output", + "_id": "l24NBpoBk9xzgdmMXFDm", + "_score": 1, + "_source": { + "traceId": "", + "instrumentationScope": { + "name": "manual-gzip", + "droppedAttributesCount": 0 + }, + "resource": { + "attributes": { + "service.name": "checkout", + "service.version": "1.4.2" + }, + "schemaUrl": "", + "droppedAttributesCount": 0 + }, + "flags": 0, + "severityNumber": 0, + "body": "payment gateway timeout", + "schemaUrl": "", + "spanId": "", + "severityText": "ERROR", + "attributes": { + "region": "eu-west-1", + "latency_ms": 1234.5 + }, + "time": "2025-02-19T21:19:59Z", + "droppedAttributesCount": 0, + "observedTimestamp": "1970-01-01T00:00:00Z" + } + } + ] + } +} +``` + ## Metrics You can use the following metrics with the `otel_logs_source` source. From 9c5ac06b5874c20e67806e1841486cbadd7379ea Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Tue, 21 Oct 2025 10:46:37 +0100 Subject: [PATCH 2/5] removing - from examples Signed-off-by: Anton Rubin --- .../pipelines/configuration/sources/otel-logs-source.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md index 9025201ce6d..ca4444f4e7f 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md @@ -51,7 +51,7 @@ To get started, create a `pipeline.yaml` file and add `otel_logs_source` as the ```yaml source: - - otel_logs_source: + otel_logs_source: ``` {% include copy.html %} @@ -59,8 +59,8 @@ To generate data in the OpenTelemetry format, set the `output_format` setting to ```yaml source: - - otel_logs_source: - output_format: otel + otel_logs_source: + output_format: otel ``` {% include copy.html %} From 6a1bb88d7932d8acb63388cd1976f70b5f731bd3 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Tue, 21 Oct 2025 11:07:41 +0100 Subject: [PATCH 3/5] removing custom port Signed-off-by: Anton Rubin --- .../pipelines/configuration/sources/otel-logs-source.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md index ca4444f4e7f..01f58a98ef7 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md @@ -72,7 +72,6 @@ The following pipeline demonstrates Data Prepper receiving OTLP logs over HTTPS otel-logs-otel-output: source: otel_logs_source: - port: 21894 ssl: true sslKeyFile: "/usr/share/data-prepper/certs/dp-key.pem" sslKeyCertChainFile: "/usr/share/data-prepper/certs/dp-cert.pem" @@ -121,7 +120,7 @@ JSON gzip -c /tmp/otel-log3.json > /tmp/otel-log3.json.gz -curl -s -X POST "https://localhost:21894/ingest/otel-logs-otel-output/v1/logs" \ +curl -s -X POST "https://localhost:21892/ingest/otel-logs-otel-output/v1/logs" \ -H 'Content-Type: application/json' \ -H 'Content-Encoding: gzip' \ --insecure \ From affe1577f66851466c3629799ba23df336ca5081 Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Tue, 21 Oct 2025 11:19:35 +0100 Subject: [PATCH 4/5] fixing vale errors Signed-off-by: Anton Rubin --- .../pipelines/configuration/sources/otel-logs-source.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md index 01f58a98ef7..61cc2883745 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md @@ -66,7 +66,7 @@ source: ## Example -The following pipeline demonstrates Data Prepper receiving OTLP logs over HTTPS using a PEM cert and key with unframed HTTP at a custom path, accepting `gzip` payloads, preserving OTEL-shaped documents, and indexing them into OpenSearch. +The following pipeline demonstrates Data Prepper receiving OTLP logs over HTTPS using a PEM cert and key with unframed HTTP at a custom path, accepting `gzip` payloads, preserving OTel-shaped documents, and indexing them into OpenSearch. ```yaml otel-logs-otel-output: From 2d0fb080397c88429591f0512ddd7d497ebd049e Mon Sep 17 00:00:00 2001 From: AntonEliatra Date: Tue, 21 Oct 2025 11:22:32 +0100 Subject: [PATCH 5/5] Update otel-logs-source.md Signed-off-by: AntonEliatra --- .../pipelines/configuration/sources/otel-logs-source.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md index 61cc2883745..9c82d445fb5 100644 --- a/_data-prepper/pipelines/configuration/sources/otel-logs-source.md +++ b/_data-prepper/pipelines/configuration/sources/otel-logs-source.md @@ -66,7 +66,7 @@ source: ## Example -The following pipeline demonstrates Data Prepper receiving OTLP logs over HTTPS using a PEM cert and key with unframed HTTP at a custom path, accepting `gzip` payloads, preserving OTel-shaped documents, and indexing them into OpenSearch. +The following pipeline demonstrates Data Prepper receiving OTLP logs over HTTPS using a PEM cert and key with unframed HTTP at a custom path, accepting `gzip` payloads, preserving OTel shaped documents, and indexing them into OpenSearch: ```yaml otel-logs-otel-output: