diff --git a/_data-prepper/pipelines/contains.md b/_data-prepper/pipelines/contains.md index bb65b4c785b..26b8b2aac57 100644 --- a/_data-prepper/pipelines/contains.md +++ b/_data-prepper/pipelines/contains.md @@ -18,7 +18,7 @@ The function returns `true` if the substring specified in the second argument is For example, if you want to check if the string `"abcd"` is contained within the value of a field named `message`, you can use the `contains()` function as follows: ``` -contains('/message', 'abcd') +'contains(/message, "abcd")' ``` {% include copy.html %} @@ -27,7 +27,7 @@ This call returns `true` if the field `message` contains the substring `abcd` or Alternatively, you can use a literal string as the first argument: ``` -contains('This is a test message', 'test') +'contains("This is a test message", "test")' ``` {% include copy.html %} @@ -35,3 +35,81 @@ In this case, the function returns `true` because the substring `test` is presen The `contains()` function performs a case-sensitive search. {: .note} + +## Example + +The following pipeline uses `contains()` to add a boolean flag `has_test` based on a substring in `/message` and to filter out non-matching events, forwarding only messages containing "ERROR" to OpenSearch: + +```yaml +contains-demo-pipeline: + source: + http: + ssl: false + + processor: + - add_entries: + entries: + - key: "has_test" + value_expression: 'contains(/message, "test")' + - drop_events: + drop_when: 'not contains(/message, "ERROR")' + + sink: + - opensearch: + hosts: ["https://opensearch:9200"] + insecure: true + username: admin + password: "admin_pass" + index_type: custom + index: "demo-index-%{yyyy.MM.dd}" +``` +{% include copy.html %} + +You can test the pipeline using the following command: + +```bash +curl -sS -X POST "http://localhost:2021/events" \ + -H "Content-Type: application/json" \ + -d '[ + {"message":"ok hello"}, + {"message":"this has test but ok"}, + {"message":"ERROR: something bad"}, + {"message":"ERROR: unit test failed"} + ]' +``` +{% include copy.html %} + +The documents stored in OpenSearch contain the following information: + +```json +{ + ... + "hits": { + "total": { + "value": 2, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "demo-index-2025.10.21", + "_id": "5YACB5oBqZitdAAb4n3r", + "_score": 1, + "_source": { + "message": "ERROR: something bad", + "has_test": false + } + }, + { + "_index": "demo-index-2025.10.21", + "_id": "5oACB5oBqZitdAAb4n3r", + "_score": 1, + "_source": { + "message": "ERROR: unit test failed", + "has_test": true + } + } + ] + } +} +```