fix(azure.eventhub): handles properties as string and drops empty fields

[zmoog]

Proposed commit message

WHAT

• Drops empty/null fields
• Renames azure.eventhub.properties as azure.eventhub.properties.raw when azure.eventhub.properties is a string instead of an object.

WHY

Drops empty/null fields

Sometimes Azure services emit log event like the following:

{
  "category": "NetworkSecurityGroupEvent",
  "operationName": "NetworkSecurityGroupEvents",
  "properties": {
    "conditions": {
      "": "",
      "destinationPortRange": "0-65535",
      "sourcePortRange": "0-65535"
    },
    "direction": "Out",
    "macAddress": "00-11-22-33-4444",
    "primaryIPv4Address": "10.0.4.6",
    "primaryIPv6Address": "ace:ace:dead:beef::9",
    "priority": 65000,
    "ruleName": "DefaultRule_AllowVnetOutBound",
    "subnetPrefix": "10.0.4.0/25",
    "type": "allow",
    "vnetResourceGuid": "{a08d316f-3c0a-428d-84ec-2977078852a5}"
  }
}

Elasticsearch cannot index "": "" field inside properties.condition, so we need to clean it up.

Renames azure.eventhub.properties as azure.eventhub.properties.raw when azure.eventhub.properties is a string instead of an object

Sometimes Azure sends azure.eventhub.properties as a string, this field should really be a an object instead, causing mapping errors like:

object mapping for [azure.eventhub.properties] tried to parse field [properties] as object, but found a concrete value

By renaming azure.eventhub.properties as azure.eventhub.properties.raw, we avoid the mapping error and give users the opportunity to handle the original value using a custom pipeline.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

[Kavindu-Dodan]

It should be (((List) object).size() == 0) isn't it ?

[zmoog]

I see it works both ways:

However size() seems like the right approach since both List and Map extend Collection.

[Kavindu-Dodan]

Interesting, I thought painless is Java, but I think .length is an addon by painless.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[Kavindu-Dodan]

LGTM

[muthu-mps]

The sample event here has the properties with fields and the conditions property has empty string and also fields with values. Can we include this format as well to the pipeline tests?

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: e09268e

History

• 💚 Build #30124 succeeded 5c54f75
• 💔 Build #30123 failed 66d2d2f

cc @zmoog

[muthu-mps]

Left a comment on adding additional events to the pipeline tests. Change looks good to me otherwise!

[elastic-sonarqube]

Quality Gate failed

Failed conditions
67.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[chemamartinez]

LGTM! In case it’s useful to know you can also apply subobjects: false to solve the conflict. Done for example at #13925.

[zmoog]

The sample event here has the properties with fields and the conditions property has empty string and also fields with values. Can we include this format as well to the pipeline tests?

Yeah, but this would require to add mappings for these fields not to make test fail, which is something I am trying to avoid in a generic integration like azure.eventhub.

[zmoog]

LGTM! In case it’s useful to know you can also apply subobjects: false to solve the conflict. Done for example at #13925.

Great point, I didn't think about using subobjects: false.

Since this is a generic integration, where users are supposed to customize pipeline and mapping, I'm trying to leave all the options on the table.

[elastic-vault-github-plugin-prod]

Package azure - 1.28.3 containing this change is available at https://epr.elastic.co/package/azure/1.28.3/