Avro schema upload with pulsar admin being rejected

[estebanz01]

hola! 👋

I've been trying to upload an schema with pulsar admin CLI tool but I'm getting hit by the following error:

2023-11-10T08:49:06,329-0500 [AsyncHttpClient-7-1] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:8080/admin/v2/schemas/public/iot/device_1/schema] Failed to perform http post request: javax.ws.rs.ClientErrorException: HTTP 422 {
"servlet":"org.glassfish.jersey.servlet.ServletContainer-43cb5f38",
"message":"Invalid schema definition data for AVRO schema",
"url":"/admin/v2/schemas/public/iot/device_1/schema",
"status":"422"
}

Any ideas on why it does not work? here's the schema

{
  "name": "data-schema",
  "type": "AVRO",
  "schema": "{\"type\": \"record\", \"namespace\": \"iot\",\"name\":\"exampleAvro\",\"fields\":[{\"name\": \"device_name\",\"type\": \"string\"},{\"name\": \"data_content\", \"type\": \"map\", \"values\": \"string\"},{\"name\": \"device_properties\", \"type\": \"map\", \"values\": \"string\"},{\"name\":\"datetime_info\", \"type\": \"string\"}, {\"name\": \"topicName\", \"type\": \"string\"}]}",
  "properties": {
    "__jsr310ConversionEnabled": "false"
  }
}

and here's the schema in a pretty format:

{
  "type": "record",
  "namespace": "iot",
  "name": "exampleAvro",
  "fields": [
    {
      "name": "device_name",
      "type": "string"
    },
    {
      "name": "data_content",
      "type": "map",
      "values": "string"
    },
    {
      "name": "device_properties",
      "type": "map",
      "values": "string"
    },
    {
      "name": "datetime_info",
      "type": "string"
    },
    {
      "name": "topicName",
      "type": "string"
    }
  ]
}

[estebanz01]

the upload works if I change the schema type to from AVRO to JSON but then my sink (ClickHouse JDBC) will try to perform an Avro schema reading, so it doesn't work.

[jak78]

Hello @estebanz01! Which version of Pulsar are you using?

[estebanz01]

hi! I'm on 3.1.0

[estebanz01]

OK, I found the issue by doing curl requests to the following endpoint: https://pulsar.apache.org/admin-rest-api/?version=3.1.1#operation/SchemasResource_testCompatibility

The response gave me the full error trace so I was able to see what was happening. Basically, the complex types need to be defined as json objects too, so maps will look like this:

{"name": "device_content", "type": {"type": "map", "values": "string", "default": {}}}

It took too long to figure it out 😄

[estebanz01]

curl -X POST http://localhost:8080/admin/v2/schemas/public/iot/device_1/compatibility \
  --header "Content-Type: application/json;charset=utf-8" -d @connectors/avro-schema-new.json

[jak78]

@estebanz01 I'm delighted you found a solution! I recognize the doc could be improved here. I'm on it.

[jak78]

#21561