kafka with SCRAM-512 users

[lzwaan]

Is it possible to use SCRAM-512 to connect to kafka?

[robambalu]

I'm not familiar with that, what is scram and how would it apply to the rdkafka based kafka adapter

[lzwaan]

Kafka SASL/SCRAM authentication

[timkpaine]

this should work via the csp conda package only, and on the csp pip package for Mac/linux only after #352 is complete

[lzwaan]

Hi Tim, do you have an example to use conda/librdkafka in combination with scram512 auth?

[lzwaan]

we don't use kerberos anymore on our kafka cluster

[timkpaine]

we don't authenticate to kafka at the csp level, we hand off to librdkafka. As of 0.0.9 we compile librdkafka with ssl, kerberos, and cyrus-sasl support, so whatever librdkafka supports under those parameters is supported by csp's kafka adapter.

[lzwaan]

I can confirm that it is working via the rd_kafka_conf_options, now I need to find a way to combine:

1. distribute over multiple servers
2. build up state after restarting the application but not get duplicates in the sink topic
3. use offsets and group.id for kafka acl authentication