restore stateful application

[lzwaan]

When using kafka as a source, after restarting a server you can re-read from a topic to build up the states.
But how can you seamlessly go from historical to realtime data without manually change the settings and without duplicates towards a sink?

[robambalu]

In the kafka adapter if you pass start_offset = KafkaStartOffset.START_TIME it will play back all data from start time ( which can be in the past ) and the seamlessly switch to realtime mode if your endtime extends into the future

https://github.com/Point72/csp/wiki/Input-Output-Adapters-API#api

[lzwaan]

when using start-offset, you can't make use of the kafka consumer groups which let you not authenticate via kafka acls
"start_offset is not supported when consuming with group_id"
Without group id, you can't make use of consumer groups and not able to distribute the application on multiple servers

[robambalu]

when using start-offset, you can't make use of the kafka consumer groups which let you not authenticate via kafka acls "start_offset is not supported when consuming with group_id" Without group id, you can't make use of consumer groups and not able to distribute the application on multiple servers

group_id implies there can be multiple consumers on the same topic. If they have the same group id kafka will round robin events across them, so they would get partial replays of the full stream. The idea is to only allow historical replay if you will be replaying all of the data rather than a random subset

[lzwaan]

But thats means you end up with duplicates on the sink side?

[robambalu]

I see what you mean, you would have to ensure you dont republish on playback yes

[lzwaan]

Is there an automatic way to start from the last kafka offset before it for example crashed without a group.id?

[robambalu]

Sorry I dont understand the question.
To start from latest offset use KafkaOffset.LATEST ( which is the default when you dont pass a group id )