Springboot 3.0

README.md

@@ -3,6 +3,9 @@
 
 This repository has the complete code related to kafka producers/consumers using spring boot.
 
+## Kafka SetUp
+- [Setup-Kafka-Using-Docker](SetUpKafkaDocker.md)
+
 
 
 - [Setup-Kafka](https://github.com/dilipsundarraj1/kafka-for-developers-using-spring-boot/blob/master/SetUpKafka.md)

SetUpKafkaDocker.md

@@ -0,0 +1,231 @@
+# Set Up Kafka in Local using Docker
+
+## Set up broker and zookeeper
+
+- Navigate to the path where the **docker-compose.yml** is located and then run the below command.
+
+```
+docker-compose up
+```
+
+## Producer and Consume the Messages
+
+- Let's going to the container by running the below command.
+
+```
+docker exec -it kafka1 bash
+```
+
+- Create a Kafka topic using the **kafka-topics** command.
+  - **kafka1:19092** refers to the **KAFKA_ADVERTISED_LISTENERS** in the docker-compose.yml file.
+
+```
+kafka-topics --bootstrap-server kafka1:19092 \
+             --create \
+             --topic test-topic \
+             --replication-factor 1 --partitions 1
+```
+
+- Produce Messages to the topic.
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-console-producer --bootstrap-server kafka1:19092 \
+                       --topic test-topic
+```
+
+- Consume Messages from the topic.
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-console-consumer --bootstrap-server kafka1:19092 \
+                       --topic test-topic \
+                       --from-beginning
+```
+
+## Producer and Consume the Messages With Key and Value
+
+- Produce Messages with Key and Value to the topic.
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-console-producer --bootstrap-server kafka1:19092 \
+                       --topic test-topic \
+                       --property "key.separator=-" --property "parse.key=true"
+```
+
+- Consuming messages with Key and Value from a topic.
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-console-consumer --bootstrap-server kafka1:19092 \
+                       --topic test-topic \
+                       --from-beginning \
+                       --property "key.separator= - " --property "print.key=true"
+```
+
+### Consume Messages using Consumer Groups
+
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-console-consumer --bootstrap-server kafka1:19092 \
+                       --topic test-topic --group console-consumer-41911\
+                       --property "key.separator= - " --property "print.key=true"
+```
+
+- Example Messages:
+
+```
+a-abc
+b-bus
+```
+
+### Set up a Kafka Cluster with 3 brokers
+
+- Run this command and this will spin up a kafka cluster with 3 brokers.
+
+```
+docker-compose -f docker-compose-multi-broker.yml up
+```
+
+- Create topic with the replication factor as 3
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-topics --bootstrap-server kafka1:19092 \
+             --create \
+             --topic test-topic \
+             --replication-factor 3 --partitions 3
+```
+
+- Produce Messages to the topic.
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-console-producer --bootstrap-server localhost:9092,kafka2:19093,kafka3:19094 \
+                       --topic test-topic
+```
+
+- Consume Messages from the topic.
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-console-consumer --bootstrap-server localhost:9092,kafka2:19093,kafka3:19094 \
+                       --topic test-topic \
+                       --from-beginning
+```
+#### Log files in Multi Kafka Cluster
+
+- Log files will be created for each partition in each of the broker instance of the Kafka cluster.
+ -  Login to the container **kafka1**.
+  ```
+  docker exec -it kafka1 bash
+  ```
+ -  Login to the container **kafka2**.
+  ```
+  docker exec -it kafka2 bash
+  ```
+
+- Shutdown the kafka cluster
+
+```
+docker-compose -f docker-compose-multi-broker.yml down
+```
+
+### Setting up min.insync.replica
+
+- Topic - test-topic
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-configs  --bootstrap-server localhost:9092 --entity-type topics --entity-name test-topic \
+--alter --add-config min.insync.replicas=2
+```
+
+- Topic - library-events
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-configs  --bootstrap-server localhost:9092 --entity-type topics --entity-name library-events \
+--alter --add-config min.insync.replicas=2
+```
+## Advanced Kafka Commands
+
+### List the topics in a cluster
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-topics --bootstrap-server kafka1:19092 --list
+
+```
+
+### Describe topic
+
+- Command to describe all the Kafka topics.
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-topics --bootstrap-server kafka1:19092 --describe
+```
+
+- Command to describe a specific Kafka topic.
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-topics --bootstrap-server kafka1:19092 --describe \
+--topic test-topic
+```
+
+### Alter topic Partitions
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-topics --bootstrap-server kafka1:19092 \
+--alter --topic test-topic --partitions 40
+```
+
+### How to view consumer groups
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-consumer-groups --bootstrap-server kafka1:19092 --list
+```
+
+#### Consumer Groups and their Offset
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-consumer-groups --bootstrap-server kafka1:19092 \
+--describe --group console-consumer-41911
+```
+
+## Log file and related config
+
+- Log into the container.
+
+```
+docker exec -it kafka1 bash
+```
+
+- The config file is present in the below path.
+
+```
+/etc/kafka/server.properties
+```
+
+- The log file is present in the below path.
+
+```
+/var/lib/kafka/data/
+```
+
+### How to view the commit log?
+
+```
+docker exec --interactive --tty kafka1  \
+kafka-run-class kafka.tools.DumpLogSegments \
+--deep-iteration \
+--files /var/lib/kafka/data/test-topic-0/00000000000000000000.log
+
+```

Undersrading-Kafka-DockerCompose.md

@@ -0,0 +1,32 @@
+# Understanding Kafka Docker Compose works
+
+- More info is available in this link - https://rmoff.net/2018/08/02/kafka-listeners-explained/
+
+## Kafka broker docker-compose config:
+
+```
+kafka1:
+  image: confluentinc/cp-kafka:7.3.2
+  hostname: kafka1
+  container_name: kafka1
+  ports:
+    - "9092:9092"
+    - "29092:29092"
+  environment:
+    KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,DOCKER://host.docker.internal:29092
+    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
+    KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+    KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
+    KAFKA_BROKER_ID: 1
+    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+    KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+    KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+  depends_on:
+    - zoo1
+```
+- KAFKA_INTER_BROKER_LISTENER_NAME
+  - Kafka brokers communicate between themselves, usually on the internal network. This is where the **KAFKA_INTER_BROKER_LISTENER_NAME** property comes in handy.
+- KAFKA_ADVERTISED_LISTENERS
+  - The config that's present in this property is the data that's shared to the clients when they are connected.
+  - Kafka clients may not be in the network where the kafka broker is running.
+    - For a broker that's running in the docker network, the client is very much likely possible outside the docker network.

docker-compose-multi-broker.yml

@@ -0,0 +1,70 @@
+version: '2.1'
+
+services:
+  zoo1:
+    image: confluentinc/cp-zookeeper:7.3.2
+    platform: linux/amd64
+    hostname: zoo1
+    container_name: zoo1
+    ports:
+      - "2181:2181"
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_SERVER_ID: 1
+      ZOOKEEPER_SERVERS: zoo1:2888:3888
+
+
+  kafka1:
+    image: confluentinc/cp-kafka:7.3.2
+    platform: linux/amd64
+    hostname: kafka1
+    container_name: kafka1
+    ports:
+      - "9092:9092"
+      - "29092:29092"
+    environment:
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,DOCKER://host.docker.internal:29092
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
+      KAFKA_BROKER_ID: 1
+      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
+    depends_on:
+      - zoo1
+
+  kafka2:
+    image: confluentinc/cp-kafka:7.3.2
+    platform: linux/amd64
+    hostname: kafka2
+    container_name: kafka2
+    ports:
+      - "9093:9093"
+      - "29093:29093"
+    environment:
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:19093,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093,DOCKER://host.docker.internal:29093
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
+      KAFKA_BROKER_ID: 2
+      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
+    depends_on:
+      - zoo1
+
+
+  kafka3:
+    image: confluentinc/cp-kafka:7.3.2
+    platform: linux/amd64
+    hostname: kafka3
+    container_name: kafka3
+    ports:
+      - "9094:9094"
+      - "29094:29094"
+    environment:
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:19094,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094,DOCKER://host.docker.internal:29094
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
+      KAFKA_BROKER_ID: 3
+      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
+    depends_on:
+      - zoo1

docker-compose.yml

@@ -0,0 +1,33 @@
+version: '2.1'
+
+services:
+  zoo1:
+    image: confluentinc/cp-zookeeper:7.3.2
+    hostname: zoo1
+    container_name: zoo1
+    ports:
+      - "2181:2181"
+    environment:
+      ZOOKEEPER_CLIENT_PORT: 2181
+      ZOOKEEPER_SERVER_ID: 1
+      ZOOKEEPER_SERVERS: zoo1:2888:3888
+
+
+  kafka1:
+    image: confluentinc/cp-kafka:7.3.2
+    hostname: kafka1
+    container_name: kafka1
+    ports:
+      - "9092:9092"
+      - "29092:29092"
+    environment:
+      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,DOCKER://host.docker.internal:29092
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
+      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
+      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
+      KAFKA_BROKER_ID: 1
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+    depends_on:
+      - zoo1

library-events-consumer/build.gradle

@@ -1,12 +1,12 @@
 plugins {
-	id 'org.springframework.boot' version '2.6.5'
-	id 'io.spring.dependency-management' version '1.0.11.RELEASE'
+	id 'org.springframework.boot' version '3.0.5'
+	id 'io.spring.dependency-management' version '1.1.0'
 	id 'java'
 }
 
 group = 'com.learnkafka'
 version = '0.0.1-SNAPSHOT'
-sourceCompatibility = '11'
+sourceCompatibility = '17'
 
 configurations {
 	compileOnly {