Skip to content
@kafka-practice

kafka-practice

MSA ํ™˜๊ฒฝ ๋„์ „ + ์ ์‘๊ธฐ - MSA with Kafka


๊ธฐ์กด์˜ ํ”„๋กœ์ ํŠธ์—์„œ ์ €๋Š” DB๋ฅผ ํ•˜๋‚˜๋กœ ์‚ฌ์šฉํ•œ MSA ํ™˜๊ฒฝ์„ ๊ตฌ์„ฑํ–ˆ์—ˆ์Šต๋‹ˆ๋‹ค.
ํ•˜์ง€๋งŒ โœจํ•˜๋‚˜์˜ DB๋กœ ๋ชจ๋“  ์„œ๋น„์Šค๋ฅผ ์—ฐ๋™โœจ ์‹œ, ๊ด€๋ฆฌ๊ฐ€ ํŽธํ•˜๋‹ค๋Š” ์žฅ์ ์ด ์žˆ์ง€๋งŒ
โœจ๋™์‹œ์„ฑ ๋ฌธ์ œโœจ ๋“ฑ์˜ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒ ๊ฐ€๋Šฅํ•˜๋‹ค๋Š” ๊ฒƒ์„ ์•Œ๊ฒŒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋ ‡๋‹ค๋ฉด, ์–ด๋–ป๊ฒŒ ์‹ค๋ฌด์—์„œ๋Š” ๋‹ค์ค‘ DB๋ฅผ ๊ตฌํ˜„ํ• ๊นŒ๋ผ๋Š” ์˜๋ฌธ์ ์ด ์ƒ๊ฒผ์Šต๋‹ˆ๋‹ค.
์ดํ›„ Kafka์— ๋Œ€ํ•ด ์•Œ๊ฒŒ๋˜์—ˆ๊ณ , โœจKafka๋ฅผ ํ†ตํ•ด MSA DB ๋™๊ธฐํ™”๋ฅผ ์ˆ˜ํ–‰ํ•œ๋‹ค๋Š” ๊ฒƒโœจ์„ ์•Œ๊ฒŒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
Kubernetes ํ™˜๊ฒฝ์—์„œ โœจKafka with SASL_TLSโœจ๋Š” ๊ตฌํ˜„ํ•ด๋ณธ ์ƒํƒœ์ด๋ฉฐ
์‹ค์ œ ํ”„๋กœ์ ํŠธ์— ์ ์šฉํ•ด๋ณด๊ธฐ ์ „, โœจ๊ฐ„์ด MSA ํ™˜๊ฒฝ with ๋…๋ฆฝ์  DBโœจ๋ฅผ ๊ตฌ์„ฑํ•ด๋ณด๋ ค ํ•ฉ๋‹ˆ๋‹ค.

โœจ ๋ชฉํ‘œ

  • Github Actions + ArgoCD๋ฅผ ํ™œ์šฉํ•˜์—ฌ CI/CD๋ฅผ ๊ตฌ์„ฑ
  • GCP Kubenetes ํ™˜๊ฒฝ์„ Terraform์„ ์ด์šฉํ•˜์—ฌ ์ˆ˜ํ–‰
  • ๊ฐ MSA ํ”„๋กœ์ ํŠธ๋ฅผ ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ์ƒ์— ๋ฐฐํฌ
  • Spring Cloud๋ฅผ ์ ์šฉํ•˜์—ฌ API Gateway ์—ญํ•  ์ˆ˜ํ–‰ ์„œ๋ฒ„ ์ƒ์„ฑ
  • Kafka๋ฅผ SASL_TLS ์ ์šฉํ•˜์—ฌ ์ฟ ๋ฒ„๋„คํ‹ฐ์Šค ์ƒ์—์„œ ๊ด€๋ฆฌ
  • kafka-user -> kafk-post ์„œ๋ฒ„๋กœ DB ๋ฐ์ดํ„ฐ ๋™๊ธฐํ™” - Transactional Outbox Pattern
  • gRPC๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ MSA ๊ฐ„ API ์š”์ฒญ
  • istio๋ฅผ ์ ์šฉํ•˜์—ฌ ์• ํ”Œ๋ฆฌ์ผ€์ด์…˜ ๊ฐ„ mTLS ํ™œ์„ฑํ™”
  • OpenTelemetry + Jaeger๋ฅผ ์ด์šฉํ•˜์—ฌ Trace ์ •๋ณด ์ˆ˜์ง‘ ๋ฐ ์‹œ๊ฐํ™”

๐ŸŒฒ ํ”„๋กœ์ ํŠธ ์•„ํ‚คํ…์ณ ๊ตฌ์กฐ - ํ”„๋กœ์ ํŠธ ๊ตฌํ˜„ ์™„๋ฃŒ ํ›„
แ„Œแ…ฆแ„†แ…ฉแ†จ แ„‹แ…ฅแ†นแ„‚แ…ณแ†ซ แ„ƒแ…กแ„‹แ…ตแ„‹แ…ฅแ„€แ…ณแ„…แ…ขแ†ท drawio ๊ฐ ๋ฆฌ์†Œ์Šค ํ๋ฆ„ ๋ณต์Šต์„ ์œ„ํ•ด ๋„ค์ž„์ŠคํŽ˜์ด์Šค๋ฅผ ์ค‘์‹ฌ์œผ๋กœ ์•„ํ‚คํ…์ณ๋ฅผ ๊ทธ๋ ธ์Šต๋‹ˆ๋‹ค.

๐Ÿ’พ ์†Œ์Šค ์ฝ”๋“œ ๊ตฌ์กฐ

  • kafka-user - ๋ฉ”์‹œ์ง€ ํ”„๋กœ๋“€์‹ฑ, gRPC ์„œ๋ฒ„
  • kafka-post - ๋ฉ”์‹œ์ง€ ์ปจ์Šˆ๋ฐ, gRPC ํด๋ผ์ด์–ธํŠธ
  • kafka-manifest - private, gitops

๐Ÿ–ฅ๏ธ Reference

  1. Transactional Outbox Pattern with Microservices and Kafka - CuriousJinan ๋ธ”๋กœ๊ทธ
  2. AmazingEffect - Github
  3. ๊ณผ์ •์„ ์ ์€ Notion



ํ›„๊ธฐ

์ด ํ”„๋กœ์ ํŠธ๋ฅผ ์ˆ˜ํ–‰ํ•˜๊ธฐ ์ „, ์ €๋Š” ๋‹จ์ˆœํžˆ Kafka๋กœ MSA๋ฅผ ๊ตฌํ˜„ํ•œ๋‹ค๊ณ ๋งŒ ์•Œ๊ณ  ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.
์ดํ›„ ์–ด๋–ป๊ฒŒ ๊ตฌํ˜„ํ•˜์ง€? ๋ผ๋Š” ์ƒ๊ฐ์— ๋ฌด์ž‘์ • ๋ธ”๋กœ๊ทธ๋ฅผ ์ฐพ์•„๋ณด์•˜๊ณ , Transactional Outbox Pattern์— ๋Œ€ํ•ด ์•Œ๊ฒŒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
ํ•ด๋‹น ๋ธ”๋กœ๊ทธ ๋‚ด์šฉ์„ ๊ทผ๊ฑฐ๋กœ, ์‹œ๋„๋ฅผ ํ•˜๋ คํ–ˆ์ง€๋งŒ ๋„ˆ๋ฌด ์–ด๋ ต๋‹ค ์‹ถ์—ˆ๊ณ  ํ•ด๋‹น ๋ธ”๋กœ๊ทธ๋Š” ์ฝ”๋“œ๋กœ๋œ ์ •๋ณด๊ฐ€ ๊ฑฐ์˜ ์—†์—ˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋ž˜์„œ ์ €๋Š” ๋ฌด์ž‘์ • ๊นƒํ—ˆ๋ธŒ์— Transactional Outbox Pattern์„ ๊ฒ€์ƒ‰ํ–ˆ๊ณ , ์œ„์˜ AmazingEffect ๋ ˆํฌ์ง€ํ† ๋ฆฌ๋ฅผ ๋ฐœ๊ฒฌํ–ˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋•์— ํ”„๋กœ์ ํŠธ๋ฅผ ์‹œ์ž‘ํ•  ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.
์ƒ๊ฐ๋ณด๋‹ค ๋งค์šฐ ์–ด๋ ค์› ์Šต๋‹ˆ๋‹ค. ์ฒ˜์Œ ๋ณด๋Š” ์ฝ”๋“œ๋“ค, ๋„ˆ๋ฌด๋‚˜ ๋งŽ์€ ํด๋ž˜์Šค๋“ค์ด ์žˆ์—ˆ์ง€๋งŒ โœจํ•˜๋‚˜ํ•˜๋‚˜ ๋ถ„์„โœจํ•˜๋ฉด์„œ โœจํ๋ฆ„์„ ํŒŒ์•…โœจํ•˜์˜€๊ณ ,
์ฝ”๋“œ๋ฅผ ์ž‘์„ฑํ•  ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.

์ฝ”๋“œ ์ž‘์„ฑ์„ ํ•˜๋ฉฐ AOP in Logging, ์ƒˆ๋กœ์šด ํŒจํ‚ค์ง€ ์•„ํ‚คํ…์ณ, Kafka in Spring, Trace ๋“ฑ์— ๋Œ€ํ•ด ์•Œ๊ฒŒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
ํŠนํžˆ๋‚˜, AOP๋Š” โœจTrace ๋ฐ Span์„ ์œ„ํ•ด ํ•„์ˆ˜์ โœจ์ด๋ฉฐ โœจ์–ด๋…ธํ…Œ์ด์…˜ ๊ธฐ๋ฐ˜ ๋กœ๊น…์„ ๊ตฌํ˜„ํ•˜๋Š” ๊ฒƒโœจ์€ ๋งค์šฐ ์žฌ๋ฏธ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค.

๋‹ค์Œ์œผ๋กœ๋Š” โœจDevOps Engineer๋กœ์จ ์—ญ๋Ÿ‰์„ ๋†’์ผ ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹คโœจ.
๋ฉฐ์น  ์ „๊นŒ์ง€๋งŒ ํ•ด๋„, ์ €๋Š” Jaeger ๋ฐ OpenTelemetry๊ฐ€ ๋ฌด์—‡์ธ์ง€ ์•Œ์ง€ ๋ชปํ–ˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋Ÿฌ๋‚˜, ๋‚ด ์ฝ”๋“œ์™€ AmazingEffect ์ฝ”๋“œ๋ฅผ ๋น„๊ตํ•˜๋ฉด์„œ ๋‹ค๋ฅธ ์ ๋“ค์„ ๋ถ„์„ํ•˜๋ฉฐ ํ•ด๋‹น ์˜คํ”ˆ์†Œ์Šค๋“ค์˜ ์กด์žฌ๋ฅผ ์•Œ๊ฒŒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.
๊ทธ๋ฆฌ๊ณ  ์ธํ„ฐ๋„ท์„ ์ฐพ์•„๋ณด๋ฉฐ ๊ณต๋ถ€ํ•˜๊ณ  ๊ตฌ์ถ•ํ•˜์—ฌ ์–ด๋–ค ํ๋ฆ„์ธ์ง€ ํŒŒ์•…ํ•˜๋ฉฐ DevOps ์ง€์‹์„ ์–ป์„ ์ˆ˜ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค..
ํŠนํžˆ, โœจSpring -> OpenTelemetry -> Jaeger ๋กœ๊น… ์‹œ์Šคํ…œ ๊ตฌ์ถ•โœจ์€ ๋งค์šฐ ์–ด๋ ค์› ์ง€๋งŒ, ์„ฑ๊ณต ํ›„์—๋Š” ๋งค์šฐ ๋ฟŒ๋“ฏํ•˜๊ณ  ์‹ ๊ธฐํ–ˆ์Šต๋‹ˆ๋‹ค.

ํ”„๋กœ์ ํŠธ๋ฅผ ์ง„ํ–‰ํ•˜๋ฉด์„œ ํ”„๋กœ์ ํŠธ ์ง„ํ–‰ ์ƒํ™ฉ ๋ฐ ๊ตฌ์ถ• ๊ณผ์ •๋“ค์€ ๋น ์ง€์ง€ ์•Š๊ณ  ํ•ด๋†“์•˜์Šต๋‹ˆ๋‹ค.
์ €๋Š” ์ด๋Ÿฌํ•œ ํ”„๋กœ์ ํŠธ ๊ฒฝํ—˜์„ ๋ฐ”ํƒ•์œผ๋กœ, ์ดํ›„ ๊ฐœ๋ฐœ ํŒ€ํ”„๋กœ์ ํŠธ์—์„œ โœจ๋” ์ž์‹ ๊ฐ์žˆ๊ฒŒ DevOps๋ฅผ ๋‹ด๋‹นโœจํ•˜๋ ค๊ณ  ํ•ฉ๋‹ˆ๋‹ค.
์ด๋ ‡๊ฒŒ ์„ฑ์žฅํ•  ์ˆ˜ ์žˆ๋Š” ๊ธฐํšŒ๋ฅผ ์ค€ GCP์™€ AmazingEffect ๋ ˆํฌ์ง€ํ† ๋ฆฌ ์ฃผ์ธ๋ถ„์—๊ฒŒ ์ •๋ง ์ •๋ง ๊ฐ์‚ฌ๋“œ๋ฆฝ๋‹ˆ๋‹ค.


์‚ฌ์ง„

์ด๋ฆ„ ์‚ฌ์ง„
ArgoCD image (3)
Jaeger image (5)
Outbox DB image (6)

Pinned Loading

  1. kafka-user kafka-user Public

    ์‚ฌ์šฉ์ž ๊ด€๋ จ Repo

    Java

  2. kafka-post kafka-post Public

    ํฌ์ŠคํŠธ ๊ด€๋ จ Repo

    Java

Repositories

Showing 3 of 3 repositories
  • .github Public

    MSA ํ™˜๊ฒฝ์—์„œ์˜ Kafka๋ฅผ ์ด์šฉํ•œ DB ๋ฐ์ดํ„ฐ ๋™๊ธฐํ™”

    kafka-practice/.githubโ€™s past year of commit activity
    0 0 0 0 Updated Apr 17, 2025
  • kafka-user Public

    ์‚ฌ์šฉ์ž ๊ด€๋ จ Repo

    kafka-practice/kafka-userโ€™s past year of commit activity
    Java 0 0 0 0 Updated Jan 4, 2025
  • kafka-post Public

    ํฌ์ŠคํŠธ ๊ด€๋ จ Repo

    kafka-practice/kafka-postโ€™s past year of commit activity
    Java 0 0 0 0 Updated Jan 2, 2025

People

This organization has no public members. You must be a member to see who’s a part of this organization.

Top languages

Loadingโ€ฆ

Most used topics

Loadingโ€ฆ