[FR]: Add a managed Kafka module

[Soviut]

The feature request

Add a managed kafka module that can spin up both a Kafka cluster as well as a set of topics.

Proposed solution

A Kafka module would be able to spin up a cluster and a list of topics with as little configuration as possible.

Additional context

Having done it by hand using the google provider, it took me a long time because I had to figure out the convention for cluster paths that are required by the topic module.

[ludoo]

I don't think we have the bandwidth for this, but you are welcome to send a PR.

[Soviut]

@ludoo I could certainly clean up what I've done manually and submit a PR, but it wouldn't necessarily cover all aspects of the module it's wrapping. Is there a way that this can be iterated on as a "beta module"?

[ludoo]

Rather than a module, this might be one of our new blueprints, which reside under fast/project-templates. The idea is to write the project reqs as a YAML that can be either dropped in our project factory, or used as a reference. And use Terraform to populate resources within the project.

[ludoo]

This is a simple example. In your case the project template should also attach to a shared vpc, like we're doing in the template we're preparing for MongoDB.

[juliocc]

@ludoo I believe @Soviut is talking about this, which does look like a good candidate for a module.

[ludoo]

But that is a single resource...

[Soviut]

@ludoo Here's the full example that I worked from. It includes the Kafka cluster resource and the topics resource.

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/managed_kafka_cluster

My variables looked something like this

variable "project_id" {
  type        = string
  description = "The project ID."
}

variable "cluster_id" {
  type        = string
  description = "The cluster ID."
}

variable "location" {
  type        = string
  description = "The location."
  default     = "us-central1"
}

variable "subnet" {
  type        = string
  description = "The subnet used by the cluster."
}

variable "vcpu_count" {
  type        = number
  description = "The number of vCPUs."
  default     = 3
}

variable "memory_bytes" {
  type        = number
  description = "The memory in bytes to allocate to the instance."
  default     = 3221225472 # 3 GiB
}

variable "topics" {
  type = list(object({
    topic_id           = string
    partition_count    = optional(number, 3)
    replication_factor = optional(number, 1)
  }))
  description = "The list of topics."
  default     = []
}

[ludoo]

@Soviut if you're still up for this, could you take a look at how we are implementing project templates in the fast-dev branch?

New style blueprints will conform to that format, and this could be a good candidate. To have an idea on how to use networking, you can check the mongodb atlas template.

[Soviut]

@ludoo I'm not sure I follow everything but I could certainly try to put together a PR that tries to follow what I see.

[ludoo]

If you are ok with starting, then just open the PR and askus questions for anything that is unclear. :)

[franpinedab]

Hi ,

Apologies if I haven’t followed the most appropriate approach, but I’ve prepared a PR to contribute to the managed Kafka. I’m happy to adjust anything needed to better align with the project’s expectations.

#3035

Thanks for the opportunity to contribute!