我搜索并寻找一种能够使用虚荣DNS名称而不是AWS生成的MSK代理DNS名称来引导Kafka客户端的方法。

我和我的团队在拼凑了不同来源的信息后，终于想出了一个解决方案。这是我们的解决方案。

对于每个MSK经纪人：

• 在同一个VPC中创建一个网络负载均衡器（NLB）（对我们来说，这是在公共子网中）
• 创建一个TLS型负载均衡器（LB）侦听器，并在主题替代名称列表中附加一个带有自定义虚拟域名的TLS证书。
• 为虚域创建TLS证书
• 创建一个TLS型负载均衡器（LB）目标组（TG），该目标组通过IP地址定位MSK代理

MSK经纪人IP地址是他们DNS的记录

这是地形配置。我希望它有帮助：

locals {
  msk_scram_addrs = split(",", aws_msk_cluster.myclust.bootstrap_brokers_sasl_scram)
  msk_scram_hosts = toset([for x in local.msk_scram_addrs : split(":", x)[0]])
  # b-1, b-2, b-3, ...
  broker_short_names = toset([
    for a in data.dns_a_record_set.myclust_scram_brokers :
    split(".", a.host)[0]
  ])
}

data "dns_a_record_set" "myclust_scram_brokers" {
  for_each = local.msk_scram_hosts

  host = each.value

  depends_on = [
    aws_msk_cluster.myclust
  ]
}

resource "aws_lb" "msk_broker" {
  for_each = local.broker_short_names

  name               = each.value
  tags               = [...]
  internal           = false
  subnets            = local.aws_vpc_public_subnets
  load_balancer_type = "network"
  idle_timeout       = 3600

  timeouts {
    create = "20m"
  }
}

resource "aws_lb_target_group" "msk_broker_scram_public" {
  for_each = local.broker_short_names

  name = each.value
  tags = [...]

  # https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/
  protocol    = "TLS"
  vpc_id      = local.aws_vpc_id
  target_type = "ip" // Targets Kafka broker by IP address.
  port        = 9196

  lifecycle {
    create_before_destroy = true
  }

  depends_on = [
    aws_lb.msk_broker
  ]
}

resource "aws_lb_target_group_attachment" "msk_broker_scram_public" {
  for_each = {
    for a in data.dns_a_record_set.myclust_scram_brokers : split(".", a.host)[0] => a.addrs[0]
  }

  target_group_arn = aws_lb_target_group.msk_broker_scram_public[each.key].arn
  target_id        = each.value // This is the Kafka broker IP address.
}

resource "aws_lb_listener" "msk_bootstrap_public" {
  for_each = local.broker_short_names

  # https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/
  protocol          = "TLS"
  port              = 9196
  certificate_arn   = module.msk_lb_cert[each.value].acm_cert.arn
  load_balancer_arn = aws_lb.msk_broker[each.value].arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.msk_broker_scram_public[each.value].arn
  }

  lifecycle {
    create_before_destroy = true
  }
}

module "msk_lb_cert" {
  source = "../certificates"

  for_each = local.broker_short_names

  org_public_zone_id = local.org_zone_id
  io_public_zone_id   = data.aws_route53_zone.io_public.zone_id

  org_domain_name = "${each.value}.${local.params.msk_org_domain}"
  vanity_domain_names = [
    "${each.value}.${local.params.msk_vanity_domain}"
  ]

  resource_health_check = false
  resource_domain_name  = aws_lb.msk_broker[each.value].dns_name
  resource_zone_id      = aws_lb.msk_broker[each.value].zone_id

  providers = {
    aws.org_zone = aws.org
    aws.io_zone = aws.io
  }
}

您可以使用NLB实现此功能。然后您将证书附加到NLB。您的证书将在NLB终止。NLB和MSK之间的连接将使用MSK证书。

目前不支持自定义域名。

为了使用R53和证书实现自定义域名，您需要在NLB终止证书，然后目标组将创建到代理的………IP地址的连接，因为在目标组中，您只能指定IP，而不是代理的域名。NLB和代理之间的SSL连接将失败，因为代理的IP地址不会添加到部署在代理端的证书中，因此NLB不会信任该连接。

唯一可行的方法是在NLB和MSK之间使用PLAINTEXT连接（端口9092）。但这不安全，不推荐这种方法。