Running Kong API Gateway and KIC separately

[pablo-balboa]

I have the Kong API gateway installed in a k8s cluster (A) and KIC running in another (B). I have managed to connect them and also when creating the ingress in cluster B, Kong automatically creates the resources in the Gateway (routes, services, etc). My issue is when looking at KIC pod logs in cluster B shows this error

error controllers.Ingress.netv1 Reconciler error {"reconcileID": "2734797c-7a3a-4bab-9956-fd5875d42e25", "error": "Service \"kong-ingress-kong-proxy\" not found"} 2025-03-03T15:29:01Z error controllers.Gateway One of publish services defined in Gateway's "konghq.com/publish-service" annotation didn't match controller manager's configuration {"GatewayV1Gateway": {"name":"kong","namespace":"kong"}, "namespace": "kong", "name": "kong", "service": "kong/kong-ingress-gateway-proxy", "error": "publish service reference \"kong/kong-ingress-gateway-proxy\" from Gateway's annotations did not match configured controller manager's publish services (\"kong/kong-ingress-kong-proxy\")"}

this is the KIC deployment definition

`apiVersion: apps/v1
kind: Deployment
metadata:
name: kong-ingress-kong
namespace: kong

labels:
app.kubernetes.io/component: app
app.kubernetes.io/instance: kong-ingress
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kong
app.kubernetes.io/version: '3.9'
helm.sh/chart: kong-2.47.0
annotations:
deployment.kubernetes.io/revision: '1'
meta.helm.sh/release-name: kong-ingress
meta.helm.sh/release-namespace: kong

spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: app
app.kubernetes.io/instance: kong-ingress
app.kubernetes.io/name: kong
template:
metadata:
creationTimestamp: null
labels:
app: kong-ingress-kong
app.kubernetes.io/component: app
app.kubernetes.io/instance: kong-ingress
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kong
app.kubernetes.io/version: '3.9'
helm.sh/chart: kong-2.47.0
version: '3.9'
annotations:
kuma.io/gateway: enabled
kuma.io/service-account-token-volume: kong-ingress-kong-token
traffic.sidecar.istio.io/includeInboundPorts: ''
spec:
volumes:
- name: kong-ingress-kong-prefix-dir
emptyDir:
sizeLimit: 256Mi
- name: kong-ingress-kong-tmp
emptyDir:
sizeLimit: 1Gi
- name: kong-ingress-kong-token
projected:
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
name: kube-root-ca.crt
items:
- key: ca.crt
path: ca.crt
- downwardAPI:
items:
- path: namespace
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
defaultMode: 420
- name: webhook-cert
secret:
secretName: kong-ingress-kong-validation-webhook-keypair
defaultMode: 420
containers:
- name: ingress-controller
image: kong/kubernetes-ingress-controller:3.4
ports:
- name: webhook
containerPort: 8080
protocol: TCP
- name: cmetrics
containerPort: 10255
protocol: TCP
- name: cstatus
containerPort: 10254
protocol: TCP
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: CONTROLLER_ADMISSION_WEBHOOK_LISTEN
value: 0.0.0.0:8080
- name: CONTROLLER_ELECTION_ID
value: kong-ingress-controller-leader-kong
- name: CONTROLLER_INGRESS_CLASS
value: kong
- name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
value: 'true'
- name: CONTROLLER_KONG_ADMIN_TOKEN
valueFrom:
secretKeyRef:
name: kong-admin-token-secret
key: kong-admin-token
- name: CONTROLLER_KONG_ADMIN_URL
value: https://poc-adminapi.dev.01.com
- name: CONTROLLER_PUBLISH_SERVICE
value: kong/kong-ingress-kong-proxy
resources: {}
volumeMounts:
- name: webhook-cert
readOnly: true
mountPath: /admission-webhook
- name: kong-ingress-kong-token
readOnly: true
mountPath: /var/run/secrets/kubernetes.io/serviceaccount
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /readyz
port: 10254
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
drop:
- ALL
runAsUser: 1000
runAsNonRoot: true
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
seccompProfile:
type: RuntimeDefault
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: kong-ingress-kong
serviceAccount: kong-ingress-kong
automountServiceAccountToken: false
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
`