Skip to content

bug(conntrack): wrong direction for end of http connections #1417

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
huntergregory opened this issue Mar 10, 2025 · 1 comment · May be fixed by #1438
Open

bug(conntrack): wrong direction for end of http connections #1417

huntergregory opened this issue Mar 10, 2025 · 1 comment · May be fixed by #1438
Assignees
@nddq
Labels
area/ebpf

Comments

@huntergregory
Copy link
Contributor

Setup

agnhost serving http on port 80. Nginx curling it

Result (AKS k8s 1.32 with ACNS)

Note the wrong direction since is_reply=true (see json):
dep-http-59bbf86d94-4xbbm:80 (ID:10892) <- toolbox-pod:39374

$ hubble observe --from-pod default/toolbox -f
Mar 10 21:11:21.536: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: SYN:true)
Mar 10 21:11:21.536: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-endpoint FORWARDED (TCP Flags: SYN:true)
Mar 10 21:11:21.537: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: ACK:true)
Mar 10 21:11:21.537: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-endpoint FORWARDED (TCP Flags: ACK:true)
Mar 10 21:11:21.538: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: PSH:true  ACK:true)
Mar 10 21:11:21.538: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-endpoint FORWARDED (TCP Flags: PSH:true  ACK:true)
Mar 10 21:11:21.543: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-endpoint FORWARDED (TCP Flags: ACK:true)
Mar 10 21:11:21.543: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: ACK:true)
Mar 10 21:11:21.549: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: FIN:true  ACK:true)
Mar 10 21:11:21.549: dep-http-59bbf86d94-4xbbm:80 (ID:10892) <- toolbox-pod:39374 (ID:10239) to-endpoint FORWARDED (TCP Flags: FIN:true  ACK:true)
Mar 10 21:11:21.550: toolbox-pod:39374 (ID:10239) -> dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: ACK:true)

In JSON:

$ hubble observe --from-pod default/toolbox -f -o json --experimental-field-mask source,destination,is_reply,l7,l4,IP
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"SYN":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"PSH":true,"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"PSH":true,"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"SYN":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"FIN":true,"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"FIN":true,"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.172","destination":"10.244.1.35","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":41780,"destination_port":80,"flags":{"ACK":true}}},"source":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default"],"pod_name":"toolbox-pod"},"destination":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"is_reply":false}}

Similarly for the reverse direction:

$ hubble observe --to-pod default/toolbox -f
Mar 10 21:11:46.370: toolbox-pod:54316 (ID:10239) <- dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: SYN:true  ACK:true)
Mar 10 21:11:46.370: toolbox-pod:54316 (ID:10239) <- dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-endpoint FORWARDED (TCP Flags: SYN:true  ACK:true)
Mar 10 21:11:46.372: toolbox-pod:54316 (ID:10239) <- dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: ACK:true)
Mar 10 21:11:46.372: toolbox-pod:54316 (ID:10239) <- dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-stack FORWARDED (TCP Flags: PSH:true  ACK:true)
Mar 10 21:11:46.374: dep-http-59bbf86d94-4xbbm:80 (ID:10892) -> toolbox-pod:54316 (ID:10239) to-stack FORWARDED (TCP Flags: FIN:true  ACK:true)
Mar 10 21:11:46.374: toolbox-pod:54316 (ID:10239) <- dep-http-59bbf86d94-4xbbm:80 (ID:10892) to-endpoint FORWARDED (TCP Flags: FIN:true  ACK:true)

In JSON:

$ hubble observe --to-pod default/toolbox -f -o json --experimental-field-mask source,destination,is_reply,l7,l4,IP
{"flow":{"IP":{"source":"10.244.1.35","destination":"10.244.1.172","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":38528,"flags":{"SYN":true,"ACK":true}}},"source":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"destination":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.35","destination":"10.244.1.172","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":38528,"flags":{"SYN":true,"ACK":true}}},"source":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"destination":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.35","destination":"10.244.1.172","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":38528,"flags":{"ACK":true}}},"source":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"destination":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.35","destination":"10.244.1.172","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":38528,"flags":{"PSH":true,"ACK":true}}},"source":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"destination":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.35","destination":"10.244.1.172","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":38528,"flags":{"FIN":true,"ACK":true}}},"source":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"destination":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.kubernetes.pod.namespace=default","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default"],"pod_name":"toolbox-pod"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.1.35","destination":"10.244.1.172","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":38528,"flags":{"FIN":true,"ACK":true}}},"source":{"ID":10892,"identity":10892,"namespace":"default","labels":["k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod-template-hash=59bbf86d94","k8s:pod=http","k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default"],"pod_name":"dep-http-59bbf86d94-4xbbm"},"destination":{"ID":10239,"identity":10239,"namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"is_reply":true}}

Cilium Comparison (also k8s 1.32 with ACNS)

$ hubble observe --from-pod default/toolbox -f
Mar 10 21:08:31.551: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-endpoint FORWARDED (TCP Flags: SYN)
Mar 10 21:08:31.545: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-stack FORWARDED (TCP Flags: SYN)
Mar 10 21:08:31.547: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-stack FORWARDED (TCP Flags: ACK)
Mar 10 21:08:31.552: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-endpoint FORWARDED (TCP Flags: ACK)
Mar 10 21:08:31.552: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Mar 10 21:08:31.553: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Mar 10 21:08:31.547: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-stack FORWARDED (TCP Flags: ACK, PSH)
Mar 10 21:08:31.549: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-stack FORWARDED (TCP Flags: ACK, FIN)
Mar 10 21:08:31.549: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-stack FORWARDED (TCP Flags: ACK)
Mar 10 21:08:31.554: default/toolbox-pod:50464 (ID:43214) -> default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-endpoint FORWARDED (TCP Flags: ACK)

$ hubble observe --from-pod default/toolbox -f -o json --experimental-field-mask source,destination,is_reply,l7,l4,IP
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"SYN":true}}},"source":{"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":3803,"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr","workloads":[{"name":"dep-http","kind":"Deployment"}]},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"SYN":true}}},"source":{"ID":36,"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"ACK":true}}},"source":{"ID":36,"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"ACK":true}}},"source":{"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":3803,"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr","workloads":[{"name":"dep-http","kind":"Deployment"}]},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"PSH":true,"ACK":true}}},"source":{"ID":36,"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"PSH":true,"ACK":true}}},"source":{"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":3803,"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr","workloads":[{"name":"dep-http","kind":"Deployment"}]},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"FIN":true,"ACK":true}}},"source":{"ID":36,"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr"},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"FIN":true,"ACK":true}}},"source":{"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":3803,"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr","workloads":[{"name":"dep-http","kind":"Deployment"}]},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"ACK":true}}},"source":{"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"ID":3803,"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr","workloads":[{"name":"dep-http","kind":"Deployment"}]},"is_reply":false}}
{"flow":{"IP":{"source":"10.244.0.230","destination":"10.244.1.146","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":52380,"destination_port":80,"flags":{"ACK":true}}},"source":{"ID":36,"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"destination":{"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr"},"is_reply":false}}
$ hubble observe --to-pod default/toolbox -f
Mar 10 21:08:39.751: default/toolbox-pod:34462 (ID:43214) <- default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Mar 10 21:08:39.755: default/toolbox-pod:34462 (ID:43214) <- default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-stack FORWARDED (TCP Flags: SYN, ACK)
Mar 10 21:08:39.752: default/toolbox-pod:34462 (ID:43214) <- default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Mar 10 21:08:39.754: default/toolbox-pod:34462 (ID:43214) <- default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Mar 10 21:08:39.756: default/toolbox-pod:34462 (ID:43214) <- default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-stack FORWARDED (TCP Flags: ACK, PSH)
Mar 10 21:08:39.758: default/toolbox-pod:34462 (ID:43214) <- default/dep-http-59bbf86d94-646hr:80 (ID:8894) to-stack FORWARDED (TCP Flags: ACK, FIN)

$ hubble observe --to-pod default/toolbox -f -o json --experimental-field-mask source,destination,is_reply,l7,l4,IP
{"flow":{"IP":{"source":"10.244.1.146","destination":"10.244.0.230","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":35146,"flags":{"SYN":true,"ACK":true}}},"source":{"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr"},"destination":{"ID":36,"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.146","destination":"10.244.0.230","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":35146,"flags":{"SYN":true,"ACK":true}}},"source":{"ID":3803,"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr","workloads":[{"name":"dep-http","kind":"Deployment"}]},"destination":{"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.146","destination":"10.244.0.230","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":35146,"flags":{"PSH":true,"ACK":true}}},"source":{"ID":3803,"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr","workloads":[{"name":"dep-http","kind":"Deployment"}]},"destination":{"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.146","destination":"10.244.0.230","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":35146,"flags":{"PSH":true,"ACK":true}}},"source":{"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr"},"destination":{"ID":36,"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.146","destination":"10.244.0.230","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":35146,"flags":{"FIN":true,"ACK":true}}},"source":{"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr"},"destination":{"ID":36,"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
{"flow":{"IP":{"source":"10.244.1.146","destination":"10.244.0.230","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":80,"destination_port":35146,"flags":{"FIN":true,"ACK":true}}},"source":{"ID":3803,"identity":8894,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default","k8s:pod=http"],"pod_name":"dep-http-59bbf86d94-646hr","workloads":[{"name":"dep-http","kind":"Deployment"}]},"destination":{"identity":43214,"cluster_name":"hgregory-03-10-acns-cilium","namespace":"default","labels":["k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default","k8s:io.cilium.k8s.policy.cluster=hgregory-03-10-acns-cilium","k8s:io.cilium.k8s.policy.serviceaccount=default","k8s:io.kubernetes.pod.namespace=default"],"pod_name":"toolbox-pod"},"is_reply":true}}
@nddq nddq self-assigned this Mar 11, 2025
@nddq nddq added the area/ebpf label Mar 11, 2025
@nddq
Copy link
Contributor

nddq commented Mar 11, 2025 •
edited
Loading

note that this bug only affects closing connections where the source and dest are on the same host the server terminated the connection

This was referenced Mar 14, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nddq nddq

Labels
area/ebpf
Projects
Retina Triage Board
Status: No status
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(conntrack): remove closing tcp connections from conntrack gracefully microsoft/retina
2 participants
@nddq @huntergregory