guestagent: k8s: use kubectl instead of client-go

[AkihiroSuda]

Part of:

• guestagent binaries are too huge (k8s.io deps should be replaced with exec("kubectl")?) #3237

TODO: drop dependency on k8s.io/api

---

Merge after:

• CI: test Kubernetes port forwarder #4118

[AkihiroSuda]

Before (964fb30)

$ du -hs _output/
128M    _output/

$ ls -lh _output/bin/limactl _output/share/lima/lima-guestagent.Linux-*
-rwxr-xr-x@ 1 suda  staff    28M Oct  1 19:47 _output/bin/limactl*
-rw-r--r--@ 1 suda  staff    14M Oct  1 19:47 _output/share/lima/lima-guestagent.Linux-aarch64.gz
-rw-r--r--@ 1 suda  staff    15M Oct  1 19:47 _output/share/lima/lima-guestagent.Linux-armv7l.gz
-rw-r--r--@ 1 suda  staff    14M Oct  1 19:47 _output/share/lima/lima-guestagent.Linux-ppc64le.gz
-rw-r--r--@ 1 suda  staff    15M Oct  1 19:47 _output/share/lima/lima-guestagent.Linux-riscv64.gz
-rw-r--r--@ 1 suda  staff    16M Oct  1 19:47 _output/share/lima/lima-guestagent.Linux-s390x.gz
-rw-r--r--@ 1 suda  staff    16M Oct  1 19:47 _output/share/lima/lima-guestagent.Linux-x86_64.gz

After (f74fbdf)

$ du -hs _output/
 88M    _output/

$ ls -lh _output/bin/limactl _output/share/lima/lima-guestagent.Linux-*
-rwxr-xr-x@ 1 suda  staff    28M Oct  1 19:49 _output/bin/limactl*
-rw-r--r--@ 1 suda  staff   8.4M Oct  1 19:49 _output/share/lima/lima-guestagent.Linux-aarch64.gz
-rw-r--r--@ 1 suda  staff   8.8M Oct  1 19:49 _output/share/lima/lima-guestagent.Linux-armv7l.gz
-rw-r--r--@ 1 suda  staff   8.4M Oct  1 19:49 _output/share/lima/lima-guestagent.Linux-ppc64le.gz
-rw-r--r--@ 1 suda  staff   8.8M Oct  1 19:49 _output/share/lima/lima-guestagent.Linux-riscv64.gz
-rw-r--r--@ 1 suda  staff   9.2M Oct  1 19:49 _output/share/lima/lima-guestagent.Linux-s390x.gz
-rw-r--r--@ 1 suda  staff   9.3M Oct  1 19:49 _output/share/lima/lima-guestagent.Linux-x86_64.gz

[jandubois]

I have not looked at this PR at all yet, but wanted to mention a couple of things I discussed with @Nino-K as requirements for his port monitoring PR:

• Keep retrying to connect to k8s indefinitely, as it will not be running yet by the time the guest agent starts.
• When the connection breaks, keep trying to reconnect with a short delay indefinitely, as the user may have stopped and restarted k8s.

For this PR also: the kubectl binary may not yet be available on the PATH when you try to invoke it, keep trying. It may also fail because the port may be open, but the apiserver not yet responding, or the kubeconfig may be missing etc. The retry on broken connections should handle this automatically.

Because of the indefinite retries, the kubernetes watcher should be opt-in (configurable in lima.yaml), so it only runs when the VM is known to run Kubernetes.

[AkihiroSuda]

The retry on broken connections should handle this automatically.

Yes, this is retried.

Because of the indefinite retries, the kubernetes watcher should be opt-in

It wasn't opt-in so far, and I don't think it has to be so, as the overhead of polling LookPath("kubectl") seems trivial.

[nirs]

Trimming the guest agent is nice, anything using client-go becomes huge quickly. But did you measure memory and cpu usage before and after this change?

With the new code we always keep kubectl watch command running, which has the similar cpu usage to what we had before in the guest agent, but now we format the json events and parse them back in the guest agent, and keep all service in memory twice, once is kubectl (using the informer) and once in the guest agent.

[AkihiroSuda]

Trimming the guest agent is nice, anything using client-go becomes huge quickly. But did you measure memory and cpu usage before and after this change?

Before

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1589 root      20   0 1284200  63488  36992 S   0.0   1.6   0:10.21 lima-guestagent

After

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                           
   1192 root      20   0 1254740  41600  19200 S   0.0   1.0   0:01.72 lima-guestagent                                                                                                                   
   2932 root      20   0 1284688  44404  33536 S   0.0   1.1   0:00.02 kubectl