NETOBSERV-2225 - Deploy static plugin at operator startup #1345

jpinsonneau · 2025-04-02T09:38:40Z

Description

Create the console plugin when FlowCollector doesn't exists to expose the new forms.

~~Suggested alternatives: #1346 & #1374~~

See netobserv/network-observability-console-plugin#763 for the forms implementations

Dependencies

n/a

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

openshift-ci · 2025-04-02T09:38:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jotak for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jpinsonneau · 2025-04-02T10:05:44Z

controllers/flowcollector_controller.go

+	// force reconcile at startup
+	go r.InitReconcile(ctx)
+
 	return nil
 }

+func (r *FlowCollectorReconciler) InitReconcile(ctx context.Context) error {
+	log := log.FromContext(ctx)
+	log.Info("Initializing resources...")
+
+	var err error
+	for attempt := range initReconcileAttempts {
+		// delay the reconcile calls to let some time to the cache to load
+		time.Sleep(5 * time.Second)
+		_, err = r.Reconcile(ctx, reconcile.Request{})
+		if err != nil {
+			log.Error(err, "Error while doing initial reconcile", "attempt", attempt)
+		} else {
+			break
+		}
+	}
+	return err
+}


☝️ I wonder if there is an out of box mechanism to trigger the loop after the cache loaded. That's why I'm using a sleep here and this will may work in all situations.

https://redhat-internal.slack.com/archives/C02939DP5L5/p1743518264445429

If I remember correctly, when a reconcile loop is failing, you can also return a time value to reschedule the reconciliation.

Yeah I gave a try with that without success.

I'm refactoring the code again to move the static content to another controller wich will be cleaner I guess. I will give another try with the reschedule time on the new controller 👍

github-actions · 2025-04-30T13:36:56Z

New images:

quay.io/netobserv/network-observability-operator:40740e0
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-sha-40740e0
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-sha-40740e0

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:40740e0 make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-sha-40740e0

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-sha-40740e0
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

memodi · 2025-05-05T18:26:20Z

/test e2e-operator

memodi · 2025-05-05T20:13:25Z

@jpinsonneau there seems to be an issue where plugin does become ready, despite pod is in Running state and no issues with Loki whatsoever:

time="2025-05-05T20:00:34Z" level=error msg="cannot unmarshal, response was: Ingester not ready: waiting for 15s after being ready\n" error="invalid character 'I' looking for beginning of value" module=handler
time="2025-05-05T20:00:34Z" level=error msg="cannot unmarshal, response was: Ingester not ready: waiting for 15s after being ready\n" error="invalid character 'I' looking for beginning of value" module=handler

the static plugin pod had this error:

$ oc -n openshift-netobserv-operator logs pod/netobserv-plugin-static-7c97f478bf-t7gj8
time="2025-05-05T20:10:16Z" level=info msg="Starting netobserv-console-plugin [build version: main-2c3b33b, build date: 2025-05-05 10:42] at log level info" module=main
time="2025-05-05T20:10:16Z" level=fatal msg="invalid config" error="neither Loki nor Prometheus is configured; at least one of them should have a URL defined" module=main

I had monolithic loki configured in my flowcollector - anything I am missing for static console plugin config?

jpinsonneau · 2025-05-06T07:19:09Z

@jpinsonneau there seems to be an issue where plugin does become ready, despite pod is in Running state and no issues with Loki whatsoever:

time="2025-05-05T20:00:34Z" level=error msg="cannot unmarshal, response was: Ingester not ready: waiting for 15s after being ready\n" error="invalid character 'I' looking for beginning of value" module=handler
time="2025-05-05T20:00:34Z" level=error msg="cannot unmarshal, response was: Ingester not ready: waiting for 15s after being ready\n" error="invalid character 'I' looking for beginning of value" module=handler

the static plugin pod had this error:

$ oc -n openshift-netobserv-operator logs pod/netobserv-plugin-static-7c97f478bf-t7gj8
time="2025-05-05T20:10:16Z" level=info msg="Starting netobserv-console-plugin [build version: main-2c3b33b, build date: 2025-05-05 10:42] at log level info" module=main
time="2025-05-05T20:10:16Z" level=fatal msg="invalid config" error="neither Loki nor Prometheus is configured; at least one of them should have a URL defined" module=main

I had monolithic loki configured in my flowcollector - anything I am missing for static console plugin config?

@memodi are you using the proper plugin image mentionned above ?
netobserv/network-observability-console-plugin#763

memodi · 2025-05-06T13:07:50Z

@memodi are you using the proper plugin image mentionned above ?
netobserv/network-observability-console-plugin#763

I was not, I didn't realize that's dependency. I'll try with it today. Thanks!

github-actions · 2025-05-06T14:33:09Z

New images:

quay.io/netobserv/network-observability-operator:1c54f0e
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-sha-1c54f0e
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-sha-1c54f0e

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:1c54f0e make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-sha-1c54f0e

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-sha-1c54f0e
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

jpinsonneau · 2025-05-06T14:42:48Z

@memodi FYI I did some changes to address #1345 (comment) comment

Just tested and the behavior remains the same 😉
It avoid using an unecessary service account and role

memodi · 2025-05-13T15:23:02Z

/jira NETOBSERV-2225

jotak · 2025-05-14T05:41:53Z

controllers/static/static_controller.go

+	log := log.FromContext(ctx)
+	log.Info("Initializing resources...")
+
+	for attempt := range initReconcileAttempts {


btw, what happens if all 5 attempts fail? The static plugin wouldn't deploy, but the rest would work normally? Or does it make the controller CLBO or so ? (my understanding is it's the first, but just want to make sure)

If the attempts fails, the static plugin will not be there at controller startup.
As soon as a reconcile loop is triggered, it will appears (ie creating a FlowCollector for example)

jotak · 2025-05-14T05:44:53Z

controllers/static/static_controller.go

+
+	r.status.SetUnknown()
+	defer r.status.Commit(ctx, r.Client)
+


After setting "Unknown", I think this controller should return if openshift isn't detected, right?
We could check it in Kind

That's done in the static reconciler using HasConsolePlugin function:

network-observability-operator/controllers/consoleplugin/consoleplugin_static_reconciler.go

Lines 40 to 77 in 1080b69

func (r *CPReconciler) reconcileStatic(ctx context.Context, desired *flowslatest.FlowCollector) error {

l := log.FromContext(ctx).WithName("console-plugin")

ctx = log.IntoContext(ctx, l)

// Retrieve current owned objects

err := r.Managed.FetchAll(ctx)

if err != nil {

return err

}

if r.ClusterInfo.HasConsolePlugin() {

if err = r.checkAutoPatch(ctx, desired, constants.StaticPluginName); err != nil {

return err

}

}

if r.ClusterInfo.HasConsolePlugin() {

// Create object builder

builder := newBuilder(r.Instance, &desired.Spec, constants.StaticPluginName)

if err = r.reconcilePlugin(ctx, &builder, &desired.Spec, constants.StaticPluginName, "NetObserv static plugin"); err != nil {

return err

}

if err = r.reconcileDeployment(ctx, &builder, &desired.Spec, constants.StaticPluginName, ""); err != nil {

return err

}

if err = r.reconcileServices(ctx, &builder, constants.StaticPluginName); err != nil {

return err

}

} else {

// delete any existing owned object

r.Managed.TryDeleteAll(ctx)

}

return nil

}

If console plugin is not available, nothing is deployed and the status goes ready

The static controller could deploy something else than the console plugin in future so I think it's better to keeps things separated here. WDYT ?

ah ok, yes sounds good, thanks!

jotak

Just a comment when not running on openshift, other than that lgtm

jpinsonneau · 2025-05-14T08:05:25Z

/restest

jotak · 2025-05-14T12:20:33Z

/lgtm

openshift-ci · 2025-06-05T13:27:00Z

@jpinsonneau: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-operator	`1080b69`	link	false	`/test e2e-operator`
ci/prow/ci-bundle-noo-bundle	`1080b69`	link	true	`/test ci-bundle-noo-bundle`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-merge-robot · 2025-06-05T13:28:13Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-merge-robot · 2025-06-05T13:28:14Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jpinsonneau force-pushed the 1942 branch from 0b47945 to efab0a0 Compare April 2, 2025 10:04

jpinsonneau commented Apr 2, 2025

View reviewed changes

jotak added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 2, 2025

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 2, 2025

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 2, 2025

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 2, 2025

jotak mentioned this pull request Apr 7, 2025

Create static console plugin in OLM bundle #1346

Closed

jpinsonneau force-pushed the 1942 branch from 5fb53db to c5da751 Compare April 14, 2025 09:04

netobserv deleted a comment from github-actions bot Apr 14, 2025

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 14, 2025

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 14, 2025

jpinsonneau force-pushed the 1942 branch 2 times, most recently from c4c57e1 to dbc4cbf Compare April 15, 2025 10:40

netobserv deleted a comment from github-actions bot Apr 15, 2025

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 15, 2025

jpinsonneau force-pushed the 1942 branch from dbc4cbf to 82efd81 Compare April 15, 2025 10:59

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 15, 2025

netobserv deleted a comment from github-actions bot Apr 15, 2025

openshift-merge-robot added the needs-rebase label Apr 15, 2025

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 15, 2025

jpinsonneau force-pushed the 1942 branch from 75f2aac to 82efd81 Compare April 15, 2025 12:39

openshift-merge-robot removed the needs-rebase label Apr 15, 2025

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 15, 2025

netobserv deleted a comment from github-actions bot Apr 15, 2025

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 15, 2025

always create console plugin

0a42edc

jpinsonneau force-pushed the 1942 branch from 82efd81 to 0a42edc Compare April 17, 2025 08:21

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 17, 2025

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Apr 30, 2025

update tests and remove ClusterRoleBinding for static plugin

0daa95e

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label May 6, 2025

lint

62b0165

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label May 6, 2025

netobserv deleted a comment from github-actions bot May 6, 2025

remove service account for static plugin

1080b69

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label May 6, 2025

jpinsonneau added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label May 6, 2025

jpinsonneau added the needs-review Tells that the PR needs a review label May 13, 2025

jotak reviewed May 14, 2025

View reviewed changes

openshift-ci bot assigned jotak May 14, 2025

openshift-ci bot added the lgtm label May 14, 2025

jotak removed the needs-review Tells that the PR needs a review label May 14, 2025

openshift-merge-robot added the needs-rebase label Jun 5, 2025

	func (r CPReconciler) reconcileStatic(ctx context.Context, desired flowslatest.FlowCollector) error {
	l := log.FromContext(ctx).WithName("console-plugin")
	ctx = log.IntoContext(ctx, l)

	// Retrieve current owned objects
	err := r.Managed.FetchAll(ctx)
	if err != nil {
	return err
	}

	if r.ClusterInfo.HasConsolePlugin() {
	if err = r.checkAutoPatch(ctx, desired, constants.StaticPluginName); err != nil {
	return err
	}
	}

	if r.ClusterInfo.HasConsolePlugin() {
	// Create object builder
	builder := newBuilder(r.Instance, &desired.Spec, constants.StaticPluginName)

	if err = r.reconcilePlugin(ctx, &builder, &desired.Spec, constants.StaticPluginName, "NetObserv static plugin"); err != nil {
	return err
	}

	if err = r.reconcileDeployment(ctx, &builder, &desired.Spec, constants.StaticPluginName, ""); err != nil {
	return err
	}

	if err = r.reconcileServices(ctx, &builder, constants.StaticPluginName); err != nil {
	return err
	}
	} else {
	// delete any existing owned object
	r.Managed.TryDeleteAll(ctx)
	}

	return nil
	}

NETOBSERV-2225 - Deploy static plugin at operator startup #1345

Are you sure you want to change the base?

NETOBSERV-2225 - Deploy static plugin at operator startup #1345

Uh oh!

Conversation

jpinsonneau commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Dependencies

Checklist

Uh oh!

openshift-ci bot commented Apr 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 30, 2025

Uh oh!

memodi commented May 5, 2025

Uh oh!

memodi commented May 5, 2025

Uh oh!

jpinsonneau commented May 6, 2025

Uh oh!

memodi commented May 6, 2025

Uh oh!

github-actions bot commented May 6, 2025

Uh oh!

jpinsonneau commented May 6, 2025

Uh oh!

memodi commented May 13, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jotak May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jotak left a comment

Choose a reason for hiding this comment

Uh oh!

jpinsonneau commented May 14, 2025

Uh oh!

jotak commented May 14, 2025

Uh oh!

openshift-ci bot commented Jun 5, 2025

Uh oh!

openshift-merge-robot commented Jun 5, 2025

Uh oh!

openshift-merge-robot commented Jun 5, 2025

Uh oh!

Uh oh!

jpinsonneau commented Apr 2, 2025 •

edited

Loading

memodi commented May 13, 2025 •

edited by openshift-ci bot

Loading

jotak May 14, 2025 •

edited

Loading