Skip to content

feat: new "immediate" trigger mode for decision log plugin #7516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: new "immediate" trigger mode for decision log plugin #7516

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions docs/content/configuration.md
View file
Original file line number Diff line number Diff line change
@@ -779,24 +779,24 @@ included in the actual bundle gzipped tarball.

## Decision Logs

| Field | Type | Required | Description |
|----------------------------------------------------|-----------|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `decision_logs.service` | `string` | No | Name of the service to use to contact remote server. If no `plugin` is specified, and `console` logging is disabled, this will default to the first `service` name defined in the Services configuration. |
| `decision_logs.partition_name` | `string` | No | Deprecated: Use `resource` instead. Path segment to include in status updates. |
| `decision_logs.resource` | `string` | No (default: `/logs`) | Full path to use for sending decision logs to a remote server. |
| `decision_logs.reporting.buffer_type` | `string` | No (default: `size`) | Toggles the type of buffer to use. The two available options are "size" or "event". Refer to the [Decision Log Plugin README](https://github.com/open-policy-agent/opa/tree/main/v1/plugins/logs/README.md) for for a detailed comparison. |
| `decision_logs.reporting.buffer_size_limit_events` | `int64` | No (default: `10000`) | Decision log buffer size limit by events. OPA will drop old events from the log if this limit is exceeded. By default, 100 events are held. This number has to be greater than zero. Only works with "event" buffer type. |
| Field | Type | Required | Description |
|----------------------------------------------------|-----------|-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `decision_logs.service` | `string` | No | Name of the service to use to contact remote server. If no `plugin` is specified, and `console` logging is disabled, this will default to the first `service` name defined in the Services configuration. |
| `decision_logs.partition_name` | `string` | No | Deprecated: Use `resource` instead. Path segment to include in status updates. |
| `decision_logs.resource` | `string` | No (default: `/logs`) | Full path to use for sending decision logs to a remote server. |
| `decision_logs.reporting.buffer_type` | `string` | No (default: `size`) | Toggles the type of buffer to use. The two available options are "size" or "event". Refer to the [Decision Log Plugin README](https://github.com/open-policy-agent/opa/tree/main/v1/plugins/logs/README.md) for for a detailed comparison. |
| `decision_logs.reporting.buffer_size_limit_events` | `int64` | No (default: `10000`) | Decision log buffer size limit by events. OPA will drop old events from the log if this limit is exceeded. By default, 100 events are held. This number has to be greater than zero. Only works with "event" buffer type. |
| `decision_logs.reporting.buffer_size_limit_bytes` | `int64` | No (default: `unlimited`) | Decision log buffer size limit in bytes. OPA will drop old events from the log if this limit is exceeded. By default, no limit is set. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. Only works with "size" buffer type. |
| `decision_logs.reporting.max_decisions_per_second` | `float64` | No | Maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. |
| `decision_logs.reporting.upload_size_limit_bytes` | `int64` | No (default: `32768`) | Decision log upload size limit in bytes. OPA will chunk uploads to cap message body to this limit. |
| `decision_logs.reporting.min_delay_seconds` | `int64` | No (default: `300`) | Minimum amount of time to wait between uploads. |
| `decision_logs.reporting.max_delay_seconds` | `int64` | No (default: `600`) | Maximum amount of time to wait between uploads. |
| `decision_logs.reporting.trigger` | `string` | No (default: `periodic`) | Controls how decision logs are reported to the remote server. Allowed values are `periodic` and `manual` (`manual` triggers are only possible when using OPA as a Go package). |
| `decision_logs.mask_decision` | `string` | No (default: `/system/log/mask`) | Set path of masking decision. |
| `decision_logs.drop_decision` | `string` | No (default: `/system/log/drop`) | Set path of drop decision. |
| `decision_logs.plugin` | `string` | No | Use the named plugin for decision logging. If this field exists, the other configuration fields are not required. |
| `decision_logs.console` | `boolean` | No (default: `false`) | Log the decisions locally to the console. When enabled alongside a remote decision logging API the `service` must be configured, the default `service` selection will be disabled. |
| `decision_logs.request_context.http.headers` | `array` | No | List of HTTP headers to include in the decision log. OPA will include the values for these headers in the decision log if they exist in the incoming HTTP request. |
| `decision_logs.reporting.max_decisions_per_second` | `float64` | No | Maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. |
| `decision_logs.reporting.upload_size_limit_bytes` | `int64` | No (default: `32768`) | Decision log upload size limit in bytes. OPA will chunk uploads to cap message body to this limit. |
| `decision_logs.reporting.min_delay_seconds` | `int64` | No (default: `300`) | Minimum amount of time to wait between uploads. |
| `decision_logs.reporting.max_delay_seconds` | `int64` | No (default: `600`) | Maximum amount of time to wait between uploads. |
| `decision_logs.reporting.trigger` | `string` | No (default: `periodic`) | Controls how decision logs are reported to the remote server. Allowed values are `periodic`, `manual` and `immediate` (`manual` triggers are only possible when using OPA as a Go package). |
| `decision_logs.mask_decision` | `string` | No (default: `/system/log/mask`) | Set path of masking decision. |
| `decision_logs.drop_decision` | `string` | No (default: `/system/log/drop`) | Set path of drop decision. |
| `decision_logs.plugin` | `string` | No | Use the named plugin for decision logging. If this field exists, the other configuration fields are not required. |
| `decision_logs.console` | `boolean` | No (default: `false`) | Log the decisions locally to the console. When enabled alongside a remote decision logging API the `service` must be configured, the default `service` selection will be disabled. |
| `decision_logs.request_context.http.headers` | `array` | No | List of HTTP headers to include in the decision log. OPA will include the values for these headers in the decision log if they exist in the incoming HTTP request. |

## Discovery

2 changes: 1 addition & 1 deletion v1/plugins/bundle/config_test.go
View file
Original file line number Diff line number Diff line change
@@ -471,7 +471,7 @@ func TestParseConfigTriggerMode(t *testing.T) {
conf: `{"b1":{"service": "s1", "trigger": "foo"}}`,
services: []string{"s1"},
wantError: true,
err: errors.New("invalid configuration for bundle \"b1\": invalid trigger mode \"foo\" (want \"periodic\" or \"manual\")"),
err: errors.New("invalid configuration for bundle \"b1\": invalid trigger mode \"foo\" (want \"periodic\", \"immediate\" or \"manual\")"),
triggerMode: nil,
},
}
33 changes: 30 additions & 3 deletions v1/plugins/logs/README.md
View file
Original file line number Diff line number Diff line change
@@ -7,9 +7,11 @@ the user to decide when to upload, drop or proxy a logged event. Each configurat
Events are uploaded in gzip compressed JSON array's at a user defined interval. This can either be triggered periodically
or manually through the SDK. The size of the gzip compressed JSON array is limited by `upload_size_limit_bytes`.

## Buffer Types

There are two buffer implementations that can be selected by setting `decision_logs.reporting.buffer_type`, defaults to `size`

## Event Buffer
### Event Buffer

* `decision_logs.reporting.buffer_type=event`

@@ -45,7 +47,7 @@ flowchart LR

```

## Size Buffer
### Size Buffer

* `decision_logs.reporting.buffer_type=size`

@@ -85,4 +87,29 @@ flowchart LR
Buffer -. POST .-> service
classDef large font-size:20pt;

```
```

## Triggers

There are three trigger options that can be selected by setting `decision_logs.reporting.trigger`, defaults to `periodic`.

### Periodic

Uploads are purposely delayed by number of seconds randomly selected between a minimum and maximum. The default delay
range is 300-600 seconds, this can be configured by setting `decision_logs.reporting.min_delay_seconds` and
`decision_logs.reporting.max_delay_seconds`.

It is recommended to use this trigger mode to prevent overloading the service with upload requests.

### Immediate

As soon as enough events are received that hit the upload size limit the plugin will trigger an upload. When using this
trigger mode the `min_delay_seconds` cannot be set as it can be considered to be 0. The `max_delay_seconds` is still
configurable in case not enough events are received to hit the upload limit.

It is recommended to use this trigger mode if you need minimum latency.

### Manual

This option can only be used when using OPA as a Go package. The OPA Go package exposes as method called [Plugin.Trigger](https://pkg.go.dev/github.com/open-policy-agent/[email protected]/v1/plugins/logs#Plugin.Trigger)
that can be called to trigger an upload.
4 changes: 3 additions & 1 deletion v1/plugins/logs/encoder.go
View file
Original file line number Diff line number Diff line change
@@ -126,7 +126,9 @@ func (enc *chunkEncoder) Flush() ([][]byte, error) {
if err := enc.writeClose(); err != nil {
return nil, err
}
return enc.reset()
// don't call enc.reset() because the uncompressed limit shouldn't be updated when forcing the buffer to be emptied
// the buffer could most likely be underutilized (<90%) and won't be an accurate data point
return enc.update(), nil
}

//nolint:unconvert
4 changes: 2 additions & 2 deletions v1/plugins/logs/encoder_test.go
View file
Original file line number Diff line number Diff line change
@@ -149,8 +149,8 @@ func TestChunkEncoderAdaptive(t *testing.T) {
actualScaleDownEvents := enc.metrics.Counter(encSoftLimitScaleDownCounterName).Value().(uint64)
actualEquiEvents := enc.metrics.Counter(encSoftLimitStableCounterName).Value().(uint64)

expectedScaleUpEvents := uint64(8)
expectedScaleDownEvents := uint64(3)
expectedScaleUpEvents := uint64(5)
expectedScaleDownEvents := uint64(0)
expectedEquiEvents := uint64(0)

if actualScaleUpEvents != expectedScaleUpEvents {
Loading
Oops, something went wrong.
Loading
Oops, something went wrong.