Skip to content

Commit d982fce

Browse files
sspainksspaink
committed
feat: new "immediate" trigger mode for decision log plugin
If reporting.trigger="immediate" chunks of events will be uploaded as soon as they are ready. Signed-off-by: sspaink <[email protected]>
1 parent 2c12366 commit d982fce

File tree

11 files changed

+623
-195
lines changed

11 files changed

+623
-195
lines changed

Diff for: ‎docs/content/configuration.md

+17-17
Original file line numberDiff line numberDiff line change
@@ -779,24 +779,24 @@ included in the actual bundle gzipped tarball.
779779

780780
## Decision Logs
781781

782-
| Field | Type | Required | Description |
783-
|----------------------------------------------------|-----------|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
784-
| `decision_logs.service` | `string` | No | Name of the service to use to contact remote server. If no `plugin` is specified, and `console` logging is disabled, this will default to the first `service` name defined in the Services configuration. |
785-
| `decision_logs.partition_name` | `string` | No | Deprecated: Use `resource` instead. Path segment to include in status updates. |
786-
| `decision_logs.resource` | `string` | No (default: `/logs`) | Full path to use for sending decision logs to a remote server. |
787-
| `decision_logs.reporting.buffer_type` | `string` | No (default: `size`) | Toggles the type of buffer to use. The two available options are "size" or "event". Refer to the [Decision Log Plugin README](https://github.com/open-policy-agent/opa/tree/main/v1/plugins/logs/README.md) for for a detailed comparison. |
788-
| `decision_logs.reporting.buffer_size_limit_events` | `int64` | No (default: `10000`) | Decision log buffer size limit by events. OPA will drop old events from the log if this limit is exceeded. By default, 100 events are held. This number has to be greater than zero. Only works with "event" buffer type. |
782+
| Field | Type | Required | Description |
783+
|----------------------------------------------------|-----------|-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
784+
| `decision_logs.service` | `string` | No | Name of the service to use to contact remote server. If no `plugin` is specified, and `console` logging is disabled, this will default to the first `service` name defined in the Services configuration. |
785+
| `decision_logs.partition_name` | `string` | No | Deprecated: Use `resource` instead. Path segment to include in status updates. |
786+
| `decision_logs.resource` | `string` | No (default: `/logs`) | Full path to use for sending decision logs to a remote server. |
787+
| `decision_logs.reporting.buffer_type` | `string` | No (default: `size`) | Toggles the type of buffer to use. The two available options are "size" or "event". Refer to the [Decision Log Plugin README](https://github.com/open-policy-agent/opa/tree/main/v1/plugins/logs/README.md) for for a detailed comparison. |
788+
| `decision_logs.reporting.buffer_size_limit_events` | `int64` | No (default: `10000`) | Decision log buffer size limit by events. OPA will drop old events from the log if this limit is exceeded. By default, 100 events are held. This number has to be greater than zero. Only works with "event" buffer type. |
789789
| `decision_logs.reporting.buffer_size_limit_bytes` | `int64` | No (default: `unlimited`) | Decision log buffer size limit in bytes. OPA will drop old events from the log if this limit is exceeded. By default, no limit is set. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. Only works with "size" buffer type. |
790-
| `decision_logs.reporting.max_decisions_per_second` | `float64` | No | Maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. |
791-
| `decision_logs.reporting.upload_size_limit_bytes` | `int64` | No (default: `32768`) | Decision log upload size limit in bytes. OPA will chunk uploads to cap message body to this limit. |
792-
| `decision_logs.reporting.min_delay_seconds` | `int64` | No (default: `300`) | Minimum amount of time to wait between uploads. |
793-
| `decision_logs.reporting.max_delay_seconds` | `int64` | No (default: `600`) | Maximum amount of time to wait between uploads. |
794-
| `decision_logs.reporting.trigger` | `string` | No (default: `periodic`) | Controls how decision logs are reported to the remote server. Allowed values are `periodic` and `manual` (`manual` triggers are only possible when using OPA as a Go package). |
795-
| `decision_logs.mask_decision` | `string` | No (default: `/system/log/mask`) | Set path of masking decision. |
796-
| `decision_logs.drop_decision` | `string` | No (default: `/system/log/drop`) | Set path of drop decision. |
797-
| `decision_logs.plugin` | `string` | No | Use the named plugin for decision logging. If this field exists, the other configuration fields are not required. |
798-
| `decision_logs.console` | `boolean` | No (default: `false`) | Log the decisions locally to the console. When enabled alongside a remote decision logging API the `service` must be configured, the default `service` selection will be disabled. |
799-
| `decision_logs.request_context.http.headers` | `array` | No | List of HTTP headers to include in the decision log. OPA will include the values for these headers in the decision log if they exist in the incoming HTTP request. |
790+
| `decision_logs.reporting.max_decisions_per_second` | `float64` | No | Maximum number of decision log events to buffer per second. OPA will drop events if the rate limit is exceeded. Only one of `buffer_size_limit_bytes`, `max_decisions_per_second` may be set. |
791+
| `decision_logs.reporting.upload_size_limit_bytes` | `int64` | No (default: `32768`) | Decision log upload size limit in bytes. OPA will chunk uploads to cap message body to this limit. |
792+
| `decision_logs.reporting.min_delay_seconds` | `int64` | No (default: `300`) | Minimum amount of time to wait between uploads. |
793+
| `decision_logs.reporting.max_delay_seconds` | `int64` | No (default: `600`) | Maximum amount of time to wait between uploads. |
794+
| `decision_logs.reporting.trigger` | `string` | No (default: `periodic`) | Controls how decision logs are reported to the remote server. Allowed values are `periodic`, `manual` and `immediate` (`manual` triggers are only possible when using OPA as a Go package). |
795+
| `decision_logs.mask_decision` | `string` | No (default: `/system/log/mask`) | Set path of masking decision. |
796+
| `decision_logs.drop_decision` | `string` | No (default: `/system/log/drop`) | Set path of drop decision. |
797+
| `decision_logs.plugin` | `string` | No | Use the named plugin for decision logging. If this field exists, the other configuration fields are not required. |
798+
| `decision_logs.console` | `boolean` | No (default: `false`) | Log the decisions locally to the console. When enabled alongside a remote decision logging API the `service` must be configured, the default `service` selection will be disabled. |
799+
| `decision_logs.request_context.http.headers` | `array` | No | List of HTTP headers to include in the decision log. OPA will include the values for these headers in the decision log if they exist in the incoming HTTP request. |
800800

801801
## Discovery
802802

Diff for: ‎v1/logging/test/test.go

+4
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,10 @@ func (l *Logger) Entries() []LogEntry {
9191
}
9292

9393
func (l *Logger) append(lvl logging.Level, f string, a ...interface{}) {
94+
// only record logs that match the level range, allows tests to look for specific errors
95+
if lvl > l.level {
96+
return
97+
}
9498
l.mtx.Lock()
9599
defer l.mtx.Unlock()
96100
*l.entries = append(*l.entries, LogEntry{

Diff for: ‎v1/plugins/bundle/config_test.go

+1-1
Original file line numberDiff line numberDiff line change
@@ -471,7 +471,7 @@ func TestParseConfigTriggerMode(t *testing.T) {
471471
conf: `{"b1":{"service": "s1", "trigger": "foo"}}`,
472472
services: []string{"s1"},
473473
wantError: true,
474-
err: errors.New("invalid configuration for bundle \"b1\": invalid trigger mode \"foo\" (want \"periodic\" or \"manual\")"),
474+
err: errors.New("invalid configuration for bundle \"b1\": invalid trigger mode \"foo\" (want \"periodic\", \"immediate\" or \"manual\")"),
475475
triggerMode: nil,
476476
},
477477
}

Diff for: ‎v1/plugins/logs/README.md

+30-3
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,11 @@ the user to decide when to upload, drop or proxy a logged event. Each configurat
77
Events are uploaded in gzip compressed JSON array's at a user defined interval. This can either be triggered periodically
88
or manually through the SDK. The size of the gzip compressed JSON array is limited by `upload_size_limit_bytes`.
99

10+
## Buffer Types
11+
1012
There are two buffer implementations that can be selected by setting `decision_logs.reporting.buffer_type`, defaults to `size`
1113

12-
## Event Buffer
14+
### Event Buffer
1315

1416
* `decision_logs.reporting.buffer_type=event`
1517

@@ -45,7 +47,7 @@ flowchart LR
4547
4648
```
4749

48-
## Size Buffer
50+
### Size Buffer
4951

5052
* `decision_logs.reporting.buffer_type=size`
5153

@@ -85,4 +87,29 @@ flowchart LR
8587
Buffer -. POST .-> service
8688
classDef large font-size:20pt;
8789
88-
```
90+
```
91+
92+
## Triggers
93+
94+
There are three trigger options that can be selected by setting `decision_logs.reporting.trigger`, defaults to `periodic`.
95+
96+
### Periodic
97+
98+
Uploads are purposely delayed by number of seconds randomly selected between a minimum and maximum. The default delay
99+
range is 300-600 seconds, this can be configured by setting `decision_logs.reporting.min_delay_seconds` and
100+
`decision_logs.reporting.max_delay_seconds`.
101+
102+
It is recommended to use this trigger mode to prevent overloading the service with upload requests.
103+
104+
### Immediate
105+
106+
As soon as enough events are received that hit the upload size limit the plugin will trigger an upload. When using this
107+
trigger mode the `min_delay_seconds` cannot be set as it can be considered to be 0. The `max_delay_seconds` is still
108+
configurable in case not enough events are received to hit the upload limit.
109+
110+
It is recommended to use this trigger mode if you need minimum latency.
111+
112+
### Manual
113+
114+
This option can only be used when using OPA as a Go package. The OPA Go package exposes as method called [Plugin.Trigger](https://pkg.go.dev/github.com/open-policy-agent/[email protected]/v1/plugins/logs#Plugin.Trigger)
115+
that can be called to trigger an upload.

Diff for: ‎v1/plugins/logs/encoder.go

+6
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,13 @@ func (enc *chunkEncoder) reset() ([][]byte, error) {
147147
}
148148

149149
mul := int64(math.Pow(float64(softLimitBaseFactor), float64(enc.softLimitScaleUpExponent+1)))
150+
// this can cause enc.softLimit to overflow into a negative value
151+
// this behaviour can be seen in TestPluginTriggerManual
150152
enc.softLimit *= mul
153+
// have MaxInt64 be the ceiling for the soft limit
154+
if enc.softLimit < 0 {
155+
enc.softLimit = math.MaxInt64 - 1
156+
}
151157
enc.softLimitScaleUpExponent += softLimitExponentScaleFactor
152158
return enc.update(), nil
153159
}

0 commit comments

Comments
 (0)