Skip to content

Conversation

@Dav1dde
Copy link
Member

@Dav1dde Dav1dde commented Oct 20, 2025

Applies only non-destructive rules to $log.body, also re-uses pii fixtures from existing tests.

@Dav1dde Dav1dde force-pushed the dav1d/logs-body-pii branch from 8eccec5 to cd52de5 Compare October 21, 2025 07:38
@Dav1dde Dav1dde marked this pull request as ready for review October 21, 2025 07:38
@Dav1dde Dav1dde requested a review from a team as a code owner October 21, 2025 07:38
@Dav1dde Dav1dde changed the title ref(ourlog): Only enable less destructive PII on the log body feat(ourlog): Only enable less destructive PII on the log body Oct 21, 2025
@Dav1dde Dav1dde force-pushed the dav1d/logs-body-pii branch from cd52de5 to 578574f Compare October 21, 2025 07:40
@Dav1dde Dav1dde self-assigned this Oct 21, 2025

/// Log body.
#[metastructure(required = true, pii = "true", trim = false)]
#[metastructure(required = true, pii = "maybe", trim = false)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this still be true? E.g. $logentry.formatted is also in REPLACE_ONLY_SELECTOR but still has pii = true.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$logentry.formatted is explicitly removed from PII instead:

static DATASCRUBBER_IGNORE: LazyLock<SelectorSpec> = LazyLock::new(|| {
    "(debug_meta.** | $frame.filename | $frame.abs_path | $logentry.formatted | $error.value | $request.headers.user-agent)"
        .parse()
        .unwrap()
});

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this some weirdness around legacy? I'd expect Pii::Maybe to be advanced rules only, is this because

applications.insert(
REPLACE_ONLY_SELECTOR.clone(),
vec![
"@email:replace".to_owned(),
"@creditcard:replace".to_owned(),
"@iban:replace".to_owned(),
"@usssn:replace".to_owned(),
"@bearer:replace".to_owned(),
],
);
is inserting it back as 'advanced' / new PII rules?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Legacy (default scrubbers) are converted to 'advanced rules' there is some magic where these default rules won't apply to maybe. But we opt out in the conversion and inject these extra rules to for specific fields now. There is a longterm plan now to unify this, but until we have that, this is the workaround.

Copy link
Member

@k-fish k-fish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine, however I don't know the ramifications of removing the password:filter. Although it is destructive, it catches auth / secrets / api keys and if those are important we really should try to replace that behaviour if it was previously working for users.


/// Log body.
#[metastructure(required = true, pii = "true", trim = false)]
#[metastructure(required = true, pii = "maybe", trim = false)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this some weirdness around legacy? I'd expect Pii::Maybe to be advanced rules only, is this because

applications.insert(
REPLACE_ONLY_SELECTOR.clone(),
vec![
"@email:replace".to_owned(),
"@creditcard:replace".to_owned(),
"@iban:replace".to_owned(),
"@usssn:replace".to_owned(),
"@bearer:replace".to_owned(),
],
);
is inserting it back as 'advanced' / new PII rules?

@Dav1dde Dav1dde enabled auto-merge October 23, 2025 06:43
@Dav1dde Dav1dde added this pull request to the merge queue Oct 23, 2025
Merged via the queue into master with commit 7d6f69e Oct 23, 2025
28 checks passed
@Dav1dde Dav1dde deleted the dav1d/logs-body-pii branch October 23, 2025 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants