http conn man: Introduce host simplification rules #40874

pugmajere · 2025-08-28T01:26:47Z

Commit Message:
http conn man: Introduce host simplification rules

Today, Envoy only supports a single wildcard ("*") in virtual host domain entries, which must be either a prefix or suffix.. This limitation greatly simplifies the virtual host matching logic, but it is inherently limiting in some ways.

This change introduces a repeated list of "host simplification rules" at the RouteConfiguration level, that provide a way to substitute away other dynamic portions of the domain without changing what is sent upstream.

For example, to match something like
*.foo.*.example.org you might write a simplification rule like: ([^.]*[.]foo[.])([^.]+)([.]example[.]org) with a substitution of \1bar\3. This then allows a virtual host domain entry of *.foo.bar.example.org to match
baz.foo.bar.example.org or wowza.foo.qux.example.org.

Host simplification rules are processed in the order they are defined.

Additional Description:
Risk Level: low (minimal new code, mostly reused, optional feature that has no impact if not enabled)
Testing: tests added
Docs Changes: proto docs update
Release Notes: changelog updated
Platform Specific Features: none
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]
I think everything is covered here.

repokitteh-read-only · 2025-08-28T01:26:52Z

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #40874 was opened by pugmajere.

see: more, trace.

repokitteh-read-only · 2025-08-28T01:26:57Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @adisuissa
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #40874 was opened by pugmajere.

see: more, trace.

pugmajere · 2025-08-28T01:27:43Z

This is in draft at the moment because my primary build environment is on MacOS, which is just an absolute nightmare for running tests. Give me a bit to finish it up.

Today, Envoy only supports a single wildcard ("*") in virtual host domain entries, which must be either a prefix or suffix. This limitation greatly simplifies the virtual host matching logic, but it is also inherently limiting. This change introduces a repeated list of "host simplification rules" at the RouteConfiguration level, that provide a way to substitute away other dynamic portions of the domain without changing what is sent upstream. For example, to match something like `*.foo.*.example.org` you might write a simplification rule like: `([^.]+[.]foo[.])([^.]+)([.]example[.]org)` with a substitution of `\1bar\3`. This then allows a virtual host domain entry of `*.foo.bar.example.org` to match `baz.foo.bar.example.org` or `wowza.foo.qux.example.org`. Host simplification rules are processed in the order they are defined. Signed-off-by: Ryan Anderson <[email protected]>

pugmajere · 2025-09-05T23:29:30Z

Rebased (for the last time, hopefully).

@alyssawilk may be interested in this, since I'm pretty sure this is copying an idea she implemented years ago for GFE.

kyessenov · 2025-09-05T23:33:15Z

api/envoy/config/route/v3/route.proto

+  // be used to implement multiple-wildcard matching, by converting
+  // all but one of the wildcards into a static string.
+  // This is similar to ignore_port_in_host_matching (above), but more flexible.
+  repeated type.matcher.v3.RegexMatchAndSubstitute host_simplification_rules = 18;


This could benefit from an example I think. regexes are pretty hard to understand (for most people), and we need to give some guidance.

Yeah, that's fair. I wrote more words, though I'm unsure on how the formatting will work out.

I also slightly improved the example from the one I used in the tests by putting the . outside the capture groups, so it would be more obvious what the substitution itself was doing.

What will happens if there are multiple rules? Will the final result will be used to find vhost or every rule's result will be used one by one?

And at least give it a meaning for name like host_rewrite_for_matching or something.

final result; I think the updated comments explain that better now, at least.

How about simplification_rules_for_host_matching?
I'm wary of host_rewrite... because it doesn't do the rewriting (intentionally) for the upstream request.... though I suppose these rules don't have to be used for "simplification", either.

Along with some more words of explanation on how this all works. Signed-off-by: Ryan Anderson <[email protected]>

pugmajere · 2025-09-06T03:42:05Z

Do the GCC CI builds use a different regex engine?

Signed-off-by: Ryan Anderson <[email protected]>

api/envoy/config/route/v3/route.proto

adisuissa · 2025-09-08T20:25:01Z

api/envoy/config/route/v3/route.proto

+  // An example may help:
+  // > host_simplification_rules:
+  // > - pattern:
+  // >    regex: "^(foo)[.]([^.]+)[.](example[.]org)$"


I remember seeing some discussions about the disadvantages of using general-regex matching (for non-prefix/non-suffix) use-cases, but I cannot recall the details, and whether it was related to host-matching.
cc @yanavlasov @mattklein123 may have more context.

I saw a proposal a while back about using domain_regex as a list in virtual hosts; this is arguably somewhat similar, but hopefully done in a way that the host matching algorithm itself can be changed to use alternate structures (a trie comes to mind, for example), where a list of regexs in the virtualhosts doesn't allow that.

Signed-off-by: Ryan Anderson <[email protected]>

pugmajere · 2025-09-12T05:51:01Z

I'm not sure I understand how the fuzz tests work, and that seems to be failing on the corpus step. Is there a reference to how those are supposed to be updated?

I don't understand why moving the lower casing to earlier makes the test pass, but it seems to, and, well, it makes as much sense as why they were failing in the first place. Signed-off-by: Ryan Anderson <[email protected]>

Signed-off-by: Ryan Anderson <[email protected]>

wbpcode

Thanks for the contribution. This do bring new complexity to our core route logic.

I am thinking is that possible to use the early_header_mutation_extensions to cover this requirement? You can complete the rewrite of the host header in the extension and use the host header after rewrite for vhost searching.
If you don't want to modify the actual host value. We can add a new vhost_header field at the RouteConfigration to specify the header key that be used to get host value for vhost searching. (by default, :authority/host). Then you can store the rewritten value in the specified header.

A simple vhost_header would much easier to understand and could also achieve your target.

wbpcode · 2025-09-18T02:17:00Z

api/envoy/config/route/v3/route.proto

+  // be used to implement multiple-wildcard matching, by converting
+  // all but one of the wildcards into a static string.
+  // This is similar to ignore_port_in_host_matching (above), but more flexible.
+  repeated type.matcher.v3.RegexMatchAndSubstitute host_simplification_rules = 18;


What will happens if there are multiple rules? Will the final result will be used to find vhost or every rule's result will be used one by one?

And at least give it a meaning for name like host_rewrite_for_matching or something.

pugmajere · 2025-09-19T23:36:11Z

I am thinking is that possible to use the early_header_mutation_extensions to cover this requirement? You can complete the rewrite of the host header in the extension and use the host header after rewrite for vhost searching. If you don't want to modify the actual host value. We can add a new vhost_header field at the RouteConfigration to specify the header key that be used to get host value for vhost searching. (by default, :authority/host). Then you can store the rewritten value in the specified header.

I don't see a way with the existing extensions to copy/mutate the value of one header by taking input from a different header, so that would also mean at least one new extension as well, possibly a few new extensions.

A simple vhost_header would much easier to understand and could also achieve your target.

If it was just a which header to use for host matching option, I'd agree, but adding early_header_mutation_extensions to the mix seems like it ends up being just as complicated, if not worse.

KBaichoo · 2025-09-29T17:35:05Z

ping @wbpcode

I think this is waiting for your input

pugmajere · 2025-09-29T23:17:10Z

I've also created #41280 for the alternate approach.

I think I still like this one better, because it's more obvious how to use it to support multiple wildcards, but I should be able to work with the other.

wbpcode · 2025-10-03T08:41:34Z

sorry for the delay. I am OOO for vacation.

I don't see a way with the existing extensions to copy/mutate the value of one header by taking input from a different header, so that would also mean at least one new extension as well, possibly a few new extensions.

Substitution formatter could be used in the header mutation rules. It could help you to mutate header with other headers/attributes.

wbpcode · 2025-10-03T08:44:58Z

If it was just a which header to use for host matching option, I'd agree, but adding early_header_mutation_extensions to the mix seems like it ends up being just as complicated, if not worse.

complexity will not disappear. But at least we can move it out from the core.

repokitteh-read-only bot added the api label Aug 28, 2025

repokitteh-read-only bot assigned adisuissa Aug 28, 2025

pugmajere force-pushed the randerson-host-simplification-rules branch from 697975a to 35ee254 Compare September 5, 2025 21:20

pugmajere changed the title ~~Introduce host simplification rules~~ router: Introduce host simplification rules Sep 5, 2025

pugmajere changed the title ~~router: Introduce host simplification rules~~ http conn man: Introduce host simplification rules Sep 5, 2025

pugmajere marked this pull request as ready for review September 5, 2025 23:25

pugmajere force-pushed the randerson-host-simplification-rules branch from 35ee254 to e6f6f9d Compare September 5, 2025 23:28

kyessenov reviewed Sep 5, 2025

View reviewed changes

Add an example to the proto definition

3bd0d82

Along with some more words of explanation on how this all works. Signed-off-by: Ryan Anderson <[email protected]>

pugmajere added 3 commits September 5, 2025 20:46

Fix the next-free-field

024a9e2

Signed-off-by: Ryan Anderson <[email protected]>

docs? I have no idea

1cae50f

Signed-off-by: Ryan Anderson <[email protected]>

docs2? I have no idea

880b39c

Signed-off-by: Ryan Anderson <[email protected]>

adisuissa reviewed Sep 8, 2025

View reviewed changes

wrap changelog properly

f6b40c3

Signed-off-by: Ryan Anderson <[email protected]>

pugmajere added 3 commits September 16, 2025 07:10

Try to make the tests pass under gcc

4a37f22

I don't understand why moving the lower casing to earlier makes the test pass, but it seems to, and, well, it makes as much sense as why they were failing in the first place. Signed-off-by: Ryan Anderson <[email protected]>

Keeping references to disappearing variables is bad, mmm k?

0926cca

Signed-off-by: Ryan Anderson <[email protected]>

Update comments about multiple rules

6eb0a2d

Signed-off-by: Ryan Anderson <[email protected]>

wbpcode reviewed Sep 18, 2025

View reviewed changes

wbpcode self-assigned this Sep 18, 2025

wbpcode added waiting api and removed api labels Oct 3, 2025

http conn man: Introduce host simplification rules #40874

Are you sure you want to change the base?

http conn man: Introduce host simplification rules #40874

Uh oh!

Conversation

pugmajere commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

repokitteh-read-only bot commented Aug 28, 2025

Uh oh!

repokitteh-read-only bot commented Aug 28, 2025

Uh oh!

pugmajere commented Aug 28, 2025

Uh oh!

pugmajere commented Sep 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pugmajere commented Sep 6, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pugmajere commented Sep 12, 2025

Uh oh!

wbpcode left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pugmajere commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KBaichoo commented Sep 29, 2025

Uh oh!

pugmajere commented Sep 29, 2025

Uh oh!

wbpcode commented Oct 3, 2025

Uh oh!

wbpcode commented Oct 3, 2025

Uh oh!

Uh oh!

pugmajere commented Aug 28, 2025 •

edited

Loading

wbpcode left a comment •

edited

Loading

pugmajere commented Sep 19, 2025 •

edited

Loading