-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Add JSON annotation in internal/gqlparser/ast to Position fields #7509
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add JSON annotation in internal/gqlparser/ast to Position fields #7509
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Amazing find! Excellent work 👏 I hope it's accepted upstream too.
As for testing, I guess adding a test just to verify that parsing some schemas don't cause infinite recursion (as would happen before the fix) might make some sense?
It didn't turn out to be a cycle as I previously speculated. It's just using a ton of memory because each JSON node includes a copy of the entire original source file. In the AST this is just a pointer to a single copy, but when JSON marshaled each Position includes the whole source file. I'll add a self contained test to the graphql caching PR based off this updated reproducer. EDIT: The commit with the test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! 😃
Annotate internal/gqlparser structs not to include Position when marshaled to JSON. This makes the JSON roundtrip succeed without allocating a ton of memory for JSON fields that will be subsequently pruned. Upstream PR: vektah/gqlparser#364 $ time ./reproducer-pre-fix Now passing 1262568 bytes to builtinGraphQLParseSchema() to reproduce the issue in ast.InterfaceToValue() Alloc = 4159 MiB TotalAlloc = 5625 MiB Sys = 5584 MiB NumGC = 18 Alloc = 8312 MiB TotalAlloc = 11169 MiB Sys = 11125 MiB NumGC = 19 Alloc = 8312 MiB TotalAlloc = 11169 MiB Sys = 11125 MiB NumGC = 19 Alloc = 16615 MiB TotalAlloc = 22247 MiB Sys = 22209 MiB NumGC = 20 Alloc = 16615 MiB TotalAlloc = 22247 MiB Sys = 22209 MiB NumGC = 20 Alloc = 16615 MiB TotalAlloc = 22247 MiB Sys = 22209 MiB NumGC = 20 Alloc = 38765 MiB TotalAlloc = 44397 MiB Sys = 44377 MiB NumGC = 20 Alloc = 33223 MiB TotalAlloc = 44397 MiB Sys = 44377 MiB NumGC = 21 $ time ./reproducer-post-fix Now passing 1262568 bytes to builtinGraphQLParseSchema() to reproduce the issue in ast.InterfaceToValue() real 0m0.545s user 0m0.201s sys 0m0.028s Signed-off-by: Rob Myers <[email protected]>
3ddc2a0
to
a10f918
Compare
✅ Deploy Preview for openpolicyagent ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Annotate internal/gqlparser/ast structs not to include Position when marshaled to JSON.
This makes the JSON roundtrip succeed without allocating a ton of memory for JSON fields that will be subsequently pruned.
I have also submitted this change for inclusion in the upstream gqlparser project.