Skip to content

Add function to generate back textual syntax from AST #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add function to generate back textual syntax from AST #29

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
972a442
Add function to generate back textual syntax from AST
miguel76 Apr 15, 2025
c2951ba
Add test for serialization and fix mapping of function args
miguel76 Apr 15, 2025
2411113
Fix all errors found with tests
miguel76 Apr 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 143 additions & 3 deletions jqjq.jq
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2090,6 +2090,139 @@ def eval_ast($query; $path; $env; undefined_func):
def eval_ast($ast):
eval_ast($ast; []; {}; undefined_func_error);

def ast_tostring:
def _pattern:
if .name then .name
elif .array then "[\([.array[] | _pattern] | join(", "))]"
elif .object then
"{\(
[.object[] | "\(
if .key then .key else .key_string.str | tojson end
)\(
if .val then ": \(.val | _pattern)" else "" end
)"] |
join(", ")
)}"
else error("unsupported type of pattern: \(.)")
end;
def _f:
def _index:
".\(
if .start or .is_slice then
"[\(
if .start then .start | _f else "" end
)\(
if .is_slice then ":" else "" end
)\(
if .end then .end | _f else "" end
)]"
elif .name then .name
elif .str then .str.str | tojson
else error("unsupported type of index: \(.)")
end
)";
. as {term: {type: $type}, $op, $func_defs}
| "\(
if $func_defs then [
$func_defs.[]
| "def \(.name)\(
if .args then "(\(.args | join("; ")))"
else ""
end
): \(.body | _f); "
] | join("")
else ""
end
)\(
if $type then .term |
"\(
if $type == "TermTypeNull" then "null"
elif $type == "TermTypeNumber" then .number | tostring
elif $type == "TermTypeString" then
if .str then
.str | tojson
else
[
.queries.[] | _f |
if .[0:1] == "\""
then .[1:-1]
else "\\\(.)"
end
] | join("") | "\"\(.)\""
end
elif $type == "TermTypeTrue" then "true"
elif $type == "TermTypeFalse" then "false"
elif $type == "TermTypeIdentity" then "."
elif $type == "TermTypeUnary" then .unary | "\(.op)\({term: .term} | _f)"
elif $type == "TermTypeIndex" then .index | _index
elif $type == "TermTypeFunc" then "\(.func.name)\(
if .func.args then
[.func.args.[] | _f] | join("; ") | "(\(.))"
else
""
end
)"
elif $type == "TermTypeObject" then
( .object.key_vals
| "{\(
[ .[] |
"\(
if .key then .key
elif .key_string then .key_string.str | tojson
else .key_query | _f
end
)\(
if .val then
": \([.val.queries.[] | _f] | join(", "))"
else ""
end
)"
] | join(", ")
)}" )
elif $type == "TermTypeArray" then .array | "[\(if .query then .query | _f else "" end)]"
elif $type == "TermTypeIf" then .if |
"if \(.cond | _f) then \(.then | _f) \(
if .elif then [
.elif.[] |
"elif \(.cond | _f) then \(.then | _f) "
] | join("")
else ""
end
)\(if .else then " else \(.else | _f)" else "" end) end"
elif $type == "TermTypeReduce" then .reduce |
"reduce \({term: .term} | _f) as \(.pattern | _pattern) (\(.start | _f);\(.update | _f))"
elif $type == "TermTypeForeach" then .foreach |
"foreach \({term: .term} | _f) as \(.pattern | _pattern) (\(.start | _f);\(.update | _f)\(
if .extract then ";\(.extract | _f)" else "" end
))"
elif $type == "TermTypeQuery" then "(\(.query | _f))"
elif $type == "TermTypeFormat" then "\(.format) \(.str | _f)"
elif $type == "TermTypeTry" then .try |
"try \(.body | _f)\(if .catch then " catch \(.catch | _f)" else "" end)"
else error("unsupported term: \(.)")
end
)\(
if .suffix_list then
[
.suffix_list[] |
if .index then .index | _index
elif .bind then .bind | " as \([.patterns[] | _pattern] | join(", ")) | \(.body | _f)"
elif .iter then "[]"
elif .optional then "?"
else error("unsupported suffix type: \(.)")
end
] | join("")
else ""
end
)"
elif $op then
(if $op | . == "," then "" else " " end) as $pad
| "\(.left | _f)\($pad)\($op) \(.right | _f)"
else error("unsupported query: \(.)")
end
)";
[_f] | join("");

def _builtins_src: "
def debug(msgs): (msgs | debug | empty), .;
def halt_error: halt_error(5);
Expand Down Expand Up @@ -2645,6 +2778,7 @@ def usage:
+ " --jq PATH jq implementation to run with\n"
+ " --lex Lex EXPR\n"
+ " --parse Lex then parse EXPR\n"
+ " --serialize Expect an AST in EXPR and serialize it as jq\n"
+ " --repl REPL\n"
+ " --no-builtins Don't include builtins\n"
+ "\n"
Expand Down Expand Up @@ -2771,6 +2905,7 @@ def parse_options:
// option(null; "repl"; .mode = "repl")
// option(null; "lex"; .mode = "lex")
// option(null; "parse"; .mode = "parse")
// option(null; "serialize"; .mode = "serialize")
// option(null; "no-builtins"; .no_builtins = true)
//
( if .args.is_short then "-\(.args.curr[:1])" else "--\(.args.curr)" end
Expand Down Expand Up @@ -2949,7 +3084,7 @@ def jqjq($args; $env):
, "\n" # input interrupted so no line entered
);

def _run_tests:
def _run_tests($testSerialize):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style nitpick: snake_case, but i'm thinking let's ignore most style stuff and i can reformat things after merge

# read jq test format:
# # comment
# expr
Expand Down Expand Up @@ -3034,7 +3169,10 @@ def jqjq($args; $env):
| "line \(.line): \(.input | tojson) | \(.expr) -> \(.output | _to_unslurped)" as $test_name
| . as $test
| try
( ($test.expr | lex | parse) as $ast
( (
($test.expr | lex | parse)
| if $testSerialize then ast_tostring | lex| parse else . end
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it too slow to always be done? also thinking if we should do some kind of round-trip assert? ex ast_tostring | . == (lex | parse | ast_tostring)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also thinking if there should be some .test-tests? round-trip serialize _builtins_src? 🤔

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not so slow, actually, I just wanted to add this test without being "too invasive". I agree it could be setup far better. Right now it was a quick way to use the tests for the new function. And it was useful cause that way I found many bugs that I fixed in the last commit. Now all those tests pass, i.e. the results are the same after the round-trip serialize-parse-eval

) as $ast
| $test.input
| [ eval_ast(
$ast;
Expand Down Expand Up @@ -3129,8 +3267,10 @@ def jqjq($args; $env):
| if $opts.action == "help" then usage
elif $opts.mode == "lex" then $opts.program | lex, "\n"
elif $opts.mode == "parse" then $opts.program | lex | parse, "\n"
elif $opts.mode == "serialize" then $opts.program | ast_tostring, "\n"
elif $opts.mode == "repl" then _repl($opts)
elif $opts.action == "run-tests" then input | _run_tests
elif $opts.action == "run-tests-without-serialize" then input | _run_tests(false)
elif $opts.action == "run-tests" then input | (_run_tests(false), _run_tests(true))
else _filter($opts)
end
);