Add function to generate back textual syntax from AST

[miguel76]

Implements the feature described in #28
The new function ast_tostring serializes an input AST (as generated by parse) in jq textual form.
It strives to cover all the features currently supported by jqjq

[wader]

Hey, thanks for adding serialize support! i'm a bit curious about your use-case, if possible are you able to share some more details? i'm doing some AST-rewriting for https://github.com/wader/fq using a gojq fork (this commit in the fq branch of my gojq fork) and wondering if could be a better fit? jqjq might be quite slow, has very poor error handling at the moment and lack some features like include/import but it of course depends on your needs :)

[wader]

Style nitpick: snake_case, but i'm thinking let's ignore most style stuff and i can reformat things after merge

[wader]

Is it too slow to always be done? also thinking if we should do some kind of round-trip assert? ex ast_tostring | . == (lex | parse | ast_tostring)

[wader]

Also thinking if there should be some .test-tests? round-trip serialize _builtins_src? 🤔

[miguel76]

It is not so slow, actually, I just wanted to add this test without being "too invasive". I agree it could be setup far better. Right now it was a quick way to use the tests for the new function. And it was useful cause that way I found many bugs that I fixed in the last commit. Now all those tests pass, i.e. the results are the same after the round-trip serialize-parse-eval

[wader]

Played around a bit with the PR, noticed this:

$ jq -n -L . 'include "jqjq"; _builtins_src | lex | parse | ast_tostring | lex | parse'
jq: error (at <unknown>): tokens left: [[{"ident":"splits"},{"lparen":"("},{"binding":"$re"},{"semi
...

Can try minimize it down a bit later, out traveling atm

[miguel76]

Replying to your previous comment...
Thanks for the interest, my purpose is to experiment with jq AST rewriting to embed in a query other functions (like API calls), so that jq itself could work as a glue. In order to execute such queries I need to manipulate the query structure.
I think gojq would be really useful for my purposes and I had a quick look of it. The difficulty I have seen is that it does not offer parsing/serialization from the command line interface (I wonder if you plan to support it in your fork).
So that would tie me to work with Go, which I would need to learn just for that purpose.
Ideally, in my project I would like to execute jq and other tools from Python or from shell.
jqjq has the advantage of being programming-language agnostic.
Nevertheless if I find a way to use gojq parsing/serialization without having to learn another programming language, I would certainly go for that :)

[wader]

Replying to your previous comment... Thanks for the interest, my purpose is to experiment with jq AST rewriting to embed in a query other functions (like API calls), so that jq itself could work as a glue. In order to execute such queries I need to manipulate the query structure.

I see, what kind of rewrites do you think you would need? for fq i think i mainly do two types of rewrites, turn <expr> into ... | try <expr> catch ... | ... and <expr> | repl into repl(<AST-for-expr>)

I think gojq would be really useful for my purposes and I had a quick look of it. The difficulty I have seen is that it does not offer parsing/serialization from the command line interface (I wonder if you plan to support it in your fork). So that would tie me to work with Go, which I would need to learn just for that purpose. Ideally, in my project I would like to execute jq and other tools from Python or from shell. jqjq has the advantage of being programming-language agnostic. Nevertheless if I find a way to use gojq parsing/serialization without having to learn another programming language, I would certainly go for that :)

My gojq fork adds support for parsing/serialization and then in fq i add some extra jq builtins to use it, see https://github.com/wader/fq/blob/19b9446f82f5b5df553f9aaa1b178b240d515179/pkg/interp/query.go#L10 so in the end you can do things like this from jq.

# _query_try, _query_func etc are some rewrite-helpers in fq
$ fq -rn '"1,2" | _query_fromstring | _query_try(_query_array; _query_func("debug")) | _query_tostring'
try [1, 2] catch debug

So if you're fine with using some kind of fork of gojq that might be an alternative. I can help you out with the go code needed so you can do everything from jq.

But of course improving jqjq's langauge support, robustness and error handling would be nice also :)

Let me know how it goes!

[miguel76]

I started experimenting with fq to do the parsing, it works perfectly.
I think I can move to fq for the parsing considering that you also suggest so and support for import/include syntax is important to me
I think it may be still useful to complete jqjq parse implementation so that is interoperable with the gojq/fq implementation.

For the AST handling/serialization, in my case I would like to include multiple interlinked jq queries, so it is good for me to have control of the traversal and serialization of the AST, albeit I guess also on that side I could do some interesting things with fq (which, by the way, seems to me an amazing project).

In the meanwhile I developed (but needs to be tested thoroughly) a generic AST traversal function roughly following the visitor pattern, to facilitate AST operations in jq. Consequently I also actually made a new implementation of the serialization based on that generic traversal function.
I could contribute both functions here (If you/we deem it useful) or publish them in another repo.

Regarding the AST representation in JSON as generated by gojq/fq, is it documented somewhere? How is it different (if it is different) from the one you use in jqjq?

[wader]

I started experimenting with fq to do the parsing, it works perfectly. I think I can move to fq for the parsing considering that you also suggest so and support for import/include syntax is important to me I think it may be still useful to complete jqjq parse implementation so that is interoperable with the gojq/fq implementation.

👍 and you can always switch to something more lightweight in the future if needed.

For the AST handling/serialization, in my case I would like to include multiple interlinked jq queries, so it is good for me to have control of the traversal and serialization of the AST, albeit I guess also on that side I could do some interesting things with fq (which, by the way, seems to me an amazing project).

Include as in pipe/chain multiple queries somehow? should be very possible, composability is one of many things jq really sines at.

https://github.com/wader/jq-lsp has a query_walk function but not sure it fit your needs, it walks and keep track of lexical scope and conditionally output {q: <query node>, env: <current scope>} objects.

Thanks for the kind words about fq, much of the credit should go to jq :)

In the meanwhile I developed (but needs to be tested thoroughly) a generic AST traversal function roughly following the visitor pattern, to facilitate AST operations in jq. Consequently I also actually made a new implementation of the serialization based on that generic traversal function. I could contribute both functions here (If you/we deem it useful) or publish them in another repo.

Sounds like it could be useful so please share. One think I've thought about is serialise with indent/style support so that one could write a jq code formatter, but it might be quite tricky.

Regarding the AST representation in JSON as generated by gojq/fq, is it documented somewhere? How is it different (if it is different) from the one you use in jqjq?

Not documented anywhere unfortunately. It's gojq's internal AST-representation as JSON more or less. Both fq och jqjq uses it. I don't it will change much in the future but if you wan't something very stable I guess you should stick to a specific version of fq or build your own gojo-based thingy.