More general python codegen #1053
Conversation
ponyisi
requested review from
BenGalewsky,
kyungeonchoi,
gordonwatts and
MattShirley
|
I don't understand how this works? How does the sidecar know to pick up the user specified file? |
|
@BenGalewsky The sidecar tells the codegen shim what file name to produce. This PR introduces a route where user code can just directly produce this file, instead of returning an awkward array which the codegen shim then writes to that file. |
| output = generated_transformer.run_query(file_path) | ||
|
|
||
| ttime = time.time() | ||
| if output_format == 'root-file': |
There was a problem hiding this comment.
What about parquet, or all the other output formats that are allowed?
There was a problem hiding this comment.
Well, this is why we should really have a "raw" option for the output format. The most obvious need for this code path is running RDataFrame transformations, where the output is in fact going to be a ROOT file.
Conceptually I guess the big question is whether we want to try to share the Python transformer code between the uproot science image (where a "translate awkward output to root/parquet" step is natural) and the C++ ROOT image (where in the end the user writing the Python code is responsible for the output ... unless we ask them to return an RDF object for snapshotting, or something - note this isn't an entirely implausible route, it's possible to do RDF -> Awkward -> Parquet).
There was a problem hiding this comment.
Ah, that is a good point. In my mind I was thinking this was common code.
We have a capabilities matrix, right - so the user can't request an unsupported format? Or if you request parquet from somethign that doesn't support it you get back root? I can't remember how we handle that now (in the old days, we'd just make best effort to pay attention to that request).
Allow the function signature
in addition to
so Python transformers can create arbitrary ROOT file output.