Can't we just substitute application code for built-in repl code?

[guest271314]

When node is run at the command line we get a REPL session

If we run the node command without any script to execute or without any arguments, we start a REPL session:

node

Since it is unlikely an application will be using the built-in REPL can't we just substitute application code for the built-in REPL code, so that all an end-user has to do is call node for the application to run?

[jviotti]

I think the problem is that the REPL code is implemented in C++, and not as a v8-based JavaScript program that we can replace. @RaisinTen might be able to confirm.

[aduh95]

REPL is a JS program: https://github.com/nodejs/node/blob/2178f1772f4f061e780780e99c709f94eb4b96cf/lib/internal/main/repl.js
C++ is involved to select what JS entry point to use (https://github.com/nodejs/node/blob/98fa488066b92a257d7d37f4ed24804850277594/src/node.cc#L505-L507), it could easily be replaced at compile time.

[jviotti]

Oh, interesting. I didn't know that. I have many questions then @RaisinTen:

• How are these files bundled in the Node.js executable?
• Does the REPL consist of a single file or multiple of them? If it's the latter, how are they bundled together?
• Is there any size/offset definition in the injected files that we can change and replace?

[RaisinTen]

How are these files bundled in the Node.js executable?

It's done by js2c - https://github.com/nodejs/node/blob/472edc775d683aed2d9ed39ca7cf381f3e7e3ce2/tools/js2c.py. If you do a release build of node, you'll see out/Release/obj/gen/node_javascript.cc which contains a variable named internal_main_repl_raw that is assigned the contents of the lib/internal/main/repl file as a static const uint8_t array. The other occurrence of the variable in this file is the one which does source_.emplace("internal/main/repl", UnionBytes{internal_main_repl_raw, 1842});, to assign this array to the "internal/main/repl" key in BuiltinSourceMap - https://github.com/nodejs/node/blob/472edc775d683aed2d9ed39ca7cf381f3e7e3ce2/src/node_builtins.h#L125, so that it can be required internally using require('internal/main/repl').

Does the REPL consist of a single file or multiple of them? If it's the latter, how are they bundled together?

I would say multiple files because it requires some key components like evalScript (https://github.com/nodejs/node/blob/7f5679b901b28c9dd330a4f7391da626602452f2/lib/internal/process/execution.js#L53-L104) which is required from internal/process/execution in here - https://github.com/nodejs/node/blob/7f5679b901b28c9dd330a4f7391da626602452f2/lib/internal/main/repl.js#L12-L14.

Is there any size/offset definition in the injected files that we can change and replace?

Injecting application code here would be the same as making changes to the source strings in the binary because there's no documented size/offset we can change to do this in the current state AFAIK. Another point to note here is that if we inject the application code here directly, it would be able to access the internal files like require('internal/main/repl'), so that would require us to run a bootstrap script which modifies fs to read from a vfs and then do something like require('internal/modules/cjs/loader').Module.runMain(process.execPath); (https://github.com/nodejs/node/blob/7f5679b901b28c9dd330a4f7391da626602452f2/lib/internal/main/run_main_module.js#L23) to set up the require() function that gets exposed to userland and finally run the application.

[guest271314]

@aduh95 Then this should be as simple as https://github.com/beaufortfrancois/download-chromium where node nightly source is used as the base source, user navigates to the page, uploads their JavaScript file(s), repl.js is replaced with the file(s), compiled on Google App Engine or comparable methods/means, code signed programatically #14 (comment)

Because code signatures operate on the binary format, any data section outside of the boundaries of the format is either not protected by the signature at all, or the OS will simply not run it (in the case of macOS)

and the node executable is output for user to download.

[RaisinTen]

Can't we just substitute application code for built-in repl code?

The node binaries are probably already going to have the logic to detect whether application code has been injected into it and execute either that or the repl code. That's what nodejs/node#42334 does already.

compiled on Google App Engine

That's definitely a possibility. If node --compile or something like that gets implemented anyone could create such a thing but I'm not sure why that would be useful.

[guest271314]

What I mean is just substitute/replace the repl cofe (JavaScript text) in the existing executable with arbitrary code supplied by user (JavaScript text). That way the user just call node and their replaced JavaScript cide is executed by node, just as calling node on command line starts repl.

[cspotcode]

What happens when they do node foo.txt does the repl still run? Or is foo.txt executed as JS?

[jviotti]

Not sure about the proposed approach in this thread, but the overall SEA idea would be to pass foo.txt as the first argument to the program embedded into the executable.

[jesec]

There are many concerns with the approach. The ones I can think of right now:

• Command arguments handling. (we want node app.js -- --arg -> ./sea --arg)
• Baking in and handling Node options. Some options must be loaded in the C++ world, before we have V8.
• REPL environment is not the same as normal. You may have noticed that you can use many built-ins in REPL without explicitly importing them.
• It is challenging to properly manipulate existing data/section in the binary post-link. Different compilers/linkers could embed data in different ways, and a marker is not necessarily present. We don't have tools to do that automatically and deterministically for all required platforms.

For the first 3 points, some may argue that logics can be added to handle the case, but that brings the complexity to the parallel of what we are already working on.

[jesec]

Indeed. We want to allow users (developers who pack the SEA) to bake Node options into the single executable. End users (users who run the SEA) shall not have control over Node options (via NODE_OPTIONS env var or anything else).

[guest271314]

If the JavaScript is in the node executable I'm relatively sure I can just use the browser to modify the file, something like

fetch('node')
.then((response) ==> response.text())
.then((text) => text.replace(/regex/,'const sea = []'))
.then((text) => new TextEncoder().encode(text))
.then((uint8) => showSaveFilePicker())
.then((handle)) => { /* write file */})

Done.

There are many concerns with the approach. The ones I can think of right now:

• Command arguments handling. (we want node app.js -- --arg -> ./sea --arg)

I thought the whole point was making a single executable application, not passing arguments to the executable whatsoever.

#16 (comment)

I don't think it counts as a single executable if you have to invoke it with specific parameters. Otherwise any zero-dependency node program could qualify merely by using -e.

We might as well just call node example.js if we are going to pass parameters to the executable. We've done nothing different from the EU perspective, I'm speaking for myself. Doesn't really make sense to me to pass arguments to a single-executable - it only does 1 thing. You want a different executable, bake the parameter into the executable and be done with it.

• Baking in and handling Node options. Some options must be loaded in the C++ world, before we have V8.

Well, we can do that too. The whole node options is just what we already have. Nothing novel about a single executable that behaves the same way as node exeuctable, or existing Node.js patterns. For me the idea is to use as little node as possible - the single executable.

• REPL environment is not the same as normal. You may have noticed that you can use many built-ins in - REPL without explicitly importing them.
  It is challenging to properly manipulate existing data/section in the binary post-link. Different compilers/linkers could embed data in different ways, and a marker is not necessarily present. We don't have tools to do that automatically and deterministically for all required platforms.

I'm saying take the existing executable, and abuse it to make it do what you want after the fact, by simple manipulation of text in the executable.

If we are going to customize node as a base source single executable application, we might as well just create a form on Node.js download Website, user fills in what they want, upload their application, and Node.js WG/Team, whoever is in charge of that domain, spits out the node executable customized: custom node builds. Done.

What I am reading appears to be a circular argument for unlimited options and indeed passing options to a single executable - leading right back to why not just use node executable and pass file name?

[ovflowd]

We might as well just call node example.js if we are going to pass parameters to the executable. We've done nothing different from the EU perspective, I'm speaking for myself. Doesn't really make sense to me to pass arguments to a single-executable - it only does 1 thing. You want a different executable, bake the parameter into the executable and be done with it.

Isn't this a little bit too restrictive?

[jviotti]

Doesn't really make sense to me to pass arguments to a single-executable - it only does 1 thing

The idea is to support building standalone CLIs built using Node.js. Almost every useful CLI out there does take arguments/options.

[ovflowd]

The idea is to support building standalone CLIs built using Node.js. Almost every useful CLI out there does take arguments/options.

Exactly!

[guest271314]

For clarity, objectively this is the same; equivalent; no different:

• Command arguments handling. (we want node app.js -- --arg -> ./sea --arg)

The application needs to handle dynamics, if any. Remember the application is already built. It just happens to use node JavaScript runtime engine; the algorithm could easily use Python, C, C++, even Bash.

[cspotcode]

The -- might cause some confusion by implying that node treats -- specially when it does not.

[guest271314]

@cspotcode I don't think there is confusion. My understanding of a single executable application is no arguments/parameters need to be passed and are not expected to receive any parameters to the runtime itself. Ideally, we just call node with the custom node build incorporating the self-contained, non-dependent (on anything outside of the runtime executable and application) - else, the moment you pass environmental variables to node outside of the SEA, or pass arguments to the application side - the questions must be asked and answered: How is this different from node example.js, and therefore why not just use node example.js?

[ljharb]

The point of an SEA is that nobody should have to, or be easily able to, have any idea whatsoever that the executable is built with node. Anything that's node-specific should no longer exist, in other words, by the time the executable is created.

There's no value in providing a way to bundle node + a JS bundle in a single executable if the end user has to know how node works - if they know that, they already have node.

[ovflowd]

There's no value in providing a way to bundle node + a JS bundle in a single executable if the end user has to know how node works - if they know that, they already have node.

I'd also agree with this. I would want to have a SEA distributed with Node.js to be as normal as any other *nix-like or Windows-like binary. Something I just execute without thinking about the specifics of Node.js environments, parameters or options.

[guest271314]

@ljharb I agree to some extent. I already have node, yet am not interested in npm, and packages. I have the same algorithm working in C, C++, Python, Bash, PHP, Node.js and QuickJS. While performance of v8 is faster than QuickJS per the record, you can't tell the difference between the source code when capturing and streaming audio (which is hard because any and all lags, etc. are perceptible in clipping, crackling, etc.).

I think we can achieve the goal here rather simply by providing a simple HTML form on the downloads Web site that accepts one or more files, writes the files to where repl would run with node. That does not mean use the repl, just use the same means that repl does to run node at command line, substituting the application therefore. Build node executable for the single user. Done.