Skip to content

Conversation

tobil4sk
Copy link
Member

This is based on the nekotools boot -c feature from neko. It allows running:

haxelib run hashlink boot main.hl

Which generate main.c, which is just the hl entry point with the bytecode embedded. It is then possible to compile this file with a simple compilation command:

cl.exe main.c /MD /I C:\HaxeToolkit\hashlink\include /link /libpath:"C:\HaxeToolkit\hashlink" libhljit.lib libhl.lib user32.lib
#or
gcc -o main main.c -lhljit -lhl -lm -Wl,-rpath -Wl,/usr/local/lib

This is a bit like hlboot.dat, however it has some advantages:

  • No unwanted cli handling
  • Executable and bytecode are in a single file
  • No arbitrary bytecode file being loaded and executed from CWD

Compared with hlc:

  • It has a much faster compilation time
  • Runtime performance may be slower than hlc, as it still runs with jit.
  • Unlike hlc this only supports x86, again due to the jit.

To make this work, I had to add a libhljit static library, which just contains everything in hl.exe except main.c.

These can be used to disable the debugger, profiler, or hot reload if it
is not desired.
It adds an extra dependency for linking
hlmodule.h and opcodes.h are already installed to `.../include/hl` to
avoid poluting the root include folder. It is easier to deal with this
for `hashlink boot` if the repository matches this structure.
@ncannasse
Copy link
Member

Instead of requiring a libhljit installed locally, why not simply distributing the required files as part of the haxelib projet, then packing them together into a single .c file (with bytecode) that can be compiled with no other dependency than libhl ?

@tobil4sk
Copy link
Member Author

Instead of requiring a libhljit installed locally, why not simply distributing the required files as part of the haxelib projet, then packing them together into a single .c file

A single main.c that includes everything would be quite a bit slower to compile. The current small c file with the precompiled static libhljit gives for a very quick compilation time.

I guess as a middle ground, the source files could be distributed with the haxelib and haxelib run hashlink boot could precompile the files on the first run into an internal static libhljit so it doesn't have to be compiled again. The downside is that the tool has to know how to compile all the other files, which is more involved than compiling a simple main.c, so some makefiles/vcxproj files would have to be distributed too.

This would mean we don't have to depend on the stability of libhljit, which is a benefit as libhl is probably more likely to be stable (since hlc also depends on its stability). This would make it less of a problem if the hashlink haxelib is out of sync with the libhl installed on the system.

@ncannasse
Copy link
Member

ncannasse commented Jan 18, 2025

A single main.c that includes everything would be quite a bit slower to compile.

I don't think so, C is very fast to compile and there's only few files for the JIT. Unless you're looking at something like 100ms optimization ? Give it a try, I'm sure it will be almost unnoticeable.

@tobil4sk
Copy link
Member Author

Give it a try, I'm sure it will be almost unnoticeable.

I'm compiling haxelib for testing:

# Small run.c with libhljit
$ time gcc -o haxelib run.c -O3 -lhljit -lhl -lm -Wl,-rpath -Wl,/usr/local/lib
gcc -o haxelib run.c -O3 -lhljit -lhl -lm -Wl,-rpath -Wl,/usr/local/lib  0.22s user 0.03s system 99% cpu 0.251 total

# Merged run.c
$ time gcc -o haxelib run.c -O3 -lhl -lm -Wl,-rpath -Wl,/usr/local/lib
gcc -o haxelib run.c -O3 -lhl -lm -Wl,-rpath -Wl,/usr/local/lib  3.43s user 0.04s system 98% cpu 3.511 total

@ncannasse
Copy link
Member

ncannasse commented Jan 18, 2025 via email

@Simn
Copy link
Member

Simn commented Jan 18, 2025

I'd actually be interested in having both options.

@ncannasse
Copy link
Member

ncannasse commented Jan 18, 2025 via email

@tobil4sk
Copy link
Member Author

tobil4sk commented Jan 18, 2025

Given it's nice to have a single C code for distribution I think it's still worth the 3 extra seconds.

The goal is to replace nekotools boot -c for haxelib compilation with a hashlink solution. Having this delay is a noticeable downgrade from how it is now with neko.

I think it's a valid concern to not want libhljit to be distributed with the hashlink binaries, since there is no guarantee of stability. It makes more sense to have the sources distributed with the hashlink haxelib, then the only concern is whether the hashlink haxelib is compatible with the libhl installed, which is a simpler requirement (that is shared by hlc).

Ultimately, the details of main.c are not really important here, it is an implementation detail that the user shouldn't have to worry about. If automatic compilation is added to the boot command, we could hide all those concerns and just take a main.hl and output main.exe.

haxelib run hashlink boot run.hl

Then the tool is free to decide how to deal with the files, whether to combine them into one or to do it separately with intermediate object files.

How about the following changes to the PR:

  • libhljit is removed from the installation
  • the required sources for compiling a stripped hashlink jit are distributed with the hashlink haxelib
  • git versions will just use the hashlink sources directly (can also support HASHLINK environment variable)
  • hashlink boot compiles immediately instead of giving you a c file to compile manually

@tobil4sk
Copy link
Member Author

One nice thing could also be to compile the exe entirely statically as we do on consoles so we have a single binary with no libhl dependency

I know it is possible to compile everything statically for hlc, but is this possible at all with jit? I tried it but it failed when trying to load std functions, so I assumed that method only works for hlc. If it could work, that would also be useful for quickly making a haxelib binary with no libhl dependency (assuming static libhl and all hdlls are pre-compiled).

@skial skial mentioned this pull request Jan 20, 2025
1 task
@ncannasse
Copy link
Member

One nice thing could also be to compile the exe entirely statically as we do on consoles so we have a single binary with no libhl dependency

I know it is possible to compile everything statically for hlc, but is this possible at all with jit? I tried it but it failed when trying to load std functions, so I assumed that method only works for hlc. If it could work, that would also be useful for quickly making a haxelib binary with no libhl dependency (assuming static libhl and all hdlls are pre-compiled).

We would need to try to load in the current binary before trying to load the hdll, it should be safe to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants