Add ExtensionDevelopmentPlugin for in-editor native extension development

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Extension development is relatively complex. It requires downloading a separate set of headers, generating them, setting up a compiler toolchain, etc. This happens independent of the usage of Godot from the editor, so development can be relatively difficult.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The idea is to have something called an "ExtensionDevelopmentPlugin" that eases the project generation, compilation, updating the headers and bindings. This way, everything can be properly done from the editor UI.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The new ExtensionDevelopmentPlugin will handle extension development. To develop an extension, a tab in the ProjectSettings will allow specifying project folders as well as the plugin used. As an example:

Path	Language
res://myextension/terrain	C++ [edit]
res://anoherextension	C# [edit]
res://yetanother	Rust [edit]

Creating an extension will make the plugin set up the proper toolchains, debug protocol, build system, build command, and everything needed. If something is not present, the plugin will verify this and let you know that its misconfigured (likely showing an error icon). Running the project, the plugin will also attempt to compile the extension and connect to an IDE for debugging.

It is important to note here that extensions that you already download compiled are not managed or tracked by this system. This system is only for extension development. That said, however, the new extension development plugin system makes the extensions in development interact with the other extensions installed by generating and providing the relevant headers (or API depending on the language) for their use.

As an example, in the config menu for an extension in development, you will be able to see a list of extensions installed and tick which ones you want to have access to. In C++ this means that the proper headers will be generated for using this extension. In C#, it means the proper .cs classes will be created to interact with it.

Finally, the extension plugin will also make sure to compile as many platforms as possible on deploy (in the case of compiled languages) but it will be the developer responsibility to install the required toolchains (there is a separate discussion ongoing about supporting GitHub CI for being able to build for every single platform, as well as also using webassembly optionally to ensure what is compiled runs everywhere, but I think this is material for other proposals).

Stock Godot will only come with support for C++ and C# development, but it is expected that you can download these plugins from the asset library, allowing support for development of other languages such as Nim, Rust, etc.

If this enhancement will not be used often, can it be worked around with a few lines of script?

This is core functionality.

Is there a reason why this should be core and not an add-on in the asset library?

This feature allows the creation of add-ons.

[fire]

I recommend the use of the https://github.com/mstorsjo/llvm-mingw package for toolchains for systems other than for mac. There's a different discussion about how to build for mac.

Benefits:

1. Small
2. Supports MSVC dbg symbols via a flag
3. Fast!
4. Can cross compile
5. We can distribute the package on the asset library.

[reduz]

@fire In this case, I think the user will be in charge of installing this, but our build templates should definitely support it.

[fire]

The idea is the user can go to the asset library, click download and start immediately developing Godot extensions in like c++.

There is no need to also then install msvc and click 5 windows or similar for mingw.

[reduz]

@fire Okay but in practice you have to remember that

• On Linux you install toolchains from package manager, which depends on distribution
• On MacOS you install toolchain from asset store
• On Windows you install manually, and some developers prefer MSVC over alternatives.

We can refer to the proper Godot documentation page for this, and the plugin can also autodetect toolchains, but user work is unavoidable here. At least it only needs to be done once, since this would be an editor wide setting.

[Ansraer]

What about just having the plugin download a docker container with the necessary build tools? This would make it possible for users to get everything they need to develop e.g. c++ with literally a single click.
In theory this would also make debugging and support easier, since everyone will use the exact same version of everything.

[reduz]

@Ansraer sure that could work to some degree, but keep in mind we can't redistribute legally a lot of stuff, like Android toolchain or MacOS/iOS headers. Would still need to do scripts for users to create those themselves.

This is in part why, if your extension does not really use anything native, webassembly is a very attractive option for deploy.

[Ansraer]

True, we could probably have some sort of setup wizard that guides the user through this. Actually having a window that tells me what I need to download, with a next button I can only press once I placed the downloaded file in the correct folder actually sounds like a really nice idea.

Another reason why I would like to use docker containers is libraries and compilers. At least master itself is currently incompatible with a bunch of compilers and libraries. Getting my dev environment for it setup and working took me a few days. The situation might be better when you only want to build gdextensions, but given how cumbersome it is to get anything installed on windows, and how many linux distros ship very outdated versions the idea of a clean controlled environment is very tempting imo.

[theraot]

Please don't make Docker mandatory.

As an option, it is ok. I'm sure it will get use.

Yet, I will not use it. Last time I tried Docker on my machine performance was very bad. Yes, I'm overdue an upgrade, but reasons, ok? Instead, I'll set up the development environment. I had no problem following the instructions from the documentation to build Godot.

I have seen the idea come up before. And I said the same thing: Please don't make Docker mandatory.

On the flip side, you could provide YAML files for CI platforms. For instance I had good experience with AppVeyor for other projects (It is a good fit for .NET projects because of its NuGet integration among other things), and you can put that on Docker if you want to. It does not have to be AppVeyor, it could be Jenkins (I think that is better for Java), it could be Github actions, or something else. I don't know what solution is a better fit for Godot and its community.

[barbaros83]

i see the comments sugesting a compiler be shipped with godot or the plugin, but if this gets you to the point where you could compile the extension with 1 command i think it should already be easy enough for most people that want to write their own extensions

[reduz]

@barbaros83 Downloading a compiler for this is literally no effort and, given large part of the Godot user base will just not need this, including it with the default download would be a large waste of disk, time and bandwidth. Don't want to go the Unity way here.

[Ansraer]

@theraot Are you sure that performance problem was actually dockers fault? Dockers overhead is usually only ~5%. There are some rare cases when it can get up to 18%, but that is caused by the security measures against attacks like spectre (which only really matter when you are running on a server). If you run docker with all security features disabled you almost never get a performance impact higher than 2%.
These days I am building a LOT of my code exclusively in containers, and I never encountered any slowdown when compared to running it directly.