Add include support in tld files

[Tisten]

We are defining "common types" in the start of every type library. It would be useful to be able to define these once and then just include them where needed.

[wc-duck]

in the tld-files you mean? Could you expand with an example?

[Tisten]

Yes, letting tld files include other tld files.
Example

vecmath.tld
{"module": "vecmath", "types": [{"point3": {"extern": true, "members" : ["name": "x", "type": "fp32", ...]}]}

physics.tld
{"module": "physics", "includes":["vecmath.tld"], "types": [{"phys_body": {"members" : ["name": "pos", "type": "point3", ...]}]}

entity.tld
{"module": "entity", "includes":["vecmath.tld"], "types": [{"entity": {"members" : ["name": "pos", "type": "point3", ...]}]}

The same is the case for metadata types.

[Tisten]

Poking @ElementalAndy about this

[ElementalAndy]

👍

[Tisten]

I'm thinking that we can keep the file handling outside of dl, and just add a callback function to the load_type_library functions, the callback will be used when an include is discovered. And then dltlc will implement such a callback.

Users who don't want to use includes can just pass in nullptr as the callback and be done with it.

If you like the idea then the question becomes if we should add the callbacks to the existing load_type_library functions and thus break api, or if we should keep the existing ones and create new duplicated functions for the callbacks?

[Tisten]

A second idea for this is to only support includes in the text version. Since the dl_typelib_write implementations will write out all types stored in the context, and these is no "origin type lib" stored per type, there is no way to split the types out into their original file structure. So the includes will just be a way to structure the types in the source format, and after reading them in the first time they will be unconditionally "bundled".

[Tisten]

Implemented a version based on my thoughts above, I think it fulfills our needs.

[wc-duck]

I'll keep the discussion here and outside of the PR.

There is the alternative of setting the include-hander on the actual dl_ctx and pass it via dl_create_params, that could be added without breaking the API and if I were to guess most users would probably set the same function for all its contexts?

I think I would prefer that if you don't have any good reason not to?

What I wouldn't really like with not supporting it in binary is that going txt -> bin -> txt would not generate the same result, and I think it should (for reference issue #128 ).
We could probably mark the type loaded via an include with a flag that it was "included" and store the strings used for includes to be able to repro all of it?

Lastly, looking at the PR it need to be clarified that an include is only a "string" that is used by the user to map it to an include and it is not forced to be a path, it is up to the user to decide if it is a path (abs/relative) or something else, DL do not care.

[wc-duck]

one more comment on the PR, I think dltlc should support includes from default paths out of the box with a -I flag for seach-paths :)

[Tisten]

one more comment on the PR, I think dltlc should support includes from default paths out of the box with a -I flag for seach-paths :)

What do you means with "default paths"?
And you want me to switch from -s to -I (capital i as opposed to -i for info)?

[Tisten]

There is the alternative of setting the include-hander on the actual dl_ctx and pass it via dl_create_params, that could be added without breaking the API and if I were to guess most users would probably set the same function for all its contexts?

I think I would prefer that if you don't have any good reason not to?

That's fine by me.

What I wouldn't really like with not supporting it in binary is that going txt -> bin -> txt would not generate the same result, and I think it should (for reference issue #128 ). We could probably mark the type loaded via an include with a flag that it was "included" and store the strings used for includes to be able to repro all of it?

I don't think that kind of flag is the way to do it, since the binary version of the type library has all types merged into it, and that is the way we want to use it. It might be better to map each type against which of the loaded type libraries brought it in, and when you want to write them out to text you select which of the read libraries you want to write out. Then you can choose to merge libraries (i.e debug dump all loaded types) or not to merge them (to be able to edit them as they were when they were created). The thing is that you already supported "includes" by reading the type libraries in the right order, and then writing them all out together. The things that were missing was the support for this in the definition files, and the support to use it in dltlc.

Lastly, looking at the PR it need to be clarified that an include is only a "string" that is used by the user to map it to an include and it is not forced to be a path, it is up to the user to decide if it is a path (abs/relative) or something else, DL do not care.

To be able to use it in dltlc it needs to be a path, not just any string. But I will clarify that you can write your own tooling which interpret that string differently.

[wc-duck]

But do the binary lib need to have all types loaded types in it or can the include feature be used there as well? So if you use includes you also get a to "include" them in when loading binary as well?

There might also be 2 ways to load them so that you can decide if you want to bundle them together or not?

I like the fact that you just made the callback get a ctx and then let the user call load on that so that you can load from bin and text! That would also open up to selecting if you want to load them "into" the final lib or just depend on inclusion when loading the binary as well.

[Tisten]

The bin version could support includes as well. We could extend dltlc to have that mode as well. But it isn't something I would do in this iteration, right now the functionality in the PR is enough for our use case.