📢 Status of Legacy Parsers and Transition to New Schema

[JFRudzinski]

We would like to provide clarity regarding the status of this repository.

This parser suite is built on NOMAD’s legacy simulation schema. Our current development efforts are directed towards building a new schema and parser framework, which will ultimately replace this codebase.

What this means for issues and bug reports

• We are not actively fixing bugs in the legacy parsers, except in exceptional or critical cases.
• Reported issues will be reviewed but will most likely be addressed as part of the new parser suite, not here.
• Development updates and improvements will happen in the new repository: nomad-parser-plugins-simulation.

Why this transition?

The new schema and parser framework aims to:

• Provide a more consistent, extensible representation of simulation data.
• Simplify parser development and reduce duplication.
• Resolve many of the known limitations of the current legacy setup.

What you can do

• If you depend on the legacy parsers: please continue to use them as-is, but be aware of their semi-frozen status.
• For future work, testing, or early contributions: follow the new repository.

We appreciate your patience and contributions while we carry out this transition.
Thank you for helping us make NOMAD’s simulation infrastructure more robust and sustainable.

[mm-may]

Great to hear, thanks for the update!
Would it be possible to have in the meantime somewhere a list of supported codes and up to which code version? The NOMAD documentation lists only the legacy parses, but seems to be at least partially outdated, as for instance CP2K output parsing is increasingly broken since CP2K versions from 2020 onwards.

[ondracka]

Thanks for the update, I have some questions though:

• What is the timeline here? Is the switch planned for some specific release or will it happen only when everything is ported to the new scheme?
• The new schema and parser framework is ready/frozen at this point? If not, how close it is?
• What is the plan to prevent regressions and achieve same/similar parsing coverage after the switch? Specifically I don't see much testing in the new repo?

[JFRudzinski]

Great to hear, thanks for the update! Would it be possible to have in the meantime somewhere a list of supported codes and up to which code version? The NOMAD documentation lists only the legacy parses, but seems to be at least partially outdated, as for instance CP2K output parsing is increasingly broken since CP2K versions from 2020 onwards.

@mm-may Thanks for the suggestion, indeed this documentation page is on our list of things to improve. We should certainly post there the latest version of each code which is known to be supported.

I need to assess the effort associated with this to know whether we can tackle this now. If it is more involved, it may need to wait until the new parser releases (see more info in my next post), as we are dedicating most of our efforts now towards getting the new parsers out.

I will be in touch about this, and sorry for the temporary inconvenience.

[JFRudzinski]

@ondracka Thanks for your feedback, see my responses below:

• What is the timeline here? Is the switch planned for some specific release or will it happen only when everything is ported to the new scheme?
  I cannot currently give an accurate timeline on which the legacy parsers will be completely removed. Over the next few months, we will be slowly adding the new parsers to the system (first to the test Oasis, then to the central deployment).

During this time, the old system will function as normal, but the new parsers will additionally act on the uploads and store the data (i.e., duplicated) under the data section of the archive.

• The new schema and parser framework is ready/frozen at this point? If not, how close it is?
  The new schema is approaching a state of stability, but there are still a number of larger changes/additions that will happen in the coming weeks or month. After that it should be rather stable, with only smaller adjustments, or additions.

The parsers are still in quite a preliminary phase (although a number of them are quite functional), but the new parsing system (background utility) is complete and stable, and we move now to tackling the heavy dev of the parsers. This will happen according to a certain priority, largely based off of current uploads + user input. I.e., if you voice here the parsers that you would like to take precedent, we will take this into account.

• What is the plan to prevent regressions and achieve same/similar parsing coverage after the switch? Specifically I don't see much testing in the new repo?
  In the parser dev phase, we will transfer the coverage of the legacy parsers and add support based on open issues. At this point we will add the corresponding test modules for this.

Please let me know if you have further questions. Thanks for your patience 🙏