Explore ontology tools with pyiron_workflow (#733)

* add tests

* Make input and output plural to be in agreement with the pyiron_workflow definitions

* add one test

* bugfix and add tests

* one more test for a full triple

* adjust line length

* be more specific about the number of triples in tests

* Add tests (but not working right now)

* Add wrong analysis

* Codacy

* Change triple and restriction plural

* Add tests

* run black

* Replace EX by NS because I guess EX stands for example

* restore dot

* Add tests

* run black

* current state

* Use pyiron namespace

* Make it clear that the predicate is just for parsing and does not have a meaning

* add test

* Add tests

* Codacy

* add tests for place holder

* Replace Placeholder by None

* Add test

* add error messages

* Add a sparql query test

* run black

* Correct tests

* Simplify test

* Add tests

* run black

* Add error message

* Move the location of _is_semantikon_class

* Add sub class parsing

* bugfix

* run black

* Remove dependence on semantikon

* run black

* reduce lines

* Add test

* Move it to workflow parser and add tests

* Another black

* Depend on semantikon

* Correct tests

* Add macro tests

* Remove workflow_to_dict

* Simplify test

* Simplify more tests

* Test with reasoner

* remove unused inherit_properties

* Update pyiron_workflow

* name space

* Liam's suggestion

* Correct errors

* Update Actions (#130)

* Update yml files

* Update conf file

* run black

* run ruff

* define __all__

* run ruff fix

* Configure ruff

Signed-off-by: liamhuber <[email protected]>

* Use more descriptive names

Signed-off-by: liamhuber <[email protected]>

* Remove unused import

Signed-off-by: liamhuber <[email protected]>

* Switch providers

Signed-off-by: liamhuber <[email protected]>

* Update badges

Signed-off-by: liamhuber <[email protected]>

---------

Signed-off-by: liamhuber <[email protected]>
Co-authored-by: liamhuber <[email protected]>

* Update test_parser.py

* replace typing by metadata

* ruff sort

* Update tests

* Update syntax and remove unused lines

* Remove unnecessary imports

* Remove redundant get_graph

* typo

* Explore ontology tools with pyiron_workflow

Signed-off-by: liamhuber <[email protected]>

* [dependabot skip] Update env file

* Downgrade pandas

Signed-off-by: liamhuber <[email protected]>

* [dependabot skip] Update env file

* Read the inequality, Huber

Signed-off-by: liamhuber <[email protected]>

* [dependabot skip] Update env file

* git-copy pyiron_ontology.parser  (#745)

* Forgot to add the parser

* run black

* Make input and output plural to be in agreement with the pyiron_workflow definitions

* small bugfixes

* add one test

* update definition of connection

* bugfix

* another programming error worth a harakiri

* make sure that the triples are tuples

* bugfix

* set default values

* convert object to URIRef only if it is not already one

* typo

* There are arbitrary values that cannot work with if

* allow triples in triple

* update semantikon version

* separate out triple parser

* bugfix and add tests

* make it possible to give custom predicates

* experiment: use sameAs instead of comesFrom

* implement inherit properties

* make inherit properties a public function

* add restriction parser

* run black

* add reasoner

* Replace [] by None

* implement validator

* Do not inherit type

* calculate number of iterations automatically

* increase number of inheritances

* Define each label as a named individual

* Name change in order to avoid confusion

* Change triple and restriction plural

* Bugfix due to pointing to data

* run black

* Remove named individual because apparently it's implied

* write workflow parser

* Replace EX by NS because I guess EX stands for example

* append workflow name

* add namespace everywhere

* restore dot

* codacy

* Add activity and entity

* current state

* Use pyiron namespace

* Add type hints

* replace value by node

* add test

* add docstring

* run black

* Add URIRef to uri just in case

* Create pyiron specific namespace

* Add tests

* Break apart parsing and business

Signed-off-by: liamhuber <[email protected]>

* Be explicit about what's missing

Signed-off-by: liamhuber <[email protected]>

* Implement restriction validator and Placeholder

* Remove Any because it's not used anymore

* I don't really understand but this silences mypy

* run black

* Replace Placeholder by None

* run black

* remove var_name because it's never used and anyway appears in the key

* Add test

* Add additional node for the values

* run black

* Small bugfix

* Correct tests

* run black

* Remove units in string

* change definition of tag

* I stop in this state now because some functions are too complex

* change definition of tag

* refactor tag

* not repeat if

* Refactor

* replace identifier by str

* Store it in the current state

* add subpropertyof

* Use subclass instead

* Add sub class parsing

* Add warning

* another one

* Remove dependence on semantikon

* Current state

* Update meta_to_dict

* run black

* define variables

* rename outputs

* bugfix

* I'm a loss

* I'm a total loss

* make ontology optional

* Make ontology trickle down

* run black

* Add check missing items

* Move it to workflow parser and add tests

* Add comments

* Run black

* Replace nodes by dicts

* Move parsing to a separate function

* Move parsing to a separate function

* Depend on semantikon

* Update header

* run black

* Check whether the owner is the same node

* Remove workflow_to_dict

* Remove all the unused stuff

* run black

* Add with values and with default

* Add docstring

* Correct imports

* Correct path for Macro

* Correct SNS name

* Liam's suggestion

* Update Actions (#130)

* Update yml files

* Update conf file

* run black

* run ruff

* define __all__

* run ruff fix

* Configure ruff

Signed-off-by: liamhuber <[email protected]>

* Use more descriptive names

Signed-off-by: liamhuber <[email protected]>

* Remove unused import

Signed-off-by: liamhuber <[email protected]>

* Switch providers

Signed-off-by: liamhuber <[email protected]>

* Update badges

Signed-off-by: liamhuber <[email protected]>

---------

Signed-off-by: liamhuber <[email protected]>
Co-authored-by: liamhuber <[email protected]>

* Update pyiron_ontology/parser.py

* Update pyiron_ontology/parser.py

* Update parser.py

* Update parser.py

* Update parser.py

* replace data_edges by edges

* typo

---------

Signed-off-by: liamhuber <[email protected]>
Co-authored-by: samwaseda <[email protected]>
Co-authored-by: Sam Dareska <[email protected]>
Co-authored-by: Sam Waseda <[email protected]>

* Update imports

And re-run notebook. And exclude the new file from mpyp. @samwaseda, mypy does not like your file XP

Signed-off-by: liamhuber <[email protected]>

* Update dependencies

Signed-off-by: liamhuber <[email protected]>

* Bump graphviz lower bound

To accommodate semantikon

Signed-off-by: liamhuber <[email protected]>

* Update imports

Signed-off-by: liamhuber <[email protected]>

* Rename test

To match renamed module

Signed-off-by: liamhuber <[email protected]>

* Edit tests

Mostly just for import scopes

Signed-off-by: liamhuber <[email protected]>

* Update imports

Signed-off-by: liamhuber <[email protected]>

* Validate ontology at connection time

Not working with restrictions yet

Signed-off-by: liamhuber <[email protected]>

* Draft connection suggestion

Signed-off-by: liamhuber <[email protected]>

* Update notebook

Signed-off-by: liamhuber <[email protected]>

* Add a back door to turn off the feature

Since it's pretty janky atm

Signed-off-by: liamhuber <[email protected]>

* Fix docstring hint

Signed-off-by: liamhuber <[email protected]>

* Add a node-class suggester

Signed-off-by: liamhuber <[email protected]>

* Modify candidate graph subgraph IO

Signed-off-by: liamhuber <[email protected]>

* Close match

pycharm is still not behaving nicely with this though

Signed-off-by: liamhuber <[email protected]>

* Update notebook

Signed-off-by: liamhuber <[email protected]>

* Fail on connection in test

Signed-off-by: liamhuber <[email protected]>

* Make knowledge mypy compliant

At least when we're non-strict

Signed-off-by: liamhuber <[email protected]>

* Remove unused import

Signed-off-by: liamhuber <[email protected]>

* Don't write recovery file

We know and expect a failure, don't clutter the file system

Signed-off-by: liamhuber <[email protected]>

* Move the import short-circuit

Over to a special method in the knowledge module

Signed-off-by: liamhuber <[email protected]>

* Adapt sleep times

It turns out `semantikon.ontology` has a heavy import overhead

Signed-off-by: liamhuber <[email protected]>

* Add a known issue to the demo notebook

Signed-off-by: liamhuber <[email protected]>

* Refine overheads

Based on results from main branch

Signed-off-by: liamhuber <[email protected]>

* Delay imports

Until they're strictly needed. It's bad form, but it saves graphs that are _not_ ontologically annotated a bunch of import time

Signed-off-by: liamhuber <[email protected]>

* Revert test timing

The tests in question don't use ontologies, so the original formulation is fine again.

Signed-off-by: liamhuber <[email protected]>

* Update notebook comment

Signed-off-by: liamhuber <[email protected]>

* Revert is_composite

Since `semantikon.ontology` imports slowly, we break the circular import inside the `channels` module anyhow

Signed-off-by: liamhuber <[email protected]>

* Fail cleanly for unknown node types

Signed-off-by: liamhuber <[email protected]>

* Rename argument

Signed-off-by: liamhuber <[email protected]>

* Narrow hint

Signed-off-by: liamhuber <[email protected]>

* Adapt export for non-functions

This is to capture `@u` working on non-functions. It doesn't really make sense for dataclasses, and we don't decorate for- and while- loops or other transformers, so only handling it in the function case should be fine.

Signed-off-by: liamhuber <[email protected]>

* Destroy, then connect

First disconnect and remove macros' single-use UI nodes _before_ using their single-connections as new input value receivers. This is a benign change for most use cases, but critical for avoiding double-receiving edges when doing ontological validation of macro creation.

This would (will!) all be much easier once `pyiron_workflow` adopts the `flowrep` pattern of single-sources for inputs and putting the single available connection and being a value receiver on the same footing.

Signed-off-by: liamhuber <[email protected]>

* Extend notebook

Signed-off-by: liamhuber <[email protected]>

* Re-execute notebook

Signed-off-by: liamhuber <[email protected]>

* Update concluding comments

Signed-off-by: liamhuber <[email protected]>

* Ontologically validate value reception

Signed-off-by: liamhuber <[email protected]>

* Update notebook

Signed-off-by: liamhuber <[email protected]>

* Update ontology term

Signed-off-by: liamhuber <[email protected]>

* Walk the recipe to the relevant parent node

Signed-off-by: liamhuber <[email protected]>

* Make error more informative

Signed-off-by: liamhuber <[email protected]>

* Update and re-run notebook

Signed-off-by: liamhuber <[email protected]>

* Catch unit failures

Signed-off-by: liamhuber <[email protected]>

* Document unit usage

Signed-off-by: liamhuber <[email protected]>

* Ontologically decorate dataclass nodes

By allowing semantikon.metadata.u kwargs as decorator arguments to apply to the output channel

Signed-off-by: liamhuber <[email protected]>

* Fix recipe/lexical path mismatch

Signed-off-by: liamhuber <[email protected]>

* Update notebook

Signed-off-by: liamhuber <[email protected]>

* Verify custom labels are ok

Signed-off-by: liamhuber <[email protected]>

* Bump semantikon

Signed-off-by: liamhuber <[email protected]>

* Reference what you can from the api

Signed-off-by: liamhuber <[email protected]>

* Give better feedback in connection errors

Signed-off-by: liamhuber <[email protected]>

* Re-execute notebook

To display more informative messages

Signed-off-by: liamhuber <[email protected]>

* Rebase exception class

Signed-off-by: liamhuber <[email protected]>

* Encode pizza example in unit tests

Signed-off-by: liamhuber <[email protected]>

* Express bad value receivers as connection errors

Signed-off-by: liamhuber <[email protected]>

* Test restrictions and subgraph negotiation

Signed-off-by: liamhuber <[email protected]>

* Add an integration test

Signed-off-by: liamhuber <[email protected]>

* Rename notebook

Signed-off-by: liamhuber <[email protected]>

* Update notebook

With a nicer header, and to remove the boring subgraph negotiation example which is fine just in the unit tests

Signed-off-by: liamhuber <[email protected]>

* Fail cleanly when validating outside a graph

I.e. for node networks with no parent workflow or macro

Signed-off-by: liamhuber <[email protected]>

* Test units (non)inheritance

Signed-off-by: liamhuber <[email protected]>

* Adapt suggestion

for validation raising errors instead of returning False

Signed-off-by: liamhuber <[email protected]>

* Don't suggest connected inputs

They're already connected! We only allow one connection

Signed-off-by: liamhuber <[email protected]>

* Refactor value receiver validation

In anticpation of needing a two-step process for suggestion tools

Signed-off-by: liamhuber <[email protected]>

* Refactor for regular connections

Signed-off-by: liamhuber <[email protected]>

* Use a dataclass

Signed-off-by: liamhuber <[email protected]>

* Use a helper method

Signed-off-by: liamhuber <[email protected]>

* Rename method

Signed-off-by: liamhuber <[email protected]>

* Refactor suggestions

I'm struggling to get them playing nicely with graphs that validate with missing triples (e.g. because of some unconnected node interacting with the workflow IO)

Signed-off-by: liamhuber <[email protected]>

* Don't suggest circular connections

Signed-off-by: liamhuber <[email protected]>

* Allow suggestions for connected input

Signed-off-by: liamhuber <[email protected]>

* Test for unrelated problems

#733 (comment)
Signed-off-by: liamhuber <[email protected]>

* Refactor: change signature

Signed-off-by: liamhuber <[email protected]>

* Include the root name

In the path information

Signed-off-by: liamhuber <[email protected]>

* Validate based on involvement

Signed-off-by: liamhuber <[email protected]>

* Extend the test

To guarantee that we're investigating the invovlement of _channels_ and not of _nodes_

Signed-off-by: liamhuber <[email protected]>

* Don't suggest for already-connected inputs

Signed-off-by: liamhuber <[email protected]>

* Directly manipulate execution digraph

Instead of adding a real connection to the workflow graph

Signed-off-by: liamhuber <[email protected]>

* 🐛 fix validation

There is still code duplication here with the testing that happens in the channels module, but we can deal with that later

Signed-off-by: liamhuber <[email protected]>

* Only check for downstream involvement

Signed-off-by: liamhuber <[email protected]>

* Scope involvement for macros

We need the path to the URI ref. Also, for units we look at the _upstream_ part!

Signed-off-by: liamhuber <[email protected]>

* Allow connection suggestions to come from a corpus of nodes

Signed-off-by: liamhuber <[email protected]>

* Just call the connection suggester to suggest classes

It might be slower, but for the time being let's shoot for stability

Signed-off-by: liamhuber <[email protected]>

* Remove duplicate code

Signed-off-by: liamhuber <[email protected]>

* Rename method and change default

To be public and false now that we use it elsewhere

Signed-off-by: liamhuber <[email protected]>

* Draft tests for the suggestions

Everything they print now makes sense to me, they just need to be converted to real tests

Signed-off-by: liamhuber <[email protected]>

* Add early stopping

Signed-off-by: liamhuber <[email protected]>

* Make exceptions more specific

I find this makes it easier to reliably test against them. Including the standard type (ValueError) here means they can also reasonably be caught in generic ways.

Signed-off-by: liamhuber <[email protected]>

* Test suggest_connections

Signed-off-by: liamhuber <[email protected]>

* Refactor suggestion collections in tests

Signed-off-by: liamhuber <[email protected]>

* Test node suggestions

Signed-off-by: liamhuber <[email protected]>

* Test corpus argument

Signed-off-by: liamhuber <[email protected]>

* Test multiple channels

Signed-off-by: liamhuber <[email protected]>

* Extend demo

Signed-off-by: liamhuber <[email protected]>

* Fix newly introduced typos and rerun notebook

Signed-off-by: liamhuber <[email protected]>

* Update module docstring

Signed-off-by: liamhuber <[email protected]>

---------

Signed-off-by: liamhuber <[email protected]>
Co-authored-by: samwaseda <[email protected]>
Co-authored-by: Sam Dareska <[email protected]>
Co-authored-by: pyiron-runner <[email protected]>
Co-authored-by: Sam Waseda <[email protected]>

Author: 3 people

.binder/environment.yml

@@ -11,6 +11,8 @@ dependencies:
 - pyiron_snippets =0.2.0
 - python >=3.11,<3.14
 - python-graphviz =0.21
+- rdflib =7.1.4
+- semantikon =0.0.22
 - setuptools>=68
 - toposort =1.10
 - typeguard =4.4.4

.ci_support/environment.yml

@@ -11,6 +11,8 @@ dependencies:
 - pyiron_snippets =0.2.0
 - python >=3.11,<3.14
 - python-graphviz =0.21
+- rdflib =7.1.4
+- semantikon =0.0.22
 - setuptools>=68
 - toposort =1.10
 - typeguard =4.4.4

.ci_support/lower_bound.yml

@@ -12,6 +12,8 @@ dependencies:
 - pint =0.24
 - python =3.11
 - pyiron_snippets =0.1.4
-- python-graphviz =0.20.0
+- python-graphviz =0.20.3
+- rdflib =7.1.4
+- semantikon =0.0.22
 - toposort =1.10
 - typeguard =4.2.0

docs/environment.yml

@@ -17,6 +17,8 @@ dependencies:
 - pyiron_snippets =0.2.0
 - python >=3.11,<3.14
 - python-graphviz =0.21
+- rdflib =7.1.4
+- semantikon =0.0.22
 - setuptools>=68
 - toposort =1.10
 - typeguard =4.4.4