Type checking/conversion between tasks in a workflow

[tclose]

I wanted to spell out what I had in mind for type-checking in a bit more detail ahead of the developers meeting tomorrow, but it is late so I will just try to jot down some ideas in point-form.

1. Can we please not globally disable attrs validators in pydra.__init__ (at least by default).
   • Not very courteous to other developers who use attrs
   • Makes the type-checking logic quite useless since most developers wouldn't know to enable type checking.
   • If performance is an issue we could have an option to disable it, but for any significant workflow I doubt type checking will be a bottleneck.
2. Favour implicit conversion instead of permissive type checks where possible
   • As I mentioned on the Mattermost channel, I believe a key design principle of workflows is to keep the tasks isolated from each other. Strong type checking helps this.
     • When writing function tasks, the task designer may rely a property of a type in one particular branch of the code, which only gets exposed in certain workflow configurations where a certain type is supplied to it (e.g. an iterator which becomes stale)
   • This could be avoided if we did implicit conversions (e.g. to a list or numpy.array) where the two types share a structural base class (e.g. Iterable, PathLike) with certain special cases (i.e. str, dict, etc...)
3. Implicit conversions would play nicely with fileformats
   • DestFileFormat(src_file_format_object) will attempt to reinterpret the file-system paths in src_file_format_object as a DestFileFormat
   • Works to/from "path-like" types, Path, str and will perform file-format checking
4. Conversions could be skipped if DestFileFormat issubtype of SrcFileFormat
   • Would allow a conceptual extension of issubclass
   • For example, I ended up implementing fileformat "qualifiers", to enable you to be able to specify the subject and modality of an image, e.g. DicomDir[Dwi, Brain, Human], and issubtype is able to work out that DicomDir[Dwi, Brain, Human] is a subtype of DicomDir[Dwi, Brain]

Anyway, I'm not sure if that all makes sense, but I need to get to bed :)

[ghisvail]

Hey Tom, thanks for taking the time to put all this in writing. I too believe a proper discussion around type safety in workflows is worth having.

[djarecka]

perhaps we can have some discussion today during the meeting

[effigies]

1. Can we please not globally disable attrs validators in pydra.__init__ (at least by default).

Agree. We should not change the behavior of other packages. If we need to (optionally) disable it in particular places, we should do so with a context manager to keep the scope clear and limited.

2. Favour implicit conversion instead of permissive type checks where possible

I'd worry about moving the semantics too far from type annotations. Would it work to use something like:

task2.inputs.in1 = cast(TargetType, task1.lzout.out)

Which could be something like:

def cast(tgt_type: type[T] | Callable[[V], T], value: Future[V]) -> Future[T]:
    ...

The default behavior would be to use normal typing rules, but any conversion could be supported in principle and we could even check the types of the source, destination and converter.

[effigies]

Attrs has a special status in mypy: https://mypy.readthedocs.io/en/stable/additional_features.html#the-attrs-package

[djarecka]

comments from the discussion:

• we should also add a check at the construction of the Workflow, since we have input_spec/output_spec for all tasks
  (some static checks might be impossible)

• adding a protocols for File: pathlike and hash