Difficulty reconciling table option types for listing vs direct usage

[ianthetechie]

When adding a data source to the context, I think the current story around types is a bit clunky, but I wanted to start a discussion to make sure that 1) I'm not missing something, and 2) this isn't just me hitting it a a bad time in the middle of a refactor ;) I tried to search for similar issues but couldn't find anything, so here I am!

What I think is clunky:

When writing some code like ctx.register_parquet, I need to pass a type ParquetReadOptions<'_>. However when I want to create a ListingTable, there is no way to pass these on. The closest I could do is to is create a ParquetFormat and (maybe a method call, like with_options later) specify some ParquetOptions which are similar but not the same. It looks like there is a TryFrom impl for ParquetWriterOptions.

There is a similar story for CSV, though a few details differ.

It looks like there is also a family of types like CsvOptions that get used in the file format APIs. Are these eventually going to be some common subset between reader and writer options which could be used for setting listing table options and the register_X (where X is a format) paths alike? Or am I just thinking about this wrong?