Skip to content

Conversation

@nick-gorman
Copy link
Member

@nick-gorman nick-gorman commented Sep 11, 2025

Summary

This PR adds:

  • MkDocs documentation
  • a command-line interface built on doit,
  • regional filtering capabilities for ISPyPSA input templates
  • snapshot level weightings
  • updates to test data sets (although trace data will need to change again)
  • some other random reorganisation

Command-Line Interface (CLI)

New ispypsa command-line interface built on doit:

  • Task structure: Six tasks with automatic dependency management: cache_required_iasr_workbook_tables, create_ispypsa_inputs,
    create_pypsa_friendly_inputs, create_and_run_capacity_expansion_model, create_operational_timeseries,
    create_and_run_operational_model
  • Optional optimization: dont_run_capacity_expansion and dont_run_operational flags to skip optimization steps

Regional Filtering Feature

Added capability to filter ISPyPSA templates by region or sub-region:

  • Implementation: New filter_template.py module for filtering by NEM regions or sub-regions
  • Functionality: Reduces model scope to specified geographic areas

Translator & Snapshots Changes

Updates to PyPSA translation and snapshot handling:

  • Snapshot weighting: Added _add_snapshot_weightings function
  • Custom constraints: Extracted custom constraint logic to separate module (custom_constraints.py, 30 lines)

Testing Data Updates

Updated test IASR data and trace files:

  • Update test IASR data table set. Inputs in tables have stripped back to minimum needed for testing.
  • Trace data: Restructured renewable and demand traces

@nick-gorman nick-gorman marked this pull request as draft September 11, 2025 06:46
@codecov
Copy link

codecov bot commented Sep 11, 2025

Codecov Report

❌ Patch coverage is 97.97571% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/ispypsa/templater/create_template.py 60.00% 1 Missing and 1 partial ⚠️
src/ispypsa/templater/filter_template.py 98.63% 0 Missing and 2 partials ⚠️
src/ispypsa/model/save.py 0.00% 1 Missing ⚠️
Files with missing lines Coverage Δ
src/ispypsa/config/__init__.py 100.00% <100.00%> (ø)
src/ispypsa/config/loader.py 100.00% <ø> (ø)
src/ispypsa/config/validators.py 95.95% <100.00%> (+2.85%) ⬆️
src/ispypsa/model/build.py 91.66% <ø> (-4.17%) ⬇️
src/ispypsa/model/custom_constraints.py 80.00% <100.00%> (ø)
src/ispypsa/model/update.py 100.00% <ø> (ø)
src/ispypsa/templater/__init__.py 100.00% <100.00%> (ø)
src/ispypsa/templater/flow_paths.py 95.49% <ø> (+0.90%) ⬆️
src/ispypsa/templater/helpers.py 97.61% <ø> (ø)
src/ispypsa/templater/manual_tables.py 100.00% <100.00%> (ø)
... and 10 more

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nick-gorman nick-gorman marked this pull request as ready for review November 12, 2025 03:16
@nick-gorman nick-gorman changed the title Mkdocs setup & CLI implementation Mkdocs setup & CLI implementation (etc etc) Nov 12, 2025
Copy link
Member

@EllieKallmier EllieKallmier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet looks so solid! Few qs as usual for my benefit :)

docs/config.md Outdated

The order in which different weather reference years are used in the model. If the
number of reference years is less than the number of model years the reference years
will be reused in the order provided.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe helpful to give explicity example of this, e.g. if there are 5 model years and 2 ref years given (A, B) the reference year cycle becomes [A, B, A, B, A] etc.

)

# Get dont_run flag from doit variables
dont_run = get_var("dont_run_capacity_expansion", "False") == "True"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a little thing and more on the semantic side (+is just opinion) - but I think to me it would make more sense to phrase and implement this flag more as an opt-out than and opt-in?

e.g., instead of a "dont_run" flag with var name "dont_run_capacity_expansion", phrasing as a "run_optimisation" flag with var name "run_optimisation_for_capacity_expansion" (or something) feels a little clearer to me. And obv default is true and manually set to false means the opt doesn't run.

I think I'm just having an aversion to something that feels close to a double negative here so am very happy to be overruled - plus this already works so :)

ValueError: If both or neither of nem_regions and isp_sub_regions are provided
"""
# Validate inputs
if (nem_regions is None) == (isp_sub_regions is None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my understanding broader validation practices: the 'both true' version of this condition is also validated before this function gets called in the create_ispypsa_inputs_template function. Is there a schema/reason for kind of splitting up validation in this way? Thinking about how I'm sometimes indecisive about where's the best/right/good place to validate this kind of thing and whether there's a handy rule or thought process I can adopt :)


if not selected_sub_regions:
logging.warning("No valid regions found for filtering")
return [], []
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe similar question to above (just a q for me) - is it generally preferred/better practice to e.g. return values here that later (after this function has been called and run) trigger an error rather than raising the error here?

I'm imagining it's probably in conjunction with logging helpful for tracing the error back if it's getting raised in the higher-level function?

@@ -1,6 +1,2 @@
Region,CCGT,CCGT with CCS,Small OCGT,Large OCGT,Reciprocating engines,Hydrogen Reciprocating engines,Biomass,1 hr Battery Storage,2 hr Battery Storage,4 hr Battery Storage,8 hr Battery Storage,Pumped Hydro3 (8 hrs storage),Pumped Hydro (24 hrs storage),Pumped Hydro (48 hrs storage),BOTN- Cethana
QLD,110.49433333333297,110.49433333333297,110.49433333333332,110.49433333333332,110.49433333333332,110.49433333333332,110.49433333333332,103.36566666666667,103.36566666666667,103.36566666666667,103.36566666666667,0,0,0,0
Region,CCGT,CCGT with CCS,Small OCGT2,Large OCGT,Reciprocating engines,Hydrogen Reciprocating engines,Biomass,1 hr Battery Storage,2 hr Battery Storage,4 hr Battery Storage,8 hr Battery Storage,Pumped Hydro3 (8 hrs storage),Pumped Hydro (24 hrs storage),Pumped Hydro (48 hrs storage),BOTN- Cethana
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like footnote back here too? (Matching?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants