Water module units + size trimming

[Wegatriespython]

Motivation

• Water module has been dysfunctional. Model size has exploded to over 200 X 200 million with the new ssp scenarios.
• Cplex is unable to report infeasibilities even with all slacks enabled.

Changes :

• Units for electr input were previously left "-",
• Units for infrastructure electr inp believed to be Kwh/m^3. source
• Convert units to MCM basis and GWa/MCM for electr input.
• Assuming Desalination electr inp units to be GWa/Km^3 and similar for water supply. *Other assumptions unit assumptions result in major mismatches.
• Model size was excessive due to :
  • Share calib constraint for cooling ( commenting this out took size from 200X200 to 50X50)
  • map_yv_ya_lt : was making vintage activity combinations including historical vintage years leading to excess combinations.
  • duplicates in data generation. Overall size reduced to 16X16
• Replace xslx files with csvs.

How to review

• Go through unit changes and assumptions. Verify.
• Evaluate reporting output.
• Ask questions on new util for year vintage activity

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update doc/whatsnew.

[adrivinca]

Thanks for the cleaning work!
I still have to go through tests.
here just some comments for keeping consistency and avoid forgetting about commented parts over time

[adrivinca]

if you have checked it by looking at the GDX, please update this comment and maybe leave a comment with the correct unit, but warning that this might change in different globiom matrix submissions

[adrivinca]

the gdx has MCM values, could you maybe note it here. so that we keep track at least. thanks

[adrivinca]

I am not sure I understand/remember this. can you please explain? here on Github

[Wegatriespython]

I was thinking of using a static yaml for the missing techs so we would be able to validate the combinations generated and ensure they make sense. Otherwise, for each run there is a risk of errors propagating

[codecov]

Codecov Report

❌ Patch coverage is 99.27007% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 76.1%. Comparing base (a76db4e) to head (d93345c).
⚠️ Report is 408 commits behind head on main.

Files with missing lines	Patch %	Lines
message_ix_models/model/water/report.py	0.0%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #371     +/-   ##
=======================================
- Coverage   77.0%   76.1%   -0.9%     
=======================================
  Files        246     246             
  Lines      19819   19866     +47     
=======================================
- Hits       15270   15131    -139     
- Misses      4549    4735    +186     

Files with missing lines	Coverage Δ
message_ix_models/model/water/build.py	74.3% <ø> (-20.7%)	⬇️
message_ix_models/model/water/data/demands.py	78.8% <100.0%> (+0.1%)	⬆️
...ssage_ix_models/model/water/data/infrastructure.py	100.0% <100.0%> (ø)
message_ix_models/model/water/data/irrigation.py	100.0% <ø> (ø)
...essage_ix_models/model/water/data/water_for_ppl.py	94.5% <100.0%> (-0.3%)	⬇️
message_ix_models/model/water/data/water_supply.py	76.5% <100.0%> (+0.2%)	⬆️
message_ix_models/model/water/utils.py	63.3% <100.0%> (+6.9%)	⬆️
...e_ix_models/tests/model/water/data/test_demands.py	100.0% <100.0%> (ø)
...dels/tests/model/water/data/test_infrastructure.py	100.0% <100.0%> (ø)
...odels/tests/model/water/data/test_water_for_ppl.py	100.0% <100.0%> (ø)
... and 3 more

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[adrivinca]

@glatterf42 I did a first round of review and I am good with the current changes.
tests pass apart from for Python 3.9, as we already discussed.
Could you please also have a look and share your feedback?
once ready @Wegatriespython can update the Whatsnew file

We have more PRs coming after this, focusing on

• the reporting script
• new test suite
• general updates to the non-technical documentation
  so, comments related to these will be addressed soon.
  thanks

[Wegatriespython]

@glatterf42 I've reworded the commits, and added to whatsnew

[glatterf42]

Thanks, this looks great. I'd just add an empty line here to enable the rendering as a headline (which I think is the intention).

[glatterf42]

Thanks, there's just one more commit remaining: "Moving pol_scen out of if cond" should just be "Move pol_scen out of if cond". I'll approve the PR already, but please also fix this before merging :)

Ah, don't worry about the test case that's currently failing: this has nothing to do with your PR. It's fine to merge by bypassing the rules in this case 😃

[adrivinca]

why is the macos test stuck?
or can you merge anyway? I can't

[glatterf42]

See #381 (comment), I've done a bit of digging, but no clear answer yet.

Yes, I could merge it if we all agree it's ready :)

[Wegatriespython]

@glatterf42 I think its ready to merge. I did the changes for "Moving Pol scen..." and the update suggestion to whatsnew

[glatterf42]

I've rerun the failing test, it should pass in about 30 minutes. If you don't want to wait that long, you can already merge this PR (or afterwards). According to our contributing guidelines, this should be done with a merge commit. GitHub's automatic commit message for that is fine, but please change the body of the commit message to say something like "Update & fix units and size of the water module".

[khaeru]

@Wegatriespython @glatterf42 while reviewing #388 I noticed that this heading and the bullet list below have a leading indent of two spaces. In the built HTML docs, the heading does not appear and the list items don't match the rest of the list.

I'll correct the change in a commit on that branch, to align with the style used for previous releases.

In the future, please closely eyeball docs additions to make sure they render as you intend. The rest of the same file can be an example for formatting.