Skip to content

Fixing inconsistent results when using seed in examples (reopened) #3621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

sscini
Copy link

@sscini sscini commented Jun 6, 2025

Fixes # .

When using the reactor_design leave_N_out example in parmest, the results from multiple runs of the example vary a lot, with random seed defined.

Summary/Motivation:

Using a random seed should allow for reproducible results. Currently, the results vary in the leave_N_out reactor design example (pyomo/pyomo/contrib/parmest/examples/reactor_design/leaveNout_example.py) for parmest even with an active random seed. I included a temporary .py file (pyomo/pyomo/contrib/parmest/examples/reactor_design/mwe_bootstrap_error_SSE.py) with the minimum working example showing the results vary with multiple runs.

Changes proposed in this PR:

  • Modify leaveNout_bootstrap_test to produce consistent outputs

TODO before converting from draft:

  • Inform pyomo team
  • Implement solution
  • Testing

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the BSD license.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

@sscini
Copy link
Author

sscini commented Jun 6, 2025

Continuation of PR # 3608. Changed the branch name to be more consistent with goal, did not realize that would close PR. Thank you @blnicho for feedback and approval of change. Will make requested changes, make sure tests don't break, and change from draft to full PR.

@sscini
Copy link
Author

sscini commented Jun 6, 2025

@adowling2 @djlaky Ready for review, changes made examples reproducible with a set seed. Seeds also added to test_file. Test results:
Ran 44 tests in 14.982s
OK (skipped=2)

print(lNo_theta.head())

parmest.graphics.pairwise_plot(lNo_theta, theta)
parmest.graphics.pairwise_plot(lNo_theta, theta, seed=524,)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra comma on this line?



if SET_GLOBAL_SEED:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternate idea: either here or in the examples file (or somewhere else the Pyomo team recommends), define

_RANDOM_SEED_FOR_TESTING = 524

Then in all the files that use this random seed, you can import this value.

Then in this file:

if _RANDOM_SEED_FOR_TESTING:
   np.random.seed(_RANDOM_SEED_FOR_TESTING)

This will mean someone only sets the testing random seed in one spot.

@sscini
Copy link
Author

sscini commented Jun 10, 2025

Plan for this week:
Address comments above, confirm tests run, add additional comments, and run black on code for formatting

@sscini
Copy link
Author

sscini commented Jun 17, 2025

Notes for Pyomo Dev Meeting 6/17/25:

  • Plan to complete plan from previous and hit 'Ready for Review' to convert to normal PR.

Context
Bootstrap, leaveNout, and confidence region examples were not reproducible with a set seed.

Problem
Global seed was set, but not utilized for certain functions, and was not present in graphics.py for any of the scipy commands

Implemented solution
Added global seeds to different functions and in test file, working to optimize and minimize how many were added.

Questions

  • Any recommendations for previous comment from Dr. Dowling?
  • Would any new documentation or additional tests be needed to make sure this would run properly?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

Successfully merging this pull request may close these issues.

3 participants