Restart from different polynomial degrees #2358

DanielDoehring · 2025-04-10T12:48:59Z

See point 1 in #2316

github-actions · 2025-04-10T12:49:12Z

src/callbacks_step/save_restart_dg.jl

DanielDoehring · 2025-04-10T13:04:54Z

src/callbacks_step/save_restart_dg.jl

+
+# Version for MPI-parallel I/O
+function interpolate_restart_file!(u, file, slice,
+                                   mesh, equations, dg, cache,


I did not add the dispatch on the mesh here as it should not be necessary.

Unless it does not work for DGMulti?

Also, this only works for Cartesian elements, right? For curved elements you'd need to include the grid metrics in the interpolation/projection step, IIRC (cc @andrewwinters5000 )

Hm yeah true - I restrict this to DGSEM for now.

Good point - I though I am in safe waters as everything happens on the reference element, but maybe I need something like this?

Trixi.jl/src/callbacks_step/amr_dg2d.jl

Lines 108 to 116 in afdb486

# Loop over all elements in old container and scale the old solution by the Jacobian

# prior to projection

for old_element_id in 1:old_n_elements

for v in eachvariable(equations)

old_u[v, .., old_element_id] .= (old_u[v, .., old_element_id] ./

old_inverse_jacobian[..,

old_element_id])

end

end

So I tested this with the 2D elixir_advection_unstructured_flag where I increase the base polydeg from 3 to 4 and then restart the subsequent run with polydeg = 3 (This is required since the mesh has polydeg 3).

This is the error report for "uniform" refinement:

L2 error: 5.43210258e-04 Linf error: 7.61325588e-03 L2 error: 4.81669631e-05 Linf error: 7.53667941e-04 L2 error: 3.98543153e-06 Linf error: 8.58091296e-05 L2 error: 3.11114766e-07 Linf error: 6.82719964e-06

which is not quite 4th order, but certainly more than 3rd order.

But it would be good if @andrewwinters5000 could make a statement here :)

Yes, this should be okay to have a lower polydeg for the mesh compared to the solver. But your projections will need to take the Jacobians into account. In the mapped coordinates it is in fact the quantity J*u that is conserved and not simply u. However, on static meshes this fact is hidden and the Jacobian is usually divided off for convenience.

Okay, so I would need to do something like
$$u_\mathrm{new} = J_\mathrm{new} J^{-1}_\mathrm{old} u_\mathrm{old}$$
here?

src/callbacks_step/save_restart_dg.jl

codecov · 2025-04-10T13:25:00Z

Codecov Report

Attention: Patch coverage is 97.27273% with 3 lines in your changes missing coverage. Please review.

Project coverage is 91.74%. Comparing base (31e3d56) to head (c756203).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/callbacks_step/save_restart_dg.jl	96.63%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2358      +/-   ##
==========================================
- Coverage   96.80%   91.74%   -5.06%     
==========================================
  Files         494      494              
  Lines       40825    40905      +80     
==========================================
- Hits        39518    37525    -1993     
- Misses       1307     3380    +2073

Flag	Coverage Δ
unittests	`91.74% <97.27%> (-5.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

src/callbacks_step/save_restart_dg.jl

test/test_mpi_p4est_3d.jl

test/test_unstructured_2d.jl

…on yet

ranocha · 2025-04-16T05:36:03Z

What is the status of this PR - ready for a review by @sloede or myself or would you like to change something based on the discussion above?

DanielDoehring · 2025-04-16T06:37:16Z

I would wait for @SimonCan to do a first review, since he originally implemented the restarting mechanism. Other than that, I am of course grateful for being pointed to any flaws you immediately observe :)

ranocha · 2025-04-16T10:13:05Z

Great, thanks. Please feel free to ping me when Simon is done with his review.

SimonCan

Looks good to me now.

sloede

I haven't reviewed the parallel part yet, but I wanted to get these comments out before the Easter break.

Overall nice idea!

sloede · 2025-04-17T10:08:01Z

src/callbacks_step/save_restart_dg.jl

+
+# Version for MPI-parallel I/O
+function interpolate_restart_file!(u, file, slice,
+                                   mesh, equations, dg, cache,


Unless it does not work for DGMulti?

src/callbacks_step/save_restart_dg.jl

sloede · 2025-04-17T12:04:54Z

src/callbacks_step/save_restart_dg.jl

+
+# Version for MPI-parallel I/O
+function interpolate_restart_file!(u, file, slice,
+                                   mesh, equations, dg, cache,


Also, this only works for Cartesian elements, right? For curved elements you'd need to include the grid metrics in the interpolation/projection step, IIRC (cc @andrewwinters5000 )

src/callbacks_step/save_restart_dg.jl

sloede · 2025-04-17T12:42:03Z

test/test_mpi_p4est_3d.jl

+                            # NOTE: Normally we would also need to exchange the `AnalysisCallback` when changing the 
+                            # semidiscretization. This, however, leads in the test environment to a confusing error 
+                            # report, as the error report from the simulation is different from the one from the test environment.
+                            # Thus, we do not exchange the `AnalysisCallback` here.


Are these even valid error measures anymore then? I mean, if the poldeg of the solver and the analysis callback do not match?

Hm, somewhat I guess? But in the tests we do not really care about the meaning of the numbers but more that there are some numbers at all (which stay constant over development), right?

sloede · 2025-04-17T13:11:19Z

examples/tree_1d_dgsem/elixir_advection_restart.jl

+semi = SemidiscretizationHyperbolic(mesh, equations, initial_condition, solver,
+                                    boundary_conditions = boundary_conditions)
+
+ode = semidiscretize(semi, tspan, restart_filename)


Should we add a keyword to semidiscretize to explicitly allow polydeg conversion (i.e., conversely to throw an error if it is missing)? I am slightly worried about people accidentally restarting simulations at different polydegs and then wondering what went wrong. Something like

Suggested change

ode = semidiscretize(semi, tspan, restart_filename)

ode = semidiscretize(semi, tspan, restart_filename; allow_polydeg_conversion=true)

maybe. Or is this too much hand-holding for the users?

I have no strong feelings about this. One thing that comes to my mind is that we should throw a warning in the AnalysisCallback if the polydegs do not "match"

Co-authored-by: Joshua Lampert <[email protected]>

JoshuaLampert

LGTM once the final conversations are resolved.

DanielDoehring · 2025-04-25T13:48:40Z

LGTM once the final conversations are resolved.

Thanks for your input, Joshua!

NEWS.md

DanielDoehring · 2025-05-24T17:00:22Z

Could we get going again here? My feature PRs start piling up again
@SimonCan @sloede

src/callbacks_step/save_restart.jl

SimonCan

Looks good to me.

ranocha

Thanks a lot! I just have a minor suggestion. Does anybody know whether @sloede wants to review the PR after requesting changes some time ago?

examples/t8code_2d_dgsem/elixir_advection_unstructured_flag.jl

Co-authored-by: Hendrik Ranocha <[email protected]>

andrewwinters5000

Just some suggested cleanup of the docstrings. One thing that occurred to me, an error should get thrown if the attempted new solver polydeg is less than the existing mesh_polydeg as one cannot guarantee free-stream preservation in this case.

examples/t8code_2d_dgsem/elixir_advection_restart_higher_polydeg.jl

src/semidiscretization/semidiscretization.jl

src/semidiscretization/semidiscretization_hyperbolic_parabolic.jl

src/solvers/dgsem/l2projection.jl

src/callbacks_step/save_restart_dg.jl

andrewwinters5000 · 2025-05-27T05:09:14Z

src/callbacks_step/save_restart_dg.jl

+
+# Version for MPI-parallel I/O
+function interpolate_restart_file!(u, file, slice,
+                                   mesh, equations, dg, cache,


Yes, this should be okay to have a lower polydeg for the mesh compared to the solver. But your projections will need to take the Jacobians into account. In the mapped coordinates it is in fact the quantity J*u that is conserved and not simply u. However, on static meshes this fact is hidden and the Jacobian is usually divided off for convenience.

Co-authored-by: Andrew Winters <[email protected]>

…i.jl into Restart_Diff_PolyDeg

Co-authored-by: Andrew Winters <[email protected]>

…i.jl into Restart_Diff_PolyDeg

DanielDoehring added 2 commits April 10, 2025 12:09

first try int pol restart

48233fc

parallel I/O

7f28d1c

DanielDoehring added the enhancement New feature or request label Apr 10, 2025

DanielDoehring added 2 commits April 10, 2025 14:58

fmt example

853ebd8

Merge branch 'main' into Restart_Diff_PolyDeg

969a29e

DanielDoehring commented Apr 10, 2025

View reviewed changes

src/callbacks_step/save_restart_dg.jl Outdated Show resolved Hide resolved

DanielDoehring commented Apr 10, 2025

View reviewed changes

stricter test vals

8058cac

DanielDoehring commented Apr 10, 2025

View reviewed changes

src/callbacks_step/save_restart_dg.jl Show resolved Hide resolved

remove print statement

261765a

DanielDoehring marked this pull request as ready for review April 10, 2025 13:25

DanielDoehring commented Apr 10, 2025

View reviewed changes

src/callbacks_step/save_restart_dg.jl Show resolved Hide resolved

DanielDoehring requested a review from SimonCan April 10, 2025 13:27

DanielDoehring commented Apr 10, 2025

View reviewed changes

test/test_mpi_p4est_3d.jl Outdated Show resolved Hide resolved

DanielDoehring and others added 2 commits April 10, 2025 15:59

news

b0503bd

Update test/test_mpi_p4est_3d.jl

407eb25

DanielDoehring commented Apr 10, 2025

View reviewed changes

test/test_unstructured_2d.jl Outdated Show resolved Hide resolved

DanielDoehring and others added 3 commits April 10, 2025 16:10

Update test/test_unstructured_2d.jl

e0b1b62

Merge branch 'main' into Restart_Diff_PolyDeg

8ee9b70

mention that mpi parallel, serial io is not supported for interpolati…

611d804

…on yet

Merge branch 'main' into Restart_Diff_PolyDeg

23dd1bf

SimonCan previously approved these changes Apr 17, 2025

View reviewed changes

sloede reviewed Apr 17, 2025

View reviewed changes

DanielDoehring added 2 commits April 23, 2025 14:01

Merge branch 'main' into Restart_Diff_PolyDeg

6202cf0

version for parallel sim but serial I/O

a158c5c

DanielDoehring and others added 2 commits April 25, 2025 15:19

Update src/solvers/dgsem/l2projection.jl

364bcc9

Co-authored-by: Joshua Lampert <[email protected]>

Merge branch 'main' into Restart_Diff_PolyDeg

cc7a40d

JoshuaLampert previously approved these changes Apr 25, 2025

View reviewed changes

ranocha requested a review from sloede April 25, 2025 13:31

DanielDoehring requested a review from ranocha May 5, 2025 08:34

Merge branch 'main' into Restart_Diff_PolyDeg

ba80a0a

DanielDoehring dismissed JoshuaLampert’s stale review via ba80a0a May 5, 2025 08:35

DanielDoehring commented May 5, 2025

View reviewed changes

NEWS.md Outdated Show resolved Hide resolved

Update NEWS.md

e86e999

SimonCan reviewed May 26, 2025

View reviewed changes

src/callbacks_step/save_restart.jl Show resolved Hide resolved

SimonCan previously approved these changes May 26, 2025

View reviewed changes

Merge branch 'main' into Restart_Diff_PolyDeg

aa40a24

ranocha reviewed May 26, 2025

View reviewed changes

examples/t8code_2d_dgsem/elixir_advection_unstructured_flag.jl Outdated Show resolved Hide resolved

Update examples/t8code_2d_dgsem/elixir_advection_unstructured_flag.jl

6100654

Co-authored-by: Hendrik Ranocha <[email protected]>

DanielDoehring dismissed SimonCan’s stale review via 6100654 May 26, 2025 18:07

ranocha requested a review from andrewwinters5000 May 27, 2025 03:36

andrewwinters5000 requested changes May 27, 2025

View reviewed changes

DanielDoehring and others added 10 commits May 27, 2025 15:25

warning for non-matching polydeg

ece3849

comment to lower

648a638

Apply suggestions from code review

fa973a5

Co-authored-by: Andrew Winters <[email protected]>

Merge branch 'Restart_Diff_PolyDeg' of github.com:DanielDoehring/Trix…

1128144

…i.jl into Restart_Diff_PolyDeg

lower

01a99d1

Apply suggestions from code review

770ecd5

Co-authored-by: Andrew Winters <[email protected]>

news

be00bb9

Merge branch 'Restart_Diff_PolyDeg' of github.com:DanielDoehring/Trix…

43ad128

…i.jl into Restart_Diff_PolyDeg

Merge branch 'main' into Restart_Diff_PolyDeg

7a09915

fix news.md

c756203

DanielDoehring marked this pull request as draft May 28, 2025 13:07

	# Loop over all elements in old container and scale the old solution by the Jacobian
	# prior to projection
	for old_element_id in 1:old_n_elements
	for v in eachvariable(equations)
	old_u[v, .., old_element_id] .= (old_u[v, .., old_element_id] ./
	old_inverse_jacobian[..,
	old_element_id])
	end
	end

	ode = semidiscretize(semi, tspan, restart_filename)
	ode = semidiscretize(semi, tspan, restart_filename; allow_polydeg_conversion=true)

Restart from different polynomial degrees #2358

Are you sure you want to change the base?

Restart from different polynomial degrees #2358

Uh oh!

Conversation

DanielDoehring commented Apr 10, 2025

Uh oh!

github-actions bot commented Apr 10, 2025

Review checklist

Purpose and scope

Code quality

Documentation

Testing

Performance

Verification

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DanielDoehring May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Apr 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ranocha commented Apr 16, 2025

Uh oh!

DanielDoehring commented Apr 16, 2025

Uh oh!

ranocha commented Apr 16, 2025

Uh oh!

SimonCan left a comment

Choose a reason for hiding this comment

Uh oh!

sloede left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DanielDoehring Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoshuaLampert left a comment

Choose a reason for hiding this comment

Uh oh!

DanielDoehring commented Apr 25, 2025

Uh oh!

Uh oh!

DanielDoehring May 27, 2025 •

edited

Loading

codecov bot commented Apr 10, 2025 •

edited

Loading

DanielDoehring Apr 23, 2025 •

edited

Loading