-
Notifications
You must be signed in to change notification settings - Fork 20
Toniof 523 allow temporal resampling #542
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
TonioF
wants to merge
32
commits into
main
Choose a base branch
from
toniof-523-allow-temporal-resampling
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
ce44981
work on integrating temporal resampling
TonioF 666633f
raise error when temporal resampling is requested
TonioF 367f0c2
fix
TonioF b118f90
integrated temporal resampling
TonioF ef6a2a3
edited list of supported resampling methods
TonioF 99f9551
introduced temporal resampling
TonioF 6e9ca51
improved temporal subsetting
TonioF 6943c2a
improved setting of bounds
TonioF 3649eb3
adjust upsampled bounds with numpy timedelta correctly
TonioF 8d8d0f2
Merge branch 'master' into toniof-523-allow-temporal-resampling
TonioF b441a58
edited downsampling methods
TonioF 00eb10f
edited changelog
TonioF 2dd3531
Merge branch 'master' into toniof-523-allow-temporal-resampling
TonioF 6b76be5
resample to center of month
TonioF 89e26bd
avoid use of outdated method
TonioF 8b9a1aa
Merge branch 'master' into toniof-523-allow-temporal-resampling
TonioF 3161ab2
pep 8
TonioF 4177a4d
work on better considering up- and downsampling
TonioF 19907c2
remodeled temporal resampling method parameter
TonioF 9dca667
pass interpolation kind and percentile threshold as additional method…
TonioF bffe1b3
added error handling
TonioF a2d629b
edited changelog
TonioF efbe060
avoid use of deprectated methods
TonioF 5f2a6e5
Merge branch 'master' into toniof-523-allow-temporal-resampling
TonioF 95a3394
added tests
TonioF e12b348
allow passing in method name without parameters
TonioF e01c893
added empty line
TonioF c098f6f
do not pass cube config in constructor
TonioF 15e6cf0
temporal resampling is done purely based on datasetresample and dataa…
TonioF a551b3c
updated changelog
TonioF 2e559b0
test fix
TonioF b45051a
test fix
TonioF File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,293 @@ | ||
| from xcube.core.new import new_cube | ||
| from xcube.core.gen2 import CubeConfig | ||
| from xcube.core.gen2.local.resamplert import CubeResamplerT | ||
| from xcube.core.gridmapping import GridMapping | ||
|
|
||
| import cftime | ||
| import numpy as np | ||
| import unittest | ||
|
|
||
|
|
||
| class CubeResamplerTTest(unittest.TestCase): | ||
|
|
||
| @staticmethod | ||
| def _get_cube(time_freq: str, time_periods: int, use_cftime: bool = False): | ||
|
|
||
| def b3(index1, index2, index3): | ||
| return index1 + index2 * 0.1 + index3 * 0.01 | ||
|
|
||
| return new_cube(variables=dict(B03=b3), | ||
| time_periods=time_periods, | ||
| time_freq=time_freq, | ||
| use_cftime=use_cftime, | ||
| time_dtype='datetime64[s]' if not use_cftime else None, | ||
| width=10, height=5, time_start='2010-08-04') | ||
|
|
||
| def test_transform_cube_no_time_period(self): | ||
| cube_config = CubeConfig(time_range=('2010-01-01', '2012-12-31')) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='M', time_periods=12) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertEqual(cube, resampled_cube) | ||
|
|
||
| def test_transform_cube_downsample_to_years(self): | ||
| cube_config = CubeConfig(time_range=('2010-01-01', '2014-12-31'), | ||
| time_period='2Y', | ||
| temporal_resampling=dict( | ||
| downsampling=('min', {})) | ||
| ) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='M', time_periods=24) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertIsNotNone(resampled_cube) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time.values, | ||
| np.array(['2011-01-01T00:00:00', '2013-01-01T00:00:00'], | ||
| dtype=np.datetime64)) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time_bnds.values, | ||
| np.array([['2010-01-01T00:00:00', '2012-01-01T00:00:00'], | ||
| ['2012-01-01T00:00:00', '2014-01-01T00:00:00']], | ||
| dtype=np.datetime64)) | ||
| self.assertEqual((2, 5, 10), resampled_cube.B03.shape) | ||
| self.assertAlmostEqual(0.0, resampled_cube.B03[0].values.min(), 8) | ||
| self.assertAlmostEqual(16.0, resampled_cube.B03[1].values.min(), 8) | ||
|
|
||
| def test_transform_cube_downsample_to_months(self): | ||
| cube_config = CubeConfig(time_range=('2010-08-01', '2010-11-30'), | ||
| time_period='2M', | ||
| temporal_resampling=dict( | ||
| downsampling=('min', {})) | ||
| ) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='W', time_periods=12) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertIsNotNone(resampled_cube) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time.values, | ||
| np.array(['2010-09-01T00:00:00', '2010-11-01T00:00:00'], | ||
| dtype=np.datetime64)) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time_bnds.values, | ||
| np.array([['2010-08-01T00:00:00', '2010-10-01T00:00:00'], | ||
| ['2010-10-01T00:00:00', '2010-12-01T00:00:00']], | ||
| dtype=np.datetime64)) | ||
| self.assertEqual((2, 5, 10), resampled_cube.B03.shape) | ||
| self.assertAlmostEqual(0.0, resampled_cube.B03[0].values.min(), 8) | ||
| self.assertAlmostEqual(8.0, resampled_cube.B03[1].values.min(), 8) | ||
|
|
||
| def test_transform_cube_downsample_to_weeks(self): | ||
| cube_config = CubeConfig(time_range=('2010-08-03', '2010-09-10'), | ||
| time_period='2W', | ||
| temporal_resampling=dict( | ||
| downsampling=('max', {})) | ||
| ) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='D', time_periods=32) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertIsNotNone(resampled_cube) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time.values, | ||
| np.array(['2010-08-08T00:00:00', '2010-08-22T00:00:00', | ||
| '2010-09-05T00:00:00'], | ||
| dtype=np.datetime64)) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time_bnds.values, | ||
| np.array([['2010-08-01T00:00:00', '2010-08-15T00:00:00'], | ||
| ['2010-08-15T00:00:00', '2010-08-29T00:00:00'], | ||
| ['2010-08-29T00:00:00', '2010-09-12T00:00:00']], | ||
| dtype=np.datetime64)) | ||
| self.assertEqual((3, 5, 10), resampled_cube.B03.shape) | ||
| self.assertAlmostEqual(10.0, resampled_cube.B03[0].values.min(), 8) | ||
| self.assertAlmostEqual(24.0, resampled_cube.B03[1].values.min(), 8) | ||
| self.assertAlmostEqual(31.0, resampled_cube.B03[2].values.min(), 8) | ||
|
|
||
| def test_transform_cube_upsample_to_months(self): | ||
| cube_config = CubeConfig(time_range=('2011-10-01', '2012-03-31'), | ||
| time_period='2M', | ||
| temporal_resampling=dict( | ||
| upsampling=('interpolate', | ||
| {'kind': 'linear'}) | ||
| )) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='Y', time_periods=2) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertIsNotNone(resampled_cube) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time.values, | ||
| np.array(['2011-11-01T00:00:00', '2012-01-01T00:00:00', | ||
| '2012-03-01T00:00:00'], | ||
| dtype=np.datetime64)) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time_bnds.values, | ||
| np.array([['2011-10-01T00:00:00', '2011-12-01T00:00:00'], | ||
| ['2011-12-01T00:00:00', '2012-02-01T00:00:00'], | ||
| ['2012-02-01T00:00:00', '2012-04-01T00:00:00']], | ||
| dtype=np.datetime64)) | ||
| self.assertEqual((3, 5, 10), resampled_cube.B03.shape) | ||
| self.assertAlmostEqual(0.33561644, | ||
| resampled_cube.B03[0].values.min(), 8) | ||
| self.assertAlmostEqual(0.50273973, | ||
| resampled_cube.B03[1].values.min(), 8) | ||
| self.assertAlmostEqual(0.66712329, | ||
| resampled_cube.B03[2].values.min(), 8) | ||
|
|
||
| def test_transform_cube_upsample_to_weeks(self): | ||
| cube_config = CubeConfig(time_range=('2010-09-01', '2010-10-10'), | ||
| time_period='4W', | ||
| temporal_resampling=dict( | ||
| upsampling=('nearest', {})) | ||
| ) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='M', time_periods=4) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertIsNotNone(resampled_cube) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time.values, | ||
| np.array(['2010-09-12T00:00:00', '2010-10-10T00:00:00'], | ||
| dtype=np.datetime64)) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time_bnds.values, | ||
| np.array([['2010-08-29T00:00:00', '2010-09-26T00:00:00'], | ||
| ['2010-09-26T00:00:00', '2010-10-24T00:00:00']], | ||
| dtype=np.datetime64)) | ||
| self.assertEqual((2, 5, 10), resampled_cube.B03.shape) | ||
| self.assertAlmostEqual(0.0, resampled_cube.B03[0].values.min(), 8) | ||
| self.assertAlmostEqual(1.0, resampled_cube.B03[1].values.min(), 8) | ||
|
|
||
| def test_transform_cube_upsample_to_days(self): | ||
| cube_config = CubeConfig(time_range=('2010-08-14', '2010-08-24'), | ||
| time_period='2D', | ||
| temporal_resampling=dict( | ||
| upsampling=('interpolate', | ||
| {'kind': 'linear'}) | ||
| )) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='W', time_periods=3) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertIsNotNone(resampled_cube) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time.values, | ||
| np.array(['2010-08-15T00:00:00', '2010-08-17T00:00:00', | ||
| '2010-08-19T00:00:00', '2010-08-21T00:00:00', | ||
| '2010-08-23T00:00:00'], | ||
| dtype=np.datetime64)) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time_bnds.values, | ||
| np.array([['2010-08-14T00:00:00', '2010-08-16T00:00:00'], | ||
| ['2010-08-16T00:00:00', '2010-08-18T00:00:00'], | ||
| ['2010-08-18T00:00:00', '2010-08-20T00:00:00'], | ||
| ['2010-08-20T00:00:00', '2010-08-22T00:00:00'], | ||
| ['2010-08-22T00:00:00', '2010-08-24T00:00:00']], | ||
| dtype=np.datetime64)) | ||
| self.assertEqual((5, 5, 10), resampled_cube.B03.shape) | ||
| self.assertAlmostEqual(0.5, | ||
| resampled_cube.B03[0].values.min(), 8) | ||
| self.assertAlmostEqual(0.78571429, | ||
| resampled_cube.B03[1].values.min(), 8) | ||
| self.assertAlmostEqual(1.07142857, | ||
| resampled_cube.B03[2].values.min(), 8) | ||
| self.assertAlmostEqual(1.35714286, | ||
| resampled_cube.B03[3].values.min(), 8) | ||
| self.assertAlmostEqual(1.64285714, | ||
| resampled_cube.B03[4].values.min(), 8) | ||
|
|
||
| def test_transform_cube_downsample_to_years_cftimes(self): | ||
| cube_config = CubeConfig(time_range=('2010-01-01', '2014-12-31'), | ||
| time_period='2Y', | ||
| temporal_resampling=dict( | ||
| downsampling=('min', {})) | ||
| ) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='M', time_periods=24, use_cftime=True) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertIsNotNone(resampled_cube) | ||
| np.testing.assert_equal(resampled_cube.time.values, | ||
| [cftime.DatetimeProlepticGregorian(2011, 1, 1), | ||
| cftime.DatetimeProlepticGregorian(2013, 1, 1)]) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time_bnds.values, | ||
| [[cftime.DatetimeProlepticGregorian(2010, 1, 1), | ||
| cftime.DatetimeProlepticGregorian(2012, 1, 1)], | ||
| [cftime.DatetimeProlepticGregorian(2012, 1, 1), | ||
| cftime.DatetimeProlepticGregorian(2014, 1, 1)]]) | ||
| self.assertEqual((2, 5, 10), resampled_cube.B03.shape) | ||
| self.assertAlmostEqual(0.0, resampled_cube.B03[0].values.min(), 8) | ||
| self.assertAlmostEqual(16.0, resampled_cube.B03[1].values.min(), 8) | ||
|
|
||
| def test_transform_cube_upsample_to_months_cftimes(self): | ||
| cube_config = CubeConfig(time_range=('2011-10-01', '2012-03-31'), | ||
| time_period='2M', | ||
| temporal_resampling=dict( | ||
| upsampling=('interpolate', | ||
| {'kind': 'linear'}) | ||
| )) | ||
| temporal_resampler = CubeResamplerT() | ||
|
|
||
| cube = self._get_cube(time_freq='Y', time_periods=2, use_cftime=True) | ||
|
|
||
| resampled_cube, grid_mapping, cube_config = temporal_resampler.\ | ||
| transform_cube(cube, | ||
| GridMapping.from_dataset(cube), | ||
| cube_config) | ||
| self.assertIsNotNone(resampled_cube) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time.values, | ||
| [cftime.DatetimeProlepticGregorian(2011, 11, 1), | ||
| cftime.DatetimeProlepticGregorian(2012, 1, 1), | ||
| cftime.DatetimeProlepticGregorian(2012, 3, 1)]) | ||
| np.testing.assert_equal( | ||
| resampled_cube.time_bnds.values, | ||
| [[cftime.DatetimeProlepticGregorian(2011, 10, 1), | ||
| cftime.DatetimeProlepticGregorian(2011, 12, 1)], | ||
| [cftime.DatetimeProlepticGregorian(2011, 12, 1), | ||
| cftime.DatetimeProlepticGregorian(2012, 2, 1)], | ||
| [cftime.DatetimeProlepticGregorian(2012, 2, 1), | ||
| cftime.DatetimeProlepticGregorian(2012, 4, 1)]]) | ||
| self.assertEqual((3, 5, 10), resampled_cube.B03.shape) | ||
| self.assertAlmostEqual(0.33561644, | ||
| resampled_cube.B03[0].values.min(), 8) | ||
| self.assertAlmostEqual(0.50273973, | ||
| resampled_cube.B03[1].values.min(), 8) | ||
| self.assertAlmostEqual(0.66712329, | ||
| resampled_cube.B03[2].values.min(), 8) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to be more pythonic here and allow for the no-args case also