Skip to content

Ensure consistent download_dataset for Copernicus data #522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

simone-silvestri
Copy link
Collaborator

@simone-silvestri simone-silvestri commented May 14, 2025

See the comment below:

The problem with Copernicus that I was hitting here, is that it can be downloaded in one file for all the dates or in different files for all the dates and the code does not enforce a unique way now, i.e. some methods assume we have only one file per Metadata while other methods (metadata_path in this case) assumes we have one file per date (i.e. one file per Metadatum). Here I am changing metadata_path to point to only one file per Metadata, however, I think it would be best to download one file per date so we can reuse downloaded files if we want to change the date. I will change the name of this PR to reflect this change.

@simone-silvestri
Copy link
Collaborator Author

simone-silvestri commented May 21, 2025

The problem with Copernicus that I was hitting here, is that it can be downloaded in one file for all the dates or in different files for all the dates and the code does not enforce a unique way now, i.e. some methods assume we have only one file per Metadata while other methods (metadata_path in this case) assumes we have one file per date (i.e. one file per Metadatum).
here I am changing metadata_path to point to only one file per Metadata, however, I think it would be best to download one file per date so we can reuse downloaded files if we want to change the date.
I will change the name of this PR to reflect this change.

@simone-silvestri simone-silvestri changed the title Fix a couple of bugs with Metadata Fix consistency issues in download_dataset for Copernicus May 21, 2025
@simone-silvestri simone-silvestri changed the title Fix consistency issues in download_dataset for Copernicus Ensure consistent download_dataset for Copernicus data May 21, 2025
Copy link

codecov bot commented May 21, 2025

Codecov Report

Attention: Patch coverage is 12.50000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 14.51%. Comparing base (8c0bb84) to head (ecbd738).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
ext/ClimaOceanPythonCallExt.jl 14.28% 6 Missing ⚠️
src/DataWrangling/Copernicus/Copernicus.jl 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #522      +/-   ##
==========================================
- Coverage   14.54%   14.51%   -0.04%     
==========================================
  Files          48       48              
  Lines        2763     2770       +7     
==========================================
  Hits          402      402              
- Misses       2361     2368       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@@ -58,8 +58,8 @@ copernicusmarine_dataset_id(::GLORYSMonthly) = "cmems_mod_glo_phy_my_0.083deg_P1
struct CMEMSHourlyAnalysis <: CopernicusDataset end
copernicusmarine_dataset_id(::CMEMSHourlyAnalysis) = "cmems_mod_glo_phy_anfc_0.083deg_PT1H-m"

CopernicusMetadata{D} = Metadata{<:CopernicusDataset, D}
CopernicusMetadatum = Metadatum{<:CopernicusDataset}
const CopernicusMetadata{D} = Metadata{<:CopernicusDataset, D}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Metadata has 3 params, right?

Copy link
Collaborator Author

@simone-silvestri simone-silvestri May 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I am not sure they are all needed because this is just dispatch type. For example

const ECCOMetadata{D} = Metadata{<:SomeECCODataset, D}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can merge this, then I'll add the extra parameter introduced in #489 everywhere.

@navidcy navidcy added the data wrangling We must feed the models so they don't get cranky label May 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data wrangling We must feed the models so they don't get cranky
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants