Add Museums Victoria fetch #223

sbarhin · 2025-10-30T15:49:54Z

Fixes

Fixes Add Museum Victoria as data source #215 by @sbarhin

Description

This PR implements automation for Museum Victoria data fetching as discussed in issue #215. The implementation follows the established patterns from existing fetch scripts.
This purpose of this file is to fetch all the records from the Museum Victoria API, then saving the necessary response fields needed for the next phase (processing phase).

Fetches data for all record types (article, item, specimen, species) from the Museum Victoria API
Prepares and saves meaningful responses into a csv file under the data/2025Q4/1-fetch directory
Next actions will be to process and report the data once the fetching script is approved by reviewers

Checklist

I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
My pull request doesn't include code or content generated with AI.
My pull request has a descriptive title (not a vague title like Update index.md).
My pull request targets the default branch of the repository (main or master).
My commit messages follow best practices.
My code follows the established code style of the repository.
I added or updated tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no
visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

sbarhin · 2025-10-30T15:55:06Z

@oree-xx I have opened a new pull request. I guess that is much better

oree-xx · 2025-10-30T15:56:35Z

@sbarhin ohh okay great.

data/2024Q2/1-fetch/gcs_fetched.csv

data/2024Q3/1-fetch/gcs_fetched.csv

pre-automation/flickr/dataset/license_total.csv

scripts/1-fetch/museums_victoria_fetch.py

scripts/2-process/museums_process.py

scripts/3-report/museums_report.py

scripts/1-fetch/museums_victoria_fetch.py

sbarhin · 2025-10-31T09:29:17Z

@TimidRobot There are several file changes in my PR, this is due to pulling from the main branch where I believe you merged a certain PR. These changes have taken effect in my branch hence those file changes in my PR.

TimidRobot · 2025-10-31T16:25:23Z

@TimidRobot There are several file changes in my PR, this is due to pulling from the main branch where I believe you merged a certain PR. These changes have taken effect in my branch hence those file changes in my PR.

Please revisit the documentation on keeping a branch/fork synchronized with upstream and on resolving merge conflicts. This PR won't be reviewed or merged while these issue are present.

sbarhin · 2025-10-31T18:31:52Z

@TimidRobot There are several file changes in my PR, this is due to pulling from the main branch where I believe you merged a certain PR. These changes have taken effect in my branch hence those file changes in my PR.

Please revisit the documentation on keeping a branch/fork synchronized with upstream and on resolving merge conflicts. This PR won't be reviewed or merged while these issue are present.

I will do that please. Thank you

sbarhin · 2025-10-31T18:48:23Z

@TimidRobot I believe we are good now

scripts/1-fetch/museums_victoria_fetch.py

sbarhin · 2025-11-02T00:10:54Z

The script takes too long to run (I canceled after 10 minutes).

Please add a --limit option so that it can be developed and tested without taking the full time.

@TimidRobot Please should the --limit apply individual record types or the total number of records altogether?

TimidRobot · 2025-11-05T08:30:51Z

The script takes too long to run (I canceled after 10 minutes).
Please add a --limit option so that it can be developed and tested without taking the full time.

@TimidRobot Please should the --limit apply individual record types or the total number of records altogether?

@sbarhin Which implementation will satisfies the stated goal?

TimidRobot · 2025-11-05T08:36:39Z

@sbarhin force-pushed the museums branch from b7d41b6 to 1c644a4 5 days ago

@sbarhin force-pushed the museums branch from 79e6ef5 to 6ee4793 3 days ago

@sbarhin force-pushed the museums branch from aa8774c to 6ee4793

@sbarhin force pushes are generally a worst practice and should be avoided when unnecessary.

sbarhin · 2025-11-05T10:48:03Z

The script takes too long to run (I canceled after 10 minutes).
Please add a --limit option so that it can be developed and tested without taking the full time.

@TimidRobot Please should the --limit apply individual record types or the total number of records altogether?

@sbarhin Which implementation will satisfies the stated goal?

I think per individual record type will do

sbarhin · 2025-11-06T08:26:59Z

@TimidRobot Please what are the next steps? Seems everything is resolved now

scripts/1-fetch/museums_victoria_fetch.py

TimidRobot · 2025-11-06T09:30:34Z

scripts/1-fetch/museums_victoria_fetch.py

+                    licence_data = media_item.get("licence")
+
+                    # COUNTING THE UNIQUE LICENCE TYPES
+                    license_short_name = licence_data.get("shortName")


Unfortunately, the license short name is not reliable (does not include version number).

scripts/1-fetch/museums_victoria_fetch.py

…into museums

sbarhin · 2025-11-06T11:07:43Z

@TimidRobot Please I have made the necessary changes

sbarhin · 2025-11-14T06:40:12Z

@TimidRobot Hello there

TimidRobot · 2025-11-14T07:22:40Z

scripts/1-fetch/museums_victoria_fetch.py

+        records_processed = 0
+        current_page = 1
+        total_pages = None
+        per_page = min(PER_PAGE, args.limit) if args.limit else PER_PAGE


I don't see any value in changing per_page. All it does is slow down the execution of the script. It does not address the issue of the script taking a very long time to complete.

Again, your comment suggested the limit would apply to the total records of each type.

The default per_page is 100 from the documentation. With the presence of --limit say 50, we fetch 50 records perpage for each record type instead of the default 100. No fetch is done once we hit the limit (50) regardless of the number of pages for that record type.
If per_page isn't altered, then the --limit won't work. It would fetch the 100 records anyway.

Yes, the --limit applies to the total records for each type regardless of the total pages for a record type. So, if I don't override the perpage with the limit (if provided) it would fetch the 100 records per page and proceed to the next page if there are more records for that type.

@TimidRobot Please see this

@TimidRobot hello there

TimidRobot · 2025-11-14T07:23:56Z

scripts/1-fetch/museums_victoria_fetch.py

+def get_requests_session():
+    """
+    Returns a configured requests session with retries and a User-Agent.
+    """
+    return shared.get_session()


This function doesn't actually do anything and should be removed.

sbarhin requested review from a team as code owners October 30, 2025 15:49

sbarhin requested review from TimidRobot and possumbilities and removed request for a team October 30, 2025 15:49

cc-open-source-bot added this to TimidRobot Oct 30, 2025

cc-open-source-bot moved this to In review in TimidRobot Oct 30, 2025

TimidRobot reviewed Oct 31, 2025

View reviewed changes

data/2024Q2/1-fetch/gcs_fetched.csv Show resolved Hide resolved

TimidRobot reviewed Oct 31, 2025

View reviewed changes

data/2024Q3/1-fetch/gcs_fetched.csv Show resolved Hide resolved

TimidRobot reviewed Oct 31, 2025

View reviewed changes

pre-automation/flickr/dataset/license_total.csv Show resolved Hide resolved

TimidRobot reviewed Oct 31, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Show resolved Hide resolved

TimidRobot reviewed Oct 31, 2025

View reviewed changes

scripts/2-process/museums_process.py Outdated Show resolved Hide resolved

TimidRobot reviewed Oct 31, 2025

View reviewed changes

scripts/3-report/museums_report.py Outdated Show resolved Hide resolved

TimidRobot reviewed Oct 31, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

TimidRobot reviewed Oct 31, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

TimidRobot reviewed Oct 31, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

TimidRobot self-assigned this Oct 31, 2025

TimidRobot changed the title ~~Add musuems_fetch.py~~ Add Museum Victoria fetch Oct 31, 2025

sbarhin force-pushed the museums branch from 6a04d00 to 66ccdf9 Compare October 31, 2025 09:19

sbarhin force-pushed the museums branch from b7d41b6 to 1c644a4 Compare October 31, 2025 18:46

TimidRobot changed the title ~~Add Museum Victoria fetch~~ Add Museums Victoria fetch Nov 1, 2025

TimidRobot reviewed Nov 1, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Show resolved Hide resolved

TimidRobot reviewed Nov 1, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

TimidRobot reviewed Nov 1, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

sbarhin added 3 commits November 1, 2025 23:51

fetching meaningful data about the size and distribution of the commons

ad4ee2b

Reviewed PR changes

abc2576

Reviewed PR changes

6ee4793

sbarhin force-pushed the museums branch from 79e6ef5 to 6ee4793 Compare November 2, 2025 00:06

This comment was marked as outdated.

Sign in to view

sbarhin force-pushed the museums branch from aa8774c to 6ee4793 Compare November 3, 2025 16:07

Fix: Making the script file executable

d326271

TimidRobot reviewed Nov 6, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

TimidRobot reviewed Nov 6, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

TimidRobot reviewed Nov 6, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

TimidRobot reviewed Nov 6, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

TimidRobot reviewed Nov 6, 2025

View reviewed changes

scripts/1-fetch/museums_victoria_fetch.py Outdated Show resolved Hide resolved

sbarhin added 7 commits November 6, 2025 10:05

Merge branch 'main' of https://github.com/creativecommons/quantifying …

855bb5a

…into museums

Implemented the use of shared.get_session()

1dc68e8

Fix: --limit fetching per record type

2f56057

Fix: --limit fetching per record type

e752f7f

Improvement: Added version number to license data

2482158

Fix: Made data files reusable

1e3a1ce

Run pre-commit to ensure integrity

2f45937

TimidRobot reviewed Nov 14, 2025

View reviewed changes

Uh oh!

Add Museums Victoria fetch #223

Are you sure you want to change the base?

Add Museums Victoria fetch #223

Conversation

sbarhin commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Fixes

Description

Checklist

Developer Certificate of Origin

Uh oh!

sbarhin commented Oct 30, 2025

Uh oh!

oree-xx commented Oct 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbarhin commented Oct 31, 2025

Uh oh!

TimidRobot commented Oct 31, 2025

Uh oh!

sbarhin commented Oct 31, 2025

Uh oh!

sbarhin commented Oct 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbarhin commented Nov 2, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

TimidRobot commented Nov 5, 2025

Uh oh!

TimidRobot commented Nov 5, 2025

Uh oh!

sbarhin commented Nov 5, 2025

Uh oh!

sbarhin commented Nov 6, 2025

Uh oh!

Uh oh!

TimidRobot Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbarhin commented Nov 6, 2025

Uh oh!

sbarhin commented Nov 14, 2025

Uh oh!

TimidRobot Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbarhin Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

sbarhin Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

sbarhin Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

sbarhin Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

TimidRobot Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

sbarhin commented Oct 30, 2025 •

edited

Loading

TimidRobot Nov 14, 2025 •

edited

Loading