Lots of accesses generated sometimes when using simplecache::

[zlmarshall]

Hello,

I have a slightly unusual question I'm sure, but I'm a bit at a loss as to where to ask for help. Redirects are welcome. I'm opening a file with uproot, which uses fsspec as part of the backend. I'm using an address like "simplecache::https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodD.exactly4lep.root" to try to get the simple file caching provided by fsspec. I've also tried with filecache::.

In both cases, I'm enabling additional logging via:

import logging
fslogc = logging.getLogger("fsspec.cached")
fslogh = logging.getLogger("fsspec.http")
fslogc.setLevel(10)
fslogh.setLevel(10)

The output I get from fsspec for most files looks something like this:

DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.cached:Opening local copy of https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root
DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data15_periodE.exactly4lep.root

The logging in fsspec.http isn't the most useful, but I guess the point is clear here: we hit the cache a fair bit. I don't pretend to understand the ordering of the output, but that's ok.

For some files — and I've not found a pattern yet, it seems somewhat random to me, and does not always happen even with the same file — I instead get hundreds of lines like:

DEBUG:fsspec.http:https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data16_periodB.exactly4lep.root

with no corresponding cache logging. Digging a bit more, the output seems to almost all be coming from http.HTTPFileSystem::_cat_file. I can't find a reason why some files would generate a huge number of requests like this. I've tried increasing the block size and the chunk size parameters (I hope correctly — they have to be passed along by uproot, so there's a chance something's going wrong there), but that doesn't seem to have any impact on this. In the end, the problem that I face is that the http server throws a 429:

HTTPError: 429 Client Error: Too Many Requests for url: https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_mc_364242.Sherpa_222_NNPDF30NNLO_WWW_3l3v_EW6.exactly4lep.root

So: does anyone have some explanation for why this might be happening, how to make sure simplecache:: is doing the right thing, or anything else I can try or test to get this running?

Thanks,
Zach

[zlmarshall]

Sorry, should've said that this was also reported to the uproot team here:
scikit-hep/uproot5#1452
They don't seem to have more ideas at the moment.

[martindurant]

It's not at all obvious how uproot interfaces with fsspec. The code sample looked like

with uproot.open("simplecache::https://....root") as f:
    # your code here

I wonder whether uprooot accepts the following instead

with fsspec.open("simplecache::https://....root") as f:
    with uproot.open(f) as ff:
        # code

or similar.

Alternatively, fsspec.open_local can give you the local path of the cached file, so you can pass that instead.

As you suggest, we would expect there to be only one call to get_file(), although that might result in multiple requests, one per block.

[zlmarshall]

Hi all,

I've gotten a little further in understanding this, but I think I'm still stuck, so I wanted to pass along a little update. Here's a much simpler reproducer of our problem:

import uproot
tree = uproot.open('simplecache::https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data16_periodB.exactly4lep.root:analysis')
for data in tree.iterate(['lep_pt', 'lep_eta', 'lep_phi', 'lep_e', 'lep_charge', 'lep_type', 'trigE', 'trigM', 'lep_isTrigMatched', 'lep_isLooseID', 'lep_isMediumID', 'lep_isLooseIso', 'lep_type', 'filteff', 'kfac', 'xsec', 'mcWeight', 'ScaleFactor_PILEUP', 'ScaleFactor_ELE', 'ScaleFactor_MUON', 'ScaleFactor_LepTRIGGER', 'sum_of_weights', 'lep_n'],library='ak’,entry_stop=72544):
    # Number of events in this batch
    nIn = len(data)
    print(nIn)

uproot.open reads a header out of the file, which is a great opportunity for caching. It seems to work as well. Looking in SimpleCacheFileSystem._open, I find that the path https://opendata.cern.ch//eos/opendata/atlas/rucio/opendata/ODEO_FEB2025_v0_exactly4lep_data16_periodB.exactly4lep.root is provided with access mode rb. The first time through, it gets to the self.fs.get_file(path, fn) call and eventually returns a local location. On the next few calls, it has a cached file to work with. I can see the local file in the cache, and it seems to be about what I want it to be in terms of file type and size. So far so good!

Then comes to looping part, in tree.iterate. Those generate calls to HTTPFileSystem._cat_file, and lots of them. Because they are asynchronous, they seem to all get filed and then fired off at about the same time. None of these seem to hit the cache, and that's what causes the error.

The accesses in there seem to use direct calls to cat_file in fsspec.implementations.cached.SimpleCacheFileSystem. It seems as though that is not hitting the cache in these cases, but I can't seem to track down why. It feels like there's some access that's being allowed that's hitting a base class instead of access through the cache.

Because of all the futures in this code, it's rather laborious to track down what's going on. Suggestions and/or help are welcome, of course.

Best,
Zach

[martindurant]

It is possible, then, that all you need if for cat_file/_cat_file to be implemented for SimpleCache . The classes were definitely built for the file-lie API in mind, but I am surprised that cat is falling back to the remote filesystem.

[zlmarshall]

That sounds like a winner to me! Is that something that you or one of the devs has some time to implement, even in draft form? I'm happy to help with testing, or even to give an implementation a try if you give me a few hints about where to start and what style you all prefer.

Best,
Zach

[martindurant]

Please ping me next week for a reminder.

I see that cat and cat_ranges (without the leasing "_") are defined for SimpleCache and should call back to AbstractFileSystem's cat_file, which uses open() and should pick up the local file. But if we are hitting _cat_file instead (which would be expected to be async), then that might be the problem - the implementation should be identical once the file is cached.

[zlmarshall]

Quick update: I did a 0th order implementation of cat_file in SimpleCache. That was easy to do, but doesn't seem to solve the problem. That's perhaps not a surprise, because cat_file goes through open, which in turn goes through _open, which is implemented in SimpleCache.

Looking at _cat_file in HTTPFileSystem, I'm not sure how to do this. HTTPFileSystem inherits from AsyncFileSystem, which is an AbstractFileSystem. SimpleCache is also an AbstractFileSystem — but is a different one. If anything is going through the HTTPFileSystem for this, or any AsyncFileSystem, then I think we're dead in the water? _cat_file itself is an asynchronous method, and no such thing exists in the AbstractFileSystem... do we actually need to 'invent' an AsyncFileSystem that uses the caching here? Or make SimpleCache inherit from AsyncFileSystem, so that it can inherit methods as well?

Sorry, this is more complicated than I had hoped!

[zlmarshall]

Ah — sorry, crossing message. Yes, indeed _cat_file is what we're after. I appreciate you're support, and will happily ping next week. I hope the above saved you... 15 seconds of thinking? :)

[martindurant]

Closed via #1881