Struggling to use obstore with s3-compatible Open Storage Network bucket

[rsignell]

I've got an s3-compatible Open Storage Network bucket that has anonymous read access. With fsspec, I would do:

import fsspec
fs_read = fsspec.filesystem('s3', anon=True, endpoint_url='https://usgs.osn.mghpcc.org')
fs_read.ls('esip/rsignell/testing')

and for writing, I specify AWS keys as environment vars and then do:

fs_write = fsspec.filesystem('s3', anon=False, endpoint_url='https://usgs.osn.mghpcc.org')
fs_write.upload('myfile.nc', 's3://esip/rsignell/testing/')

How can I instantiate S3Store objects for these two use cases?

[kylebarron]

I think you should be able to do something like

store = S3Store(skip_signature=True, endpoint="https://usgs.osn.mghpcc.org")

I think you may need

store = S3Store(skip_signature=True, endpoint="https://usgs.osn.mghpcc.org", virtual_hosted_style_request=True)

[rsignell]

I'm still getting "forbidden". This should be reproducible:

from obstore.store import S3Store
from virtualizarr import open_virtual_dataset
from virtualizarr.parsers import NetCDF3Parser
from virtualizarr.registry import ObjectStoreRegistry

bucket = "s3://esip"
store = S3Store(bucket, skip_signature=True, endpoint="https://usgs.osn.mghpcc.org", virtual_hosted_style_request=True)
registry = ObjectStoreRegistry({bucket: store})
parser = NetCDF3Parser()
path = "rsignell/testing/shyfem/nos00.nc"
url = f"{bucket}/{path}"
vds = open_virtual_dataset(url, registry=registry, parser=parser, 
                           loadable_variables=[])

I guess this could be a virtualizarr issue?

But this returns nothing also:

import obstore 
list_stream_with_prefix = obstore.list(store, prefix="esip/rsignell/")
for batch in list_stream_with_prefix:
    for meta in batch:
        print(f"  Name: {meta.path}, Size: {meta.size} bytes")

User error?

[kylebarron]

You should be able to provide a reproducible example with only obstore code. Can you perform any list calls or downloads on that bucket?

[rsignell]

Yep, sorry, here's an example only with obstore. The code below returns no errors, but also lists no files.
User error? (I'm a little confused about bucket, store & prefix)

import obstore
from obstore.store import S3Store

bucket = "s3://esip"
store = S3Store(bucket, skip_signature=True, endpoint="https://usgs.osn.mghpcc.org", virtual_hosted_style_request=True)

list_stream_with_prefix = obstore.list(store, prefix="esip/rsignell/testing/shyfem")
for batch in list_stream_with_prefix:
    for meta in batch:
        print(f"  Name: {meta.path}, Size: {meta.size} bytes")

For reference, this works:

import fsspec
fsspec.filesystem('s3', anon=True, endpoint_url='https://usgs.osn.mghpcc.org')
fs.ls('esip/rsignell/testing/shyfem')

returning

['esip/rsignell/testing/shyfem/nos00.nc',
 'esip/rsignell/testing/shyfem/nos01.nc']

[rsignell]

Okay, @maxrjones figured this out!

endpoint = "https://usgs.osn.mghpcc.org"
path = "esip/rsignell/testing/shyfem"
file = "nos00.nc"
scheme = "s3://"

# Create the S3Store
store = S3Store.from_url(f"{scheme}{path}", endpoint=endpoint, skip_signature=True, region='us-east-1')
for obs in obstore.list(store):
    print(obs)

produces:

[{'path': 'nos00.nc', 'last_modified': datetime.datetime(2025, 7, 17, 21, 1, 10, 978000, tzinfo=datetime.timezone.utc), 'size': 2068427360, 'e_tag': '"2742f2bc8880d316d866e23fd628e576-395"', 'version': None}, {'path': 'nos01.nc', 'last_modified': datetime.datetime(2025, 7, 11, 17, 46, 32, 270000, tzinfo=datetime.timezone.utc), 'size': 2068427836, 'e_tag': '"d3ad0e3296934431521f7aff8c2e1ddf-395"', 'version': None}]

[kylebarron]

You shouldn't need the region parameter; does it work without it?