feat: Multi-database and AWS S3 storage support

[DevStarlight]

Description

This PR introduces comprehensive database and storage flexibility to the Hyperfy server infrastructure. The changes add multi-database support (PostgreSQL, MySQL, SQLite) and AWS S3 storage capabilities, improving deployment flexibility and scalability options.

Components affected:

• Server infrastructure (database layer)
• Storage management system
• Asset handling and file storage
• World maintenance scripts

Key improvements:

• Database flexibility: Generic DB_TYPE configuration supporting PostgreSQL, MySQL, and SQLite
• Storage options: STORAGE_TYPE configuration for local file system or AWS S3 storage
• Enhanced deployment: Better support for cloud deployments and production environments
• Improved maintenance: Updated clean-world script supports all database types

Fixes # (issue)

Type of change

Please delete options that are not relevant.

• New component
• Component enhancement

How Has This Been Tested?

Please describe your tests:

• Component functionality tests
• Integration tests with other Hyperfy components
• World integration testing
• Cross-browser testing

Test Configuration:

• Hyperfy Version: 0.13.0
• Test world setup: Local development environment with SQLite fallback
• Browser(s): Chromium
• Node.js version: v22.13.1

Checklist:

• My code follows Hyperfy's component architecture
• I have performed a self-review
• I have documented the component's usage
• I have tested the component in multiple scenarios
• My changes maintain compatibility with existing worlds
• I have updated the component documentation if needed

[ashconnell]

Just some quick observations, let me know your thoughts.

[ashconnell]

These changes would default to postgres for new people attempting to quickly spin up new worlds and I think we should continue to stick to sqlite as the default, so that people don't have to spend a whole lot of time launching a postgres docker container etc.

Additionally these are a lot of new environment variables that (might) be simplified by just using URI's like this:

# by default set to empty which uses sqlite in the world folder
DB_TYPE=
DB_URL=

# optionally configured to connect to a remote postgres database
DB_TYPE=pg
DB_URL=https://username:password@localhost:5432/database?schema=public

I don't think many people use mysql so i'd be happier to just not support that right now.

[DevStarlight]

I find your suggestion fine and I've updated both the code and the .env.example file to simplify the database setup as you recommended. Now, only two environment variables are used:

• DB_TYPE (leave empty for SQLite, or set to pg for Postgres)

• DB_URL (leave empty for SQLite, or provide a Postgres connection URI)

MySQL support and all the old related variables have been removed. This makes the default experience much simpler for new users, who can now spin up worlds with SQLite out of the box.

[ashconnell]

S3 has become a bit of a standard, for example CloudFlare R2 can actually use S3-specified libraries for managing files on R2. So perhaps 'aws' should just be called 's3'?

[DevStarlight]

You're right!

I've updated the codebase to use s3 instead of aws for the storage type and environment variables. This makes the storage integration more generic and compatible with any S3-compatible provider (such as CloudFlare R2, DigitalOcean Spaces, etc.), not just AWS S3.

All relevant code, environment variables, and documentation have been updated accordingly.

[ashconnell]

Maybe this could be done in the future, but maintaing two separate world clean scripts might be better abstracted so that there are functions like deleteFileS3() and deleteFileLocal() so that any changes to the general logic in a shared world-clean script would automatically maintain both.

[DevStarlight]

I've refactored the clean-world scripts to use shared utilities with functions like deleteFileS3() and deleteFileLocal().
Now both storage types stay in sync automatically when we update the cleanup logic - no more maintaining two separate scripts! The code is much cleaner and easier to maintain.

The scripts are organized in a nice modular structure (scripts/clean-world/) but still work exactly the same way with npm run world:clean.

Thanks for the feedback - this is definitely a much better approach!

[ashconnell]

Again highly likely that this will work with Cloudflare R2 as its S3 compatible, so we could just call this S3Storage.js etc and drop the AWS naming.

[ashconnell]

Is it possible to rebase this branch so that it doesn't have a bunch of dev commits merged back on top of it?

This helps a lot to keep git history clean

[DevStarlight]

I did a rebase of the branch with dev and I think I didn't mess up anything 🤞!

[ashconnell]

pg here twice

[ashconnell]

can we do uri's for S3 too?

[DevStarlight]

Implemented S3 URIs as suggested!

Now you can use STORAGE_URL=s3://bucket-name/folder-path/assets/ with standard AWS credentials, (just like DB_URL approach).

[ashconnell]

we're still fragmenting here, eg any changes to the way cleaning works needs to be carefully updated in both of these and is likely to diverge because they are very different from each other. this is a risky piece of code.

i think this could be mitigated by having a single control flow that uses an abstraction to read and write from the db and files.

for example we have this code currently:

let blueprints = new Set()
const blueprintRows = await db('blueprints')
for (const row of blueprintRows) {
  const blueprint = JSON.parse(row.data)
  blueprints.add(blueprint)
}

this would work with both database options.

but then we have this:

console.log(`deleting ${filesToDelete.length} assets`)
for (const fileAsset of filesToDelete) {
  const fullPath = path.join(assetsDir, fileAsset)
  if (!DRY_RUN) {
    fs.removeSync(fullPath)
  }
  console.log('delete asset:', fileAsset)
}

which only works with the filesystem, but could be rewritten to work with both (this is an example only):

console.log(`deleting ${filesToDelete.length} assets`)
for (const fileAsset of filesToDelete) {
  if (!DRY_RUN) {
    await storage.deleteAsset(fileAsset)
  }
  console.log('delete asset:', fileAsset)
}

this would allow us to have a SINGLE script for world cleaning instead of needing to keep two in sync and risk them diverging/breaking

[DevStarlight]

I understand your concern about uncontrolled growth - if we end up needing more storage types (Azure, GCP, etc.), we'd indeed need to rethink the architecture toward something more modular.

I've consolidated everything into a single script (scripts/clean-world.mjs) that completely eliminates duplication. The solution maintains the simple and direct style of the original code, but adds conditional AWS S3 support.

Does this approach seem better? It maintains simplicity while completely eliminating the risk you mentioned.