Make backup work without buildout

[mauritsvanrees]

Plone 6 might slowly step away from buildout and move to pip. For installing packages this seems best. But what about buildout recipes, like this one? Even when you step away from buildout, you still want an easy way to create backups.

An option I see for collective.recipe.backup:

• Copy most code to a new package zodbbackup. It could be in the same repository at first, at least before doing a release.
• This creates a script bin/zodbbackup. It reads options from the command line and/or from a config file.
• Probably also a script bin/zodbrestore, although it may be easier to call bin/zodbbackup --restore.
• Maybe let the new package create bin/blobbackup, so bin/zodbbackup can mostly call bin/repozo followed by bin/blobbackup. But only if this makes things simpler.
• In the recipe keep only the current __init__.py.
• The recipe still creates scripts like bin/backup, bin/zipbackup.
• bin/backup calls bin/zodbbackup with command line options, or with a path to a config file.
• In case of a config file, the recipe will create it during the buildout run, for example parts/backup we may create backup.conf, zipbackup.conf, maybe (zip)restore.conf.
• And in case it is not clear: without buildout you would pip install zodbbackup and figure out the correct options or config for bin/zodbbackup yourself.

Most tests will move to the new package. The split should make the testing easier. The recipe only needs to test that options from the buildout config correctly end up in the scripts and/or the created config. Plus a few integration tests for the common case, not for all kinds of edge cases: the edge cases can be tested in the new package.

It might be time for some simplifications:

• Python 3 only. Same versions as Plone 6: 3.7, 3.8, 3.9.
• Drop support for additional filestorages. I have myself used Data.fs and Catalog.fs, but that was long ago. If someone still really needs this, the alternative is probably to create one backup script for each filestorage.
• Always use blob_timestamps. I always have this on. Maybe we should first make a release of the current package where the default is changed from false to true.

Probably some other options can be removed:

• enable_fullbackup (default false)
• gzip (default true)
• gzip_blob (backwards compatibility alias for archive_blob)
• keep_blob_days (not needed I think when we always use blob_timestamps, although it surprises me that the docs do not say this)
• maybe only_blobs
• quick (default true)

Does this sound like a plan most people could get on board with?
Are there alternatives?
(Ignoring RelStorage, because then you don't need this recipe at all.)

[mauritsvanrees]

The above can be a lot of work, but it is time I started working on this, get things going, though slowly. I will extract a few issues.

[mauritsvanrees]

I did a lot of cleanup already, a few more PRs are coming in.
Plan is to do a release 5 with this.
And then during the Alpine City Sprint 2023, started today, I might work on splitting the package in a pip-part and a buildout-part. That would be version 6.

I wanted to see if we can set blob_timestamps True as only option, so no more blobstorage.0, etc. But seems a bit too tricky, and we would still need to keep some code so we can restore blob backups with those names. So I will start with logging a warning when you have blob_timestamps = false.

[mauritsvanrees]

I have released collective.recipe.backup 5.0.0. This is still purely Buildout based. But it is a cleaner, smaller base for moving toward a pip-based solution with a small Buildout wrapper.