feat: Add incremental re-hashing API for large models

[edonadei]

Motivation

This PR implements incremental model re-hashing to solve a critical performance problem when signing large ML models. Currently, when a user makes a small change to a large model (e.g., updating README.md in a 500GB model), the entire model must be re-hashed before re-signing, which can take hours. This makes it impractical to update documentation or configuration files in large models.

This PR adds a Python API that reuses digests from previous signatures for unchanged files, only re-hashing files that were added or modified. For a 500GB model with a 1KB documentation update, this reduces re-hashing time from hours to seconds (~500,000x speedup).

Changes

1. signing.manifest_from_signature() - Extracts and verifies manifest from existing signature files, ensuring old signatures haven't been tampered with before reusing their hashes
2. IncrementalSerializer - Core implementation that compares current model state against existing manifest and only re-hashes changed files
3. Config.use_incremental_serialization() - Integrates incremental serializer into hashing API
4. sign_incremental() - High-level convenience API that combines verification + extraction + incremental hashing + signing

Design Decision

Following the discussion in #160, this implementation uses a user-driven approach where changed files are specified via the files_to_hash parameter (e.g., from git diff). This was chosen over automatic change detection because:

• File metadata (mtime) is unreliable across systems and git operations
• Users know what changed via their workflow (git, manual tracking, etc.)
• Keeps implementation simple and reliable
• Works with any file tracking system

The implementation also reuses the existing Verifier.verify() path instead of manually parsing DSSE envelopes, which:

• Eliminates code duplication
• Ensures consistent verification logic
• Adds security by verifying old signatures before trusting their hashes

Test Coverage:

• tests/manifest_test.py: Tests for manifest data structures
• tests/_serialization/incremental_test.py: 7 tests for incremental serializer (new files, modified files, deleted files, mixed scenarios, empty manifests)
• tests/hashing_config_test.py: Tests for incremental serialization parameter extraction
• tests/_signing/signing_test.py: Tests for DSSE payload parsing and verification

Future Work: Based on maintainer feedback, these could be added in follow-up PRs:

• CLI support (--incremental flag)
• Documentation in README
• Git helper utility (get_changed_files_from_git())
• Integration tests with large models

Questions for Maintainers:

• Is the files_to_hash parameter approach acceptable, or would you prefer a different change detection mechanism? ✅ Looks good
• Should CLI support and documentation be in this PR or separate follow-ups? ✅ Will we done in a next PR
• Is the breaking change (adding required verification parameters) acceptable for the incremental signing API?

Testing this PR

# Create and sign an initial model
from model_signing import signing
from pathlib import Path

model_dir = Path("test_model")
model_dir.mkdir()
(model_dir / "weights.bin").write_bytes(b"x" * 1000000)  # 1MB file
(model_dir / "README.md").write_text("Version 1")

# Initial sign
signing.Config().use_elliptic_key_signer(
    private_key=Path("test.key")
).sign(model_dir, "model.sig.v1")

# Modify only README
(model_dir / "README.md").write_text("Version 2 - updated docs")

# Sign incrementally - only re-hashes README.md, reuses weights.bin digest
# Must verify the old signature before reusing its hashes
signing.Config().use_elliptic_key_signer(
    private_key=Path("test.key")
).sign_incremental(
    model_dir,
    old_signature_path="model.sig.v1",
    new_signature_path="model.sig.v2",
    identity="[email protected]",
    oidc_issuer="https://example.com/oauth",
    files_to_hash=[model_dir / "README.md"]
)

# Verify the new signature works
from model_signing import verifying
verifying.Config().use_elliptic_key_verifier(
    public_key=Path("test.pub")
).verify(model_dir, "model.sig.v2")

[edonadei]

@mihaimaruseac if you can take a look at this, I tried to follow up with the last discussions from #160 (from 2024 that's old) in that thread and tried to implement a solution. It's a bit long and probably imperfect, but I'm open to feedback, there's a questions for maintainer sections to start the discussion.

[mihaimaruseac]

Amazing! Will take a look this week (JupyterCon)

[edonadei]

Ping @mihaimaruseac if you have some time for a review :)

[mihaimaruseac]

Sorry for the delay. Looking now

[mihaimaruseac]

I like the general direction of this, but let's try to reuse existing infrastructure as much as possible, instead of duplicating.

I didn't look at the tests too much, but I noticed we have new hardcoded values. Can we use the existing goldens approach for this? Or use the fixture mechanism?

[mihaimaruseac]

This should probably call the verification path first and then use

model-transparency/src/model_signing/_signing/signing.py

Line 353 in 992a56b

def verify(self, signature: Signature) -> manifest.Manifest:

to get the manifest directly without duplication.

[edonadei]

To double check (I didn't implemented it yet). Do you mean to do something like this?

# in manifest.py
def from_signature(
        cls,
        signature_path: pathlib.Path,
        *,
        identity: str,
        oidc_issuer: str,
        use_staging: bool = False,
    ) -> Self:
    signature = sign_sigstore.Signature.read(signature_path)
        verifier = sign_sigstore.Verifier(
            identity=identity,
            oidc_issuer=oidc_issuer,
            use_staging=use_staging,
        )
    return verifier.verify(signature)

[mihaimaruseac]

Yes, but I think this can be at a higher level, not in manifest.py (as that could create a circular dependency). We could do it in signing.py, which can import verifying.py.

[edonadei]

Ok that's what I tried to do! Will refresh the review for you to take a second look at it

[mihaimaruseac]

I think the current approach works, but you need to fix the linting issues (name shadowing)

[mihaimaruseac]

• Is the files_to_hash parameter approach acceptable, or would you prefer a different change detection mechanism?

This is perfect. We'll have users specify these in advance, in a clean pipeline, rather than us having to write and maintain the detection logic

• Should CLI support and documentation be in this PR or separate follow-ups?

It's ok to do it after.

• Would you like me to add a CHANGELOG entry now or wait until the approach is approved?

Same here for later, assuming we collect everything