Skip to content

refactor: model manager v3 #8607

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 119 commits into
base: main
Choose a base branch
from
Draft

refactor: model manager v3 #8607

wants to merge 119 commits into from

Conversation

psychedelicious
Copy link
Collaborator

@psychedelicious psychedelicious commented Oct 9, 2025 •
edited
Loading

Summary

This PR:

  • Refactors the model manager's identification (aka probing/classification) system. The new system is an evolution of the classify API introduced in Model classification api #7742. All legacy probe logic has been ported over to the new system, too.
  • Introduces the concept of "unknown" models (i.e. a model that we couldn't positively identify) so that a failure to identify a model doesn't require the user to re-download the model.
  • Normalizes model storage into a flat directory structure.
  • Introduces correctness tests for model identification (aka probing/classification). Previously, we did not test for correctness except for a very small number of cases. There is a detailed guide for adding test cases, which is pretty easy.

It adds 3 migrations:

  • Update the models table, removing the UNIQUE constraint on the combination of name, base and model type. The only unique thing for models is now the model's path on disk.
  • Migrate model configs in the DB where necessary.

This PR was previously split into 3 smaller PRs:

TODO

  • Update the model manager UI to let users fix an incorrectly-identified model. This is partially working.
  • Add guide for adding support for a new model type (identification, loading, testing).

Related Issues / Discussions

Gajillions of related issues.

QA Instructions

  • Because this PR does both DB and model file storage migrations, it needs careful testing on backed-up Invoke installs. I will add more detail about how to test it when I do the next alpha release.
  • The new model identification system works differently from the legacy probe and it is likely some things won't identify correctly. We'll need to test installing a variety of models, ideally adding them to the new model identification test suite.

Merge Plan

This will need an alpha release before RC.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

@github-actions github-actions bot added api python PRs that change python files Root invocations PRs that change invocations backend PRs that change backend files services PRs that change app services frontend PRs that change frontend files python-tests PRs that change python tests labels Oct 9, 2025
@psychedelicious psychedelicious mentioned this pull request Oct 9, 2025
Closed
5 tasks
@psychedelicious psychedelicious force-pushed the psyche/refactor/mm-3 branch 2 times, most recently from 5044e15 to e155f10 Compare October 10, 2025 01:21
@psychedelicious
feat(mm): add UnknownModelConfig
3af504e
@psychedelicious
refactor(ui): move model categorisation-ish logic to central location…
102fb26 
…, simplify model manager models list
@psychedelicious
refactor(ui)refactor(ui): more cleanup of model categories
48ff8f9
@psychedelicious
refactor(ui): remove unused excludeSubmodels
39ad14b 
I can't remember what this was for and don't see any reference to it.
Maybe it's just remnants from a previous implementation?
@psychedelicious
feat(nodes): add unknown as model base
55e3ccf
@psychedelicious
chore(ui): typegen
df954eb
@psychedelicious
feat(ui): add unknown model base support in ui
3ea6b39
@psychedelicious
feat(ui): allow changing model type in MM, fix up base and variant se…
2a5a84a 
…lects
@psychedelicious
feat(mm): omit model description instead of making it "base type file…
0e79fed 
…name model"
@psychedelicious
feat(app): add setting to allow unknown models
c5418ef
@psychedelicious
feat(ui): allow changing model format in MM
3d270ca
@psychedelicious
feat(app): add the installed model config to install complete events
78d9659
@psychedelicious
chore(ui): typegen
e7530f6
@psychedelicious
feat(ui): toast warning when installed model is unidentified
c4a4e8b
@psychedelicious
docs: update config docstrings
f421b7b
@psychedelicious
chore(ui): typegen
7ccb689
@psychedelicious
tests(mm): fix test for MM, leave the UnknownModelConfig class in the…
4d3715e 
… list of configs
@psychedelicious
tidy(ui): prefer types from zod schemas for model attrs
e4d68ca
@psychedelicious
chore(ui): lint
7fcd140
@psychedelicious
fix(mm): ModelOnDisk skips dirs when looking for weights
2553278 
Previously a path w/ any of the known weights suffixes would be seen as
a weights file, even if it was a directory. We now check to ensure the
candidate path is actually a file before adding it to the list of
weights.
@psychedelicious
feat(mm): add method to get main model defaults from a base
8222d55
@psychedelicious
feat(mm): do not log when multiple non-unknown model matches
0865c99
@psychedelicious
refactor(mm): continued iteration on model identifcation
1373b44
@psychedelicious
tests(mm): refactor model identification tests
4334cf7 
Overhaul of model identification (probing) tests. Previously we didn't
test the correctness of probing except in a few narrow cases - now we
do.

See tests/model_identification/README.md for a detailed overview of the
new test setup. It includes instructions for adding a new test case. In
brief:

- Download the model you want to add as a test case
- Run a script against it to generate the test model files
- Fill in the expected model type/format/base/etc in the generated test
metadata JSON file

Included test cases:
- All starter models
- A handful of other models that I had installed
- Models present in the previous test cases as smoke tests, now also
tested for correctness
@psychedelicious
fix(mm): omit type/format/base when creating unknown config instance
0befee2
@psychedelicious
feat(mm): use ValueError for model id sanity checks
1bdc20c
@psychedelicious
feat(mm): add flag for updating models to allow class changes
ba91c4f
@psychedelicious
tests(mm): fix remaining MM tests
ca0bdac
@psychedelicious
feat: allow users to edit models freely
fca6ee3
@psychedelicious
feat(ui): add warning for model settings edit
438286c
@psychedelicious
tests(mm): flux state dict tests
c011ea5
@psychedelicious
tidy: remove unused file
5a006b9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@blessedcoolant blessedcoolant Awaiting requested review from blessedcoolant blessedcoolant will be requested when the pull request is marked ready for review blessedcoolant is a code owner

@maryhipp maryhipp Awaiting requested review from maryhipp maryhipp will be requested when the pull request is marked ready for review maryhipp is a code owner

@hipsterusername hipsterusername Awaiting requested review from hipsterusername hipsterusername will be requested when the pull request is marked ready for review hipsterusername is a code owner

@lstein lstein Awaiting requested review from lstein lstein will be requested when the pull request is marked ready for review lstein is a code owner

@ebr ebr Awaiting requested review from ebr ebr will be requested when the pull request is marked ready for review ebr is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

api backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests Root services PRs that change app services

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@psychedelicious