Skip to content

[bug]: OOM with sam-vit-huge #8046

Open
@xerohaveit

Description

@xerohaveit

Is there an existing issue for this problem?

  • I have searched the existing issues

Operating system

Windows

GPU vendor

Nvidia (CUDA)

GPU model

RTX 4050

GPU VRAM

6gb

Version number

5.12.0

Browser

community launcher

Python dependencies

No response

What happened

my invokeai.yaml contains these args:

enable_partial_loading: true
keep_ram_copy_of_weights: false
clear_queue_on_startup: true
lazy_offload: true
pytorch_cuda_alloc_conf: "backend:cudaMallocAsync"

using object detection now gives me OOM error, I don't remember since which invokeai version exactly this started to happen (I guess around last February/March not sure) , but I remember it was running normally on an older version since the transition to the launcher app.
no matter the input and output size of the image, it was running normally back then and now it doesn't.

I tried some basic workarounds in the original config file, changed torch_dtype = float32 and changed it to float16, changed use_bfloat16 = false into true, both gave me errors either when changing both values together or changing one of them at a time.

object detection downloads sam-vit-huge model and its config files into "\models.download_cache" folder, I really don't think this method is right since the app doesn't detect anything inside that folder in manual search mode, downloaded models in this folder can't be copied into a new machine for example and use them with a new installation of the invokeai app.
also, it's very limiting regarding the choice of specific models, in this case object detection models.

I deleted sam-vit-huge, downloaded sam-vit-large and its config, and put it inside sam-vit-huge folder and object detection runs without an issue, except it's less precise than vit-huge obviously.


What you expected to happen

  • .download_cache models could be handeld the same way as .temp models; downloaded then sorted to proper folders.
    -fix the newly-introduced OOM error when using sam-vit-huge segmentation model.
    -maybe give some liberty to use different implementation of sam-vit, or better: segmentation models in general.

How to reproduce the problem

No response

Additional context

No response

Discord username

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions