The first-time model compilation takes a long time

[max-my]

I encountered an issue where the first-time model compilation takes a long time, and there is a delay of several seconds even on different performance PCs.
Here is my compilation code:

std::shared_ptr<ov::Model> SegmentationModel::prepareModel(ov::Core& core)
{
    std::shared_ptr<ov::Model> model = core.read_model(modelFileName);
    prepareInputsOutputs(model);
    ov::set_batch(model, 1);
    return model;
}

ov::CompiledModel SegmentationModel::compileModel(std::string& deviceName, ov::Core& core)
{
    auto model = prepareModel(core);
    core.set_property({ov::enable_mmap(true)});
    compiledModel = core.compile_model(model, deviceName, {ov::cache_dir(cachePath)});
    return compiledModel;
}

The dynamic library I packaged includes: libopenvino.dll libopenvino_c.dll libopenvino_intel_cpu_plugin.dll libopenvino_intel_gpu_plugin.dll libopenvino_ir_frontend.dll
Could you please help me identify where the problem might be?

[ilya-lavrenov]

Hi @max-my

You are using ov::cache_dir(cachePath) which enables OpenVINO models caching feature, which means on first application run you will have to compile model from scratch and store results to cachePath on a disk, while the second and next application runs leverage a compiled blob from cachePath and application starts faster.

So, it's expected I would say. Could you please try to remove ov::cache_dir(cachePath) option and confirm that in all cases application runs slower?

More details about the feature is here https://docs.openvino.ai/2025/openvino-workflow/running-inference/optimize-inference/optimizing-latency/model-caching-overview.html

[max-my]

I tried removing ov::cache_dir(cachePath), and now every compilation is slow. Is this because I'm missing some acceleration libraries?

The dynamic library I packaged includes: libopenvino.dll libopenvino_c.dll libopenvino_intel_cpu_plugin.dll libopenvino_intel_gpu_plugin.dll libopenvino_ir_frontend.dll

Additionally, if I use ov::cache_dir(cachePath), the compilation becomes fast after the first run.

[ilya-lavrenov]

it's expected. Please, the information at the link I have provided above.