Observing kotlin coroutines might lead to memory leak

[jensbaitingerbosch]

When an Observation.Scope is opened, the current scope is stored in the previousScope. For Kotlin Coroutines many coroutines are scheduled on the same thread. When they open a scope, all scopes in one thread (of which there is typically one per CPU) will get chained via the previous scopes.

While in Theory they might get eventually cleaned up, I could obsevre in our productive system a SimpleObservation object with 107 MB retained memory size (due to chained scopes) after a few hours uptime.

Environment

• Micrometer version 1.14.5
• Micrometer registry -
• OS: macOS 15.4 (but the issue should be independent)
• Java version: openjdk 23.0.2 2025-01-21

To Reproduce
Here is a demonstrator to demonstrate the chains. It schedules a coroutine in each iteration which then opens a scope, and calls delay(), then the next coroutine is scheduled.

val registry = ObservationRegistry.create()

suspend fun main() =
    coroutineScope {
        registry.observationConfig().observationHandler { false } // add a handler so that the registry is not noop

        for (i in 1..100_000) {
            launch {
                val observation = Observation.start("iteration $i", registry)
                observation.openScope().use {
                    printCurrentObservation()
                    delay(1.seconds)
                }
                observation.stop()
            }
        }
    }

fun printCurrentObservation() {
    val scope = registry.currentObservationScope
    println(
        scope?.currentObservation?.context?.name + " len: " + scope.previousScopesLength + " " +
            scope?.allPreviousScopes(),
    )
}

private val Observation.Scope.previousScopesLength: Int
    get() = 1 + (this.previousObservationScope?.previousScopesLength ?: 0)

private fun Observation.Scope.allPreviousScopes(): String =
    this.currentObservation.context.name + "->" + this.previousObservationScope?.allPreviousScopes()

Expected behavior
Every coroutine should be treated like a virtual thered, so when a fresh coroutine is sceduled it should start with a null scope, that should prevent the chaining.

Workaround
To prevent the issue, I closed the currentScope before opening a new one.

 observationRegistry.currentObservationScope?.close()
 observation.openScope().use {
      ...
}

[shakuzen]

Thank you for all the details. Unfortunately, I'm not very familiar with Kotlin coroutines or our instrumentation for it, and we don't seem to have its usage documented right now. However, see #3256 where we added it. It looks like your code is not using that. Could you take a look and see if you can use that and if it solves the problem? If you're able to get it working and would like to help more, a pull request to add documentation would be much appreciated. I don't think I'll have time to look into it this week, but I will try to take a look when I have time.