Show async stack frames in the recovered exception stack trace

[mvicsokolova]

Hi!
Currently, for coroutine builders like async, which rely on the user to handle an exception (via await), we do not attach the async stack trace to the recovered exception.

For an example like this:

private suspend fun f2() {
    yield()
    if (true) throw RecoverableTestException("aaaa")
    yield()
}

private suspend fun f1() {
    yield()
    f2()
    yield()
}

private suspend fun f() {
    yield()
    f1()
    yield()
}

fun main() = runBlocking {
    val deferred = async {
        f()
    }
    deferred.await()
}

We get the following recovered exception stack trace:

kotlinx.coroutines.testing.RecoverableTestException: aaaa
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest.f2(StackTraceRecoveryTest.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest.access$f2(StackTraceRecoveryTest.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest$f2$1.invokeSuspend(StackTraceRecoveryTest.kt)
	at _COROUTINE._BOUNDARY._(CoroutineDebugging.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest$test$1.invokeSuspend(StackTraceRecoveryTest.kt)
Caused by: kotlinx.coroutines.testing.RecoverableTestException: aaaa
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest.f2(StackTraceRecoveryTest.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest.access$f2(StackTraceRecoveryTest.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest$f2$1.invokeSuspend(StackTraceRecoveryTest.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt)
	....

And it would be great to attach the async frames as well, so that f1 and f frames would also be in the stack trace, like:

kotlinx.coroutines.testing.RecoverableTestException: aaaa
  at kotlinx.coroutines.exceptions.StackTraceRecoveryTest.f2(StackTraceRecoveryTest.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest.access$f2(StackTraceRecoveryTest.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest$f2$1.invokeSuspend(StackTraceRecoveryTest.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest.f1(StackTraceRecoveryTest.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest.f(StackTraceRecoveryTest.kt)
	at _COROUTINE._BOUNDARY._(CoroutineDebugging.kt)
	at kotlinx.coroutines.exceptions.StackTraceRecoveryTest$test$1.invokeSuspend(StackTraceRecoveryTest.kt)
Caused by: kotlinx.coroutines.testing.RecoverableTestException: aaaa
.....

[mvicsokolova]

This task seems to be similar to the support of async stack traces for Flow, which was implemented for IDEA in the coroutines fork (JetBrains#5).

It proved to be very useful and now I'm thinking on how to add this additional debug info in a maybe more concise manner, so that we could include it in the main library.

So, if you have any ideas or there were any alternative solutions already proposed, it would be great to discuss 👀

[dkhalanskyjb]

There are several, independent feature requests here, I'll try to separate them:

1. Deferred.await should attach its own stacktrace to the thrown exception if the stacktrace recovery is enabled.
2. Concurrent Flow should pretend that the whole flow processing is done in one call stack instead of the values being transferred between multiple call stacks.
3. Nested suspend function calls should be reflected in the call stack after suspensions.

Here's a reply to each one:

1. As stated, I consider this impossible: I remember seeing cases where the identity of the exception thrown by Deferred.await was important (as exceptions were registered as needing cleanup and then queried by their identity). This is not simply a case of not breaking user code, though: we have this behavior documented in https://kotlinlang.org/api/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-deferred/await.html: Unless the calling coroutine is cancelled, await will return the same result on each invocation: [...] if the Deferred completed exceptionally, await will rethrow the same exception. We can reconsider this when designing a replacement for await, which we may need to do anyway to avoid the most common case of CancellationException can be thrown not only to indicate cancellation #3658 So, this proposal is on the table, just not for Deferred.await but for its replacement.
2. Yes, this is a reasonable request. Semantically, a Flow presents itself as a linear, nested emission of values. flowOn should behave like withContext in this regard, for example, which means linking stacks together.
3. This is somewhat out of hands of kotlinx.coroutines, since this example does not even touch kotlinx.coroutines entities except at the level of coroutine builders. Yes, stack frames do get lost from exceptions, but this happens in Kotlin itself/its standard library, not in kotlinx.coroutines. This can be fixed, as proven by this project: https://github.com/Anamorphosee/stacktrace-decoroutinator This instruments every continuation to leave its stack frames in thrown exceptions. Whether we want such a mechanism in kotlinx.coroutines as an enhancement of stacktrace recovery is an open question that depends on how significant the runtime impact of the Decoroutinator is. We can research this solution in more detail and maybe recommend it officially (possibly after sending some pull requests if some improvements come to mind), or adopt this code into our library (preferably as the original author's contribution) to make onboarding of new users more straightforward.

I think it's worth focusing on each one separately, so it may be better to split this issue into several ones.