Add error handling for lambdas

[andybee]

If you use a plugin to create a custom Lambda, and then invoke it and it throws an error, Sandbox is halted.

Ideally, Sandbox would catch and report the error to the console, rather than abort, to allow an easier development workflow.

[filmaj]

Trying to reproduce this.. how did you set up the plugin to invoke the custom lambda?

[andybee]

Sorry for the half-arsed report. I've created a demo repo: https://github.com/andybee/demo-custom-lambda-error

npm install then npx arc sandbox and press i (I've included the @architect/plugin-lambda-invoker so you can run the custom Lambda).

When it throws, Sandbox goes with it, unlike how other function types are handled.

[filmaj]

Awesome, thank you for the demo repo 🙏 Confirmed

[filmaj]

So I added an @events to your demo repo and did the same thing: threw an error in it, invoked. And it also crashed sandbox.. so perhaps this isn't isolated to only custom lambdas? Will test this out with other first class arc pragmas.

Maybe this is a lambda invoker problem?

[filmaj]

OK I forked your demo and expanded it a bit: https://github.com/filmaj/demo-custom-lambda-error

• I added an @events test that throws an error
• I added an @http route /test that publishes to the test event

When I load up sandbox and visit the /test route over HTTP, I can see the event is triggered and an exception is thrown, but it doesn't crash sandbox. Using the i plugin invoker, though, triggering the event does crash sandbox.

How would you typically use your custom lambda within the context of an actual application? I'm trying to suss out what sorts of scenarios these event-source-less lambdas serve and how one would go about testing them locally without the plugin invoker. Is it the case that you cannot easily test custom lambdas locally without the plugin invoker? If so, then this further confirms my suspicion that this is an issue with the plugin invoker (and I will take a look and see if I can follow the stack trace to confirm my suspicion).

For the record, here are stack traces for using plugin-invoker triggering the custom lambda:

? Which event do you want to invoke? · @customLambdas example
⚬ Invoker Invoking @customLambdas example with empty payload

Error: example exception thrown
    at handler (file:///Users/filmaj/src/demo-custom-lambda-error/src/functions/example/index.mjs:2:9)
    at run (file:///Users/filmaj/src/demo-custom-lambda-error/src/functions/example/[eval1]:1:3167)
    at async main (file:///Users/filmaj/src/demo-custom-lambda-error/src/functions/example/[eval1]:1:3382)
⚬ Sandbox Plugin invoked @customLambdas example
node:internal/process/promises:394
    triggerUncaughtException(err, true /* fromPromise */);
    ^

Error: example exception thrown
    at handler (file:///Users/filmaj/src/demo-custom-lambda-error/src/functions/example/index.mjs:2:9)
    at run (file:///Users/filmaj/src/demo-custom-lambda-error/src/functions/example/[eval1]:1:3167)
    at async main (file:///Users/filmaj/src/demo-custom-lambda-error/src/functions/example/[eval1]:1:3382)

Node.js v22.14.0

.. and here it is triggering the event:

⚬ Invoker Invoking @events test with empty payload

{
  "Records": [
    {
      "Sns": {
        "Message": "{}"
      }
    }
  ]
}

Error: boom
    at handler (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/index.mjs:4:9)
    at run (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/[eval1]:1:3167)
    at async main (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/[eval1]:1:3382)
⚬ Sandbox Plugin invoked @events test
node:internal/process/promises:394
    triggerUncaughtException(err, true /* fromPromise */);
    ^

Error: boom
    at handler (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/index.mjs:4:9)
    at run (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/[eval1]:1:3167)
    at async main (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/[eval1]:1:3382)

Node.js v22.14.0

[filmaj]

I ran arc sandbox -d in debug mode to get further info on how the code flows through Sandbox in both cases of invoking an @event with the plugin invoker and without (via an HTTP route w/ arc-functions). The traces are identical through sandbox's internal invoke-lambda code. The end of the trace looks like this in the non-crashing, HTTP-triggered case:

⚬ Sandbox [696f8e62-9dd5-4886-bbc2-ad8b222876e4] Invocation successfully completed
⚬ Sandbox @events test completed

.. but it looks like this in the crashing, plugin-invoker case:

⚬ Sandbox [ec28e6d2-d01d-4c32-8e00-17c001a1e9e6] Invocation successfully completed
⚬ Sandbox Plugin invoked @events test
node:internal/process/promises:394
    triggerUncaughtException(err, true /* fromPromise */);
    ^

Error: boom
    at handler (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/index.mjs:4:9)
    at run (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/[eval1]:1:3167)
    at async main (file:///Users/filmaj/src/demo-custom-lambda-error/src/events/test/[eval1]:1:3382)

Node.js v22.14.0

I think I've found the issue.. will transfer this issue over to the invoker plugin repo.

[filmaj]

The issue ends up being relatively simple:

• sandbox exposes an invoke utility for invoking plugins. This method return a Promise, and rejects in many situations, raised runtime errors of plugin lambda being one of them.
• This plugin uses that invoke utility, but doesn't catch exceptions raised by it. Thus, if that utility raises an exception for any reason, this plugin falls apart and takes sandbox down with it.

This is why invoking any lambda with this plugin - first-class arc pragma or custom - yields the same experience with runtime errors: sandbox crash.

The fix should be pretty easy: wrap the invocation in a try/catch and report the error. I've tested a hacky fix locally with a patch like this:

diff --git a/src/index.js b/src/index.js
index c798312..4f91d2e 100644
--- a/src/index.js
+++ b/src/index.js
@@ -77,8 +77,7 @@ let sandbox = {
           mocks = await getMod(jsMocks)
         }
         else if (existsSync(cjsMocks)) {
-          // eslint-disable-next-line
-            mocks = require(cjsMocks)
+          mocks = require(cjsMocks)
         }
         else if (existsSync(mjsMocks)) {
           mocks = await getMod(mjsMocks)
@@ -155,7 +154,12 @@ let sandbox = {
         let msg = `Invoking @${pragma} ${name}`
         msg += ` with ${mockName === 'empty' ? 'empty' : `'${mockName}'`} payload`
         update.status(msg)
-        await invoke({ pragma, name, payload })
+        try {
+          await invoke({ pragma, name, payload })
+        }
+        catch (e) {
+          update.warn(e)
+        }
         start()
       }
     })

Gonna work on a PR for this, but it may take me a bit as I think this repo needs some tests, too.

[andybee]

Thanks for digging in to this, bit embarrassed I didn't twig this being a problem with the invoker rather than Sandbox itself. Fix looks good, but happy to wait for your further findings/tests/PR.

I have spotted a couple of fun edge cases with the invoker, so I'll use this as a nudge to investigate and propose some PRs for it.