make a proper ApiGatewayToSqs event #3160

johnduffell · 2025-10-21T12:10:44Z

Having converted ticket tailor lambda to SrCDK in #3153

I noticed that there are a few issues

API gateway returns a hard coded 200 OK to ticket tailor even if it couldn't write to SQS for any reason (meaning no alarms and no retries)
the header needed is written to an attribute of the SQS message rather than into its body (sqs attributes seem to be designed for routing etc rather than actual data)
related to the above, the VTL is custom for that single ticket tailor header, making it not really reusable as an SRCDK thing. we do not have access to anything else from the lambda including pathParameters, headers, queryStringParameters, httpMethod and path.

This PR aims to fix these issues in the ticket tailor lambda, making the ApiGatewayToSqs pattern more robust and reusable. Main changes:

only return 200 if writing to SQS succeeded, otherwise 500
write json object containing all useful HTTP fields to SQS, rather than body only. Also add zod schema to parse it back again.

Bonus updates:

added a tickettailor README view in-branch here so that people know how to test it in CODE in future.
use the logger throughout, add input email as logger context

See inline comments.

Tested in CODE
https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fticket-tailor-webhook-CODE/log-events/2025$252F11$252F21$252F$255B$2524LATEST$255Dc927c51551ae4816af123a0b2be7072f

johnduffell · 2025-10-21T12:14:28Z

cdk/lib/__snapshots__/ticket-tailor-webhook.test.ts.snap

              "ResponseTemplates": {
                "application/json": "{ "status": "accepted" }",
              },
+              "SelectionPattern": "2\\d{2}",


this means it will only use 200 response if the SQS response is a 200.
Previously it was using 200 for everything, even if SQS rejected the request due to permissions or other issues

johnduffell · 2025-10-21T12:17:06Z

cdk/lib/__snapshots__/ticket-tailor-webhook.test.ts.snap

+            {
+              "ResponseTemplates": {
+                "application/json": "{ "message": "Internal Server Error - could not queue message" }",
+              },
+              "StatusCode": "500",
+            },


by default for any non-2xx response from SQS it will now return a 500. e.g. if there is an SQS outage or credentials issue.
Hopefully this will do two new things:

prompt ticket tailor to retry,

trigger the existing API gateway 5xx alarm

(The only tricky thing is investigating - I'm not sure where API gateway logs the integration responses - but this is an existing issue)

johnduffell · 2025-10-21T12:18:33Z

cdk/lib/__snapshots__/ticket-tailor-webhook.test.ts.snap

+            "application/json": "#set($json = "{")
+#set($json = $json + """pathParameters"":{")
+#set($allHeaders = $input.params().path)
+#foreach($header in $allHeaders.keySet())
+#set($json = $json + """$header"":""$util.escapeJavaScript($allHeaders.get($header)).replaceAll(""\\\\'"",""'"")""")
+#if($foreach.hasNext)
+#set($json = $json + ",")
+#end
+#end
+#set($json = $json + "},")
+#set($json = $json + """headers"":{")
+#set($allHeaders = $input.params().header)
+#foreach($header in $allHeaders.keySet())
+#set($json = $json + """$header"":""$util.escapeJavaScript($allHeaders.get($header)).replaceAll(""\\\\'"",""'"")""")
+#if($foreach.hasNext)
+#set($json = $json + ",")
+#end
+#end
+#set($json = $json + "},")
+#set($json = $json + """queryStringParameters"":{")
+#set($allHeaders = $input.params().querystring)
+#foreach($header in $allHeaders.keySet())
+#set($json = $json + """$header"":""$util.escapeJavaScript($allHeaders.get($header)).replaceAll(""\\\\'"",""'"")""")
+#if($foreach.hasNext)
+#set($json = $json + ",")
+#end
+#end
+#set($json = $json + "},")
+#set($json = $json + """body"":""$util.escapeJavaScript($input.body).replaceAll(""\\\\'"",""'"")"",")
+#set($json = $json + """httpMethod"":""$util.escapeJavaScript($context.httpMethod).replaceAll(""\\\\'"",""'"")"",")
+#set($json = $json + """path"":""$util.escapeJavaScript($context.path).replaceAll(""\\\\'"",""'"")"",")
+#set($json = $json + """mappingSource"": ""SrCDK""}")
+Action=SendMessage&MessageBody=$util.urlEncode($json)",


this is a bit hard to read, it would be easier to look at the CDK.
In essence it's building a JSON payload similar to an APIGatewayProxyEvent to put into the queue

johnduffell · 2025-10-21T12:20:11Z

cdk/lib/cdk/ApiGatewayToSqs.ts

+function iterateParam(targetName: string, sourceName: string) {
+	return `#set($json = $json + """${targetName}"":{")
+#set($allHeaders = $${sourceName})
+#foreach($header in $allHeaders.keySet())
+#set($json = $json + """$header"":""$util.escapeJavaScript($allHeaders.get($header)).replaceAll(""\\\\'"",""'"")""")
+#if($foreach.hasNext)
+#set($json = $json + ",")
+#end
+#end
+#set($json = $json + "},")`;
+}
+
+function insertSingleProp(targetName: string, sourceName: string) {
+	return `#set($json = $json + """${targetName}"":""$util.escapeJavaScript($${sourceName}).replaceAll(""\\\\'"",""'"")"",")`;
+}


these are utility methods to generate VTL that take values or objects from input variables and write them to the json safely
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
https://docs.aws.amazon.com/apigateway/latest/developerguide/rest-api-parameter-mapping-sources.html

johnduffell · 2025-10-21T12:22:16Z

handlers/ticket-tailor-webhook/src/index.ts

+const apiGatewayToSqsEventSchema = z.object({
+	pathParameters: z.record(z.string()),
+	headers: z.record(z.string()),
+	queryStringParameters: z.record(z.string()),
+	body: z.string(),
+	httpMethod: z.string(),
+	path: z.string(),
+});
+
+export type ApiGatewayToSqsEvent = z.infer<typeof apiGatewayToSqsEventSchema>;


This would be generic for all lambdas using ApiGatewayToSqs, but for now let's leave it here.

johnduffell · 2025-10-21T12:24:16Z

cdk/lib/cdk/ApiGatewayToSqs.ts

+						iterateParam(`pathParameters`, `input.params().path`),
+						iterateParam(`headers`, `input.params().header`),
+						iterateParam(`queryStringParameters`, `input.params().querystring`),
+						insertSingleProp(`body`, `input.body`),
+						insertSingleProp('httpMethod', 'context.httpMethod'),
+						insertSingleProp('path', 'context.path'),


as you can see I've added in a few bits we don't need for the existing lambda - we only need headers at the moment (in fact just one particular header)

The reason I added more is that this is a reusable construct and no one is going to want to fiddle about with VTL expressions just to put a queue in front of an existing lambda.

The idea of replicating the direct proxy lambda data structure (APIGatewayProxyEvent) is to make it as easy as possible to insert a queue in front of an existing lambda.

There would need to be a bit more tidying up of types before that's possible, but that can happen when it's needed.

we could slightly simplify things if we were using API gateway V2 SQS service integration [1], but I think we would still need to use VTL to build the body correctly, and it would be more invasive to make that much of a change. We should assess whether v2 is something we should move to for some/all of our APIs as it seems cheaper and more lightweight for common/simple use cases.
[1] https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-aws-services-reference.html#SQS-SendMessage

johnduffell · 2025-10-21T12:26:35Z

handlers/ticket-tailor-webhook/src/index.ts

+				await processValidSqsRecord(parsedEvent);
 			} else {
 				console.error('Request failed validation. Processing terminated.');
 				await putMetric('ticket-tailor-webhook-validation-failure');


I suspect there was supposed to be an alarm on this value and it was missed

johnduffell · 2025-10-21T13:36:39Z

cdk/lib/__snapshots__/ticket-tailor-webhook.test.ts.snap

+          {
+            "StatusCode": "500",
+          },


we have to tell API gateway that 500 is an allowed response code, (otherwise it returns its own 500 error in place of the one we hard coded)

johnduffell · 2025-10-21T13:39:06Z

handlers/ticket-tailor-webhook/src/apiGatewayToSqsEvent.ts

+export const apiGatewayToSqsEventSchema = z.object({
+	pathParameters: z.record(z.string()),
+	headers: z.record(z.string()),
+	queryStringParameters: z.record(z.string()),
+	body: z.string(),
+	httpMethod: z.string(),
+	path: z.string(),
+});
+export type ApiGatewayToSqsEvent = z.infer<typeof apiGatewayToSqsEventSchema>;


this is new the APIGatewayProxyEvent-like type. It can be moved to some kind of router module as and when it's needed in another lambda

johnduffell · 2025-10-21T13:41:03Z

handlers/ticket-tailor-webhook/src/index.ts

+			const parsedEvent: ApiGatewayToSqsEvent =
+				apiGatewayToSqsEventSchema.parse(JSON.parse(sqsRecord.body));


rather than treating the sqs body as the http body, we actually parse the sqs body to an object with its own nested body (and headers etc).

johnduffell · 2025-10-21T13:41:24Z

handlers/ticket-tailor-webhook/src/validateRequest.ts


-	if (!(typeof signatureWithTs === 'string')) {
-		console.error(
+	if (!signatureWithTs) {


it's always a string now, but it could be missing

johnduffell · 2025-10-21T13:42:32Z

handlers/ticket-tailor-webhook/test/testFixtures.ts


 export const emailAddress = `[email protected]`;

+const validBody = `{"id":"wh_1072228","created_at":"2024-08-20 13:20:26","event":"ORDER.CREATED","resource_url":"https:\\/\\/api.tickettailor.com\\/v1\\/orders\\/or_46629271","payload":{"object":"order","id":"or_46629271","buyer_details":{"address":{"address_1":null,"address_2":null,"address_3":null,"postal_code":null},"custom_questions":[],"email":"${emailAddress}","first_name":"joe&^","last_name":"griffiths%","name":"joe&^ griffiths%","phone":null},"created_at":1724160016,"credited_out_amount":0,"currency":{"base_multiplier":100,"code":"gbp"},"event_summary":{"id":"ev_4467889","end_date":{"date":"2024-12-13","formatted":"Fri 13 Dec 2024 10:30 PM","iso":"2024-12-13T22:30:00+00:00","time":"22:30","timezone":"+00:00","unix":1734129000},"event_id":"ev_4467889","event_series_id":"es_1354460","name":"CODE","start_date":{"date":"2024-08-28","formatted":"Wed 28 Aug 2024 6:00 PM","iso":"2024-08-28T18:00:00+01:00","time":"18:00","timezone":"+01:00","unix":1724864400},"venue":{"name":null,"postal_code":null}},"issued_tickets":[{"object":"issued_ticket","id":"it_72697654","add_on_id":null,"barcode":"R59xesv","barcode_url":"https:\\/\\/cdn.tickettailor.com\\/userfiles\\/cache\\/barcode\\/st\\/attendee\\/72697654\\/ef31abaf2ddf8d484483.jpg","checked_in":"false","created_at":1724160026,"custom_questions":[],"description":"General Admission","email":"[email protected]","event_id":"ev_4467889","event_series_id":"es_1354460","first_name":"joe&^","full_name":"joe&^ griffiths%","group_ticket_barcode":null,"last_name":"griffiths%","order_id":"or_46629271","qr_code_url":"https:\\/\\/cdn.tickettailor.com\\/userfiles\\/cache\\/barcode\\/qr\\/attendee\\/72697654\\/9ef2823a01c811da7614.png","reference":null,"reservation":null,"source":"checkout","status":"valid","ticket_type_id":"tt_4328701","updated_at":1724160026,"voided_at":null}],"line_items":[{"object":"line_item","id":"li_96505270","booking_fee":0,"description":"General Admission","item_id":"tt_4328701","quantity":1,"total":0,"type":"ticket","value":0}],"marketing_opt_in":null,"meta_data":[],"payment_method":{"external_id":null,"id":null,"instructions":null,"name":null,"type":"no_cost"},"referral_tag":null,"refund_amount":0,"refunded_voucher_id":null,"status":"completed","status_message":null,"subtotal":0,"tax":0,"tax_treatment":"exclusive","total":0,"total_paid":0,"txn_id":"--"}}`;


extracted from validSQSRecord.body below

johnduffell · 2025-10-21T13:43:16Z

handlers/ticket-tailor-webhook/test/testFixtures.ts

+		headers: {
+			'tickettailor-webhook-signature':
+				't=1724160026,v1=a3dbd8cfb0f04a0a9b0dd9d2547f1dd1a51e60d528a4edaee3bc02085517bd50',
+		},


tickettailor-webhook-signature is here instead of in messageAttributes below

johnduffell · 2025-10-21T13:43:53Z

handlers/ticket-tailor-webhook/test/testFixtures.ts

-			binaryListValues: [],
-			dataType: 'String',
-		},
+export const validSQSBody: ApiGatewayToSqsEvent = {


now these only need to be the simpler ApiGatewayToSqsEvent rather than the whole SQS record

Copilot

Pull Request Overview

This PR refactors the ticket-tailor-webhook lambda to improve the ApiGatewayToSqs pattern by moving webhook headers from SQS message attributes into the message body, implementing proper error handling with 500 responses, and replacing console.log with structured logging.

Modified the API Gateway VTL template to construct a JSON event structure mimicking APIGatewayProxyEvent
Updated the webhook handler to parse and validate the new event structure using Zod
Replaced all console.log/console.error/console.warn calls with structured logger calls

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
cdk/lib/cdk/ApiGatewayToSqs.ts	Refactored VTL template to build complete event JSON with headers, query parameters, and path parameters; added 500 error response handling
handlers/ticket-tailor-webhook/src/apiGatewayToSqsEvent.ts	Added new Zod schema and type definition for the ApiGatewayToSqs event structure
handlers/ticket-tailor-webhook/src/index.ts	Updated handler to parse and validate new event structure; replaced console.log with logger
handlers/ticket-tailor-webhook/src/validateRequest.ts	Changed to read signature from headers instead of message attributes; replaced console logging with logger
handlers/ticket-tailor-webhook/src/idapiService.ts	Replaced console.log/console.error with logger calls
handlers/ticket-tailor-webhook/test/testFixtures.ts	Refactored test fixtures to use new ApiGatewayToSqsEvent structure instead of SQS message attributes
handlers/ticket-tailor-webhook/test/validateRequest.test.ts	Updated test references to use new fixture names matching the event body structure
cdk/lib/snapshots/ticket-tailor-webhook.test.ts.snap	Updated snapshots to reflect new VTL template and error responses

handlers/ticket-tailor-webhook/src/validateRequest.ts

Co-authored-by: Copilot <[email protected]>

johnduffell · 2025-10-21T19:24:27Z

handlers/ticket-tailor-webhook/README.md

+## How to Test
+
+### Local Testing
+TODO create some runManual scripts to test locally and in CODE.


this is a TODO for another day, it's always nice when a lambda has

a quick test script you can run after deploying to CODE, and

a script to run the local code against CODE identity etc and log the output to test the integrations

johnduffell · 2025-10-21T19:26:23Z

handlers/ticket-tailor-webhook/src/idapiService.ts

 	);

-	console.log(`Fetching user type for provided email.`);
+	logger.log(`Fetching user type for provided email.`);


good opportunity to switch to use the logger module which adds some useful context

johnduffell · 2025-10-21T19:33:41Z

handlers/ticket-tailor-webhook/src/index.ts

+async function processValidSqsRecord(sqsRecord: ApiGatewayToSqsEvent) {
 	const ticketTailorRequest = JSON.parse(sqsRecord.body) as TicketTailorRequest;
 	const email = ticketTailorRequest.payload.buyer_details.email;
+	logger.mutableAddContext(email);


adding the email as context makes it easier to search the logs (e.g. find all log statements where this person/record tried before in the last 14 days)

johnduffell · 2025-10-21T19:36:28Z

handlers/ticket-tailor-webhook/src/index.ts

 		async (sqsRecord) => {
-			console.log(
+			logger.resetContext();


I'm not sure if because it's async it might run the next iteration while it's waiting for the previous one to complete and therefore mix up the contexts?

# Conflicts: # handlers/ticket-tailor-webhook/src/index.ts

make a proper ApiGatewayToSqs event

e343921

johnduffell commented Oct 21, 2025

View reviewed changes

fix tests etc

5dd713b

johnduffell commented Oct 21, 2025

View reviewed changes

johnduffell requested a review from Copilot October 21, 2025 13:44

johnduffell marked this pull request as ready for review October 21, 2025 13:44

Copilot AI reviewed Oct 21, 2025

View reviewed changes

handlers/ticket-tailor-webhook/src/validateRequest.ts Outdated Show resolved Hide resolved

handlers/ticket-tailor-webhook/src/validateRequest.ts Show resolved Hide resolved

handlers/ticket-tailor-webhook/src/validateRequest.ts Show resolved Hide resolved

johnduffell and others added 2 commits October 21, 2025 14:47

fix existing typo

420b680

Co-authored-by: Copilot <[email protected]>

add instructions for testing in CODE

f0868df

johnduffell commented Oct 21, 2025

View reviewed changes

use email as logger context

aa60e17

johnduffell commented Oct 21, 2025

View reviewed changes

Merge branch 'main' into jd/cdk/fix-tickettailor

90fb661

# Conflicts: # handlers/ticket-tailor-webhook/src/index.ts

shtukas approved these changes Nov 21, 2025

View reviewed changes

johnduffell merged commit 37f46ab into main Nov 21, 2025
56 checks passed

johnduffell deleted the jd/cdk/fix-tickettailor branch November 21, 2025 13:23

		const parsedEvent: ApiGatewayToSqsEvent =
		apiGatewayToSqsEventSchema.parse(JSON.parse(sqsRecord.body));


		export const emailAddress = `[email protected]`;

		const validBody = `{"id":"wh_1072228","created_at":"2024-08-20 13:20:26","event":"ORDER.CREATED","resource_url":"https:\\/\\/api.tickettailor.com\\/v1\\/orders\\/or_46629271","payload":{"object":"order","id":"or_46629271","buyer_details":{"address":{"address_1":null,"address_2":null,"address_3":null,"postal_code":null},"custom_questions":[],"email":"${emailAddress}","first_name":"joe&^","last_name":"griffiths%","name":"joe&^ griffiths%","phone":null},"created_at":1724160016,"credited_out_amount":0,"currency":{"base_multiplier":100,"code":"gbp"},"event_summary":{"id":"ev_4467889","end_date":{"date":"2024-12-13","formatted":"Fri 13 Dec 2024 10:30 PM","iso":"2024-12-13T22:30:00+00:00","time":"22:30","timezone":"+00:00","unix":1734129000},"event_id":"ev_4467889","event_series_id":"es_1354460","name":"CODE","start_date":{"date":"2024-08-28","formatted":"Wed 28 Aug 2024 6:00 PM","iso":"2024-08-28T18:00:00+01:00","time":"18:00","timezone":"+01:00","unix":1724864400},"venue":{"name":null,"postal_code":null}},"issued_tickets":[{"object":"issued_ticket","id":"it_72697654","add_on_id":null,"barcode":"R59xesv","barcode_url":"https:\\/\\/cdn.tickettailor.com\\/userfiles\\/cache\\/barcode\\/st\\/attendee\\/72697654\\/ef31abaf2ddf8d484483.jpg","checked_in":"false","created_at":1724160026,"custom_questions":[],"description":"General Admission","email":"[email protected]","event_id":"ev_4467889","event_series_id":"es_1354460","first_name":"joe&^","full_name":"joe&^ griffiths%","group_ticket_barcode":null,"last_name":"griffiths%","order_id":"or_46629271","qr_code_url":"https:\\/\\/cdn.tickettailor.com\\/userfiles\\/cache\\/barcode\\/qr\\/attendee\\/72697654\\/9ef2823a01c811da7614.png","reference":null,"reservation":null,"source":"checkout","status":"valid","ticket_type_id":"tt_4328701","updated_at":1724160026,"voided_at":null}],"line_items":[{"object":"line_item","id":"li_96505270","booking_fee":0,"description":"General Admission","item_id":"tt_4328701","quantity":1,"total":0,"type":"ticket","value":0}],"marketing_opt_in":null,"meta_data":[],"payment_method":{"external_id":null,"id":null,"instructions":null,"name":null,"type":"no_cost"},"referral_tag":null,"refund_amount":0,"refunded_voucher_id":null,"status":"completed","status_message":null,"subtotal":0,"tax":0,"tax_treatment":"exclusive","total":0,"total_paid":0,"txn_id":"--"}}`;

make a proper ApiGatewayToSqs event #3160

make a proper ApiGatewayToSqs event #3160

Uh oh!

Conversation

johnduffell commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnduffell Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnduffell Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnduffell Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnduffell Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnduffell Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnduffell Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

johnduffell commented Oct 21, 2025 •

edited

Loading

johnduffell Oct 21, 2025 •

edited

Loading

johnduffell Oct 21, 2025 •

edited

Loading

johnduffell Oct 21, 2025 •

edited

Loading

johnduffell Oct 21, 2025 •

edited

Loading

johnduffell Oct 21, 2025 •

edited

Loading

johnduffell Oct 21, 2025 •

edited

Loading