test: reproduce commonly cited timeout error #1390

danieljbruce · 2025-05-06T19:38:57Z

Description

The main intent of this PR is to supply a test confirming that users can see the changes introduced by this PR. Primarily, the runQuery test provides an error stack that users can see so that they can identify the root cause of the timeout errors they are seeing. Confirming this behaviour is in place with a test is crucial for solving issues like #1176 going forward.

Impact

This is a crucial step towards solving #1176 giving us the tools we need to explore client library behaviour when the error from the issue occurs. Having this test in place ensures the impact from the fix in Gax is locked in which will assist users with determining the root cause of errors they are seeing in the future.

Testing

This PR only adds tests and no source code changes.

mock-server/datastore-server.ts: Changes are made to the mock server so that test code can provide its own service configuration providing custom responses for different server grpc endpoints.

test/mock-server-tester.ts: This file is meant to be a grpc endpoint agnostic utility file for providing standard grpc services to the mock server. For example, the sendErrorSeries method can be used to make the mock server send a series of errors with the same code and the shutdownServer method shuts down the mock server.

test/with-runquery-mockserver.ts: This file runs a test where runQuery calls return a deterministic set of UNAVAILABLE errors and then it checks the error message to see that it reports the timeout error along with the list of UNAVAILABLE errors in order.

Additional Information

Some related PRs:
googleapis/gax-nodejs#1740
googleapis/gax-nodejs#1650

Next Steps

We have a test for UNAVAILABLE on runQuery now, but we should apply the right fixes for DEADLINE_EXCEEDED errors
Let's add a test involving transactions

- Added tests to ensure error is passed back to user - Changed source code to use passthrough for runQuery - changed the mock server to allow overriding of the default service

use

of what they do

danieljbruce · 2025-05-06T19:59:01Z

test/try-server.ts

+        done();
+      } catch (e) {
+        done(e);
+      }


The only real change is to make the test runner throw an error if the script throws an error instead of failing silently

danieljbruce · 2025-05-06T20:02:23Z

test/with-runquery-mockserver.ts

+            // The error message is based on client library behavior.
+            assert.strictEqual(
+              (e as Error).message,
+              '4 DEADLINE_EXCEEDED: error details: error count: 1',


The error message should be something like this. We need to do a bit more investigation into what the client should do in this case to determine the exact error message should be, but for now it's fine to backlog this as a TODO.

danieljbruce · 2025-05-06T20:04:23Z

test/mock-server-tester.ts

@@ -0,0 +1,126 @@
+// Copyright 2025 Google LLC


This file contains utilities for working with the mock server that include a function for producing a deterministic set of errors and a function for shutting down the mock server.

danieljbruce · 2025-05-06T20:05:05Z

mock-server/datastore-server.ts

-  callback: (arg1: string | null, arg2: {}) => {},
-) {
-  // SET A BREAKPOINT HERE AND EXPLORE `call` TO SEE THE REQUEST.
-  callback(null, {message: 'Hello'});


This grpcEndpoint function has just been moved inside the only place it is used.

iam-ethan-huang · 2025-05-06T22:30:42Z

test/mock-server-tester.ts

+   * `DEADLINE_EXCEEDED` code (4), a details message indicating the number of
+   * errors generated so far by this instance, and some metadata.
+   *
+   * @returns {ServiceError} A `ServiceError` object representing a simulated
+   *   gRPC error.
+   */
+  generateError() {


nit - without reading the comment, it's actually hard interpret generateError to generate a DEADLINE_EXCEEDED error.

I restructured this so that we pass the error code into generateError. This might make the intent of this method more clear.

iam-ethan-huang · 2025-05-06T22:35:19Z

test/with-createreadstream-mockserver.ts

@@ -0,0 +1,51 @@
+// Copyright 2025 Google LLC


nit - the test file is also a bit confusing. I think your intention is to differentiate the test between non-retryable error like NOT_FOUND v.s. a DEADLINE_EXCEEDED error.

But the createreadstream v.s. runquery in the name made it confusing as it seems to do with the functionalities.

The intent is to differentiate between the calls to the lookup grpc endpoint and the runQuery grpc endpoint.

Note that this PR is in a Draft stage. Although the early feedback is appreciated, I think I'm going to make some larger adjustments like removing this file so that we can lock in the progress we have with the UNAVAILABLE test on runQuery. This gives us the confidence that we can see the error stack for these types of calls.

…into 414574369-test-error-stack-end-to-end # Conflicts: # src/request.ts

…into 414574369-test-error-stack-end-to-end

…b.com/googleapis/nodejs-datastore into 414574369-test-error-stack-end-to-end

This reverts commit 69dbb30.

danieljbruce · 2025-05-22T20:48:36Z

mock-server/datastore-server.ts

-export function startServer(cb: () => void) {
+export function startServer<ResponseType>(
+  cb: () => void,
+  serviceConfigurationOverride?: MockServiceConfiguration<ResponseType>,


This is where a new service can be provided to the mock server.

daniel-sanche · 2025-05-23T19:10:11Z

test/mock-server-tester.ts

+    const metadata = new grpc.Metadata();
+    metadata.set(
+      'grpc-server-stats-bin',
+      Buffer.from([0, 0, 116, 73, 159, 3, 0, 0, 0, 0]),


what is this buffer?

daniel-sanche · 2025-05-23T19:13:47Z

test/mock-server-tester.ts

+   * This method is designed to be used in mock server setups for testing purposes.
+   * It returns a function that, when called, will invoke a callback with a
+   * pre-configured error object, simulating a gRPC service responding with an error.
+   * The error includes a DEADLINE_EXCEEDED code (4), a details message indicating the


why DEADLINE_EXCEEDED? Doesn't it take an arbitrary error code?

leahecole

Is there a doc about the mock server tester or the mock server itself? Not sure if this is something your team typically does for your repos, but it seems like a lot of maintenance overhead to introduce. I am seeing yellow flags at reintroducing something somewhat generic that does a series of errors - isn't that what we should be testing in gax? (Which I do know you're working on, I just wonder if this is redundant?)

leahecole · 2025-05-23T19:38:18Z

mock-server/datastore-server.ts

+// eslint-disable-next-line n/no-extraneous-import
+import {ServiceError} from 'google-gax';
+import {Server} from '@grpc/grpc-js';


These should probably be added to a package.json instead of disabling extraneous imports

leahecole · 2025-05-23T19:39:40Z

test/mock-server-tester.ts

+    const metadata = new grpc.Metadata();
+    metadata.set(
+      'grpc-server-stats-bin',
+      Buffer.from([0, 0, 116, 73, 159, 3, 0, 0, 0, 0]),


danieljbruce added 17 commits May 6, 2025 10:49

send error back in the mock server

a6d82ef

Mock server error tests with fixes

b09bab3

- Added tests to ensure error is passed back to user - Changed source code to use passthrough for runQuery - changed the mock server to allow overriding of the default service

Change the source code back - no problems

68c888b

Add some console logs

e9c0add

Add mechanism for shutting down the server

57e5064

Move the shutdown server function

cbf59bf

Rename the service method to sendNonRetryableError

eb61f46

Delete shutdown comment

19fbec5

Only run the run query test

4c230a1

Eliminate unused import

30a7838

Complete reproduction of the timeout error

ae6f01c

Rename the file to more accurately describe its

e36bd3d

use

Change the error message

60303d6

Add a class for containing the error series

0005a22

More clear intent in test

a9bf0df

skip the test

941c6eb

Update tests with more accurate descriptions

2b4691d

of what they do

product-auto-label bot added size: m Pull request size is medium. api: datastore Issues related to the googleapis/nodejs-datastore API. labels May 6, 2025

Cleanup the types everywhere

1eacc70

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels May 6, 2025

danieljbruce commented May 6, 2025

View reviewed changes

danieljbruce changed the title ~~414574369 test error stack end to end~~ test: reproduce commonly cited timeout error May 6, 2025

iam-ethan-huang reviewed May 6, 2025

View reviewed changes

Modify the test to use UNAVAILABLE

8683a77

danieljbruce and others added 18 commits May 8, 2025 10:34

Merge branch 'main' of https://github.com/googleapis/nodejs-datastore …

d726c44

…into 414574369-test-error-stack-end-to-end # Conflicts: # src/request.ts

Add a couple of console logs

5bdd6bc

Reduce timeout

e7c0f43

Refactor regex pattern

c8312af

Factor replace value

5b1bcaf

Merge branch 'main' into 414574369-test-error-stack-end-to-end

f144fd5

Provide a projectId

d5b3e63

Merge branch 'main' of https://github.com/googleapis/nodejs-datastore …

6899406

…into 414574369-test-error-stack-end-to-end

Merge branch '414574369-test-error-stack-end-to-end' of https://githu…

ee1c0cd

…b.com/googleapis/nodejs-datastore into 414574369-test-error-stack-end-to-end

Revert import

7609fc0

move grpc service definition back to where it was

676d561

Thread ResponseType parameter through server confg

9c2f9a7

Remove SuccessType

015e5a6

delete file - not used

69dbb30

Revert "delete file - not used"

a5f9730

This reverts commit 69dbb30.

linting fix

5dbbefc

Replace any in call type

efa7631

Eliminate sendNonRetryableError, it is not used

e1e6599

product-auto-label bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels May 22, 2025

danieljbruce added 2 commits May 22, 2025 15:41

Revert try-server changes

8ffcea1

remove console errors and console logs

e36357d

danieljbruce commented May 22, 2025

View reviewed changes

Eliminate some comments

a36a7de

danieljbruce marked this pull request as ready for review May 22, 2025 20:58

danieljbruce requested review from a team as code owners May 22, 2025 20:58

danieljbruce requested a review from leahecole May 22, 2025 20:59

daniel-sanche reviewed May 23, 2025

View reviewed changes

leahecole reviewed May 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test: reproduce commonly cited timeout error #1390

test: reproduce commonly cited timeout error #1390

Uh oh!

danieljbruce commented May 6, 2025 •

edited

Loading

Uh oh!

danieljbruce May 6, 2025

Uh oh!

danieljbruce May 6, 2025

Uh oh!

danieljbruce May 6, 2025

Uh oh!

danieljbruce May 6, 2025

Uh oh!

iam-ethan-huang May 6, 2025

Uh oh!

danieljbruce May 7, 2025

Uh oh!

iam-ethan-huang May 6, 2025

Uh oh!

danieljbruce May 7, 2025 •

edited

Loading

Uh oh!

danieljbruce May 22, 2025

Uh oh!

daniel-sanche May 23, 2025

Uh oh!

leahecole May 23, 2025

Uh oh!

daniel-sanche May 23, 2025

Uh oh!

leahecole left a comment

Uh oh!

leahecole May 23, 2025

Uh oh!

leahecole May 23, 2025

Uh oh!

Uh oh!

test: reproduce commonly cited timeout error #1390

Are you sure you want to change the base?

test: reproduce commonly cited timeout error #1390

Uh oh!

Conversation

danieljbruce commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Impact

Testing

Additional Information

Next Steps

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieljbruce May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leahecole left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danieljbruce commented May 6, 2025 •

edited

Loading

danieljbruce May 7, 2025 •

edited

Loading