Skip to content

Conversation

nickfujita
Copy link

@nickfujita nickfujita commented Aug 28, 2025

Description

  1. Large queries causing crashes - Messages split across TCP packets weren't properly reassembled
  2. ECONNRESET crashes - Abrupt client disconnects caused unhandled promise rejections
  3. Single connection limitation - Only one client could connect at a time

Root Causes

1. Message Fragmentation

PostgreSQL wire protocol messages can span multiple TCP packets, but the implementation treated each packet as a complete message. Large queries (>64KB) would be split, causing PGlite to crash on incomplete messages.

2. Unhandled Disconnections

When clients disconnected without proper protocol termination (common with connection pooling), the exclusive lock promise rejection wasn't caught, crashing the entire server process.

3. Connection-Level Locking

The original implementation held an exclusive PGlite lock for the entire connection lifetime, preventing multiple tools from connecting simultaneously.

Solutions

Message Buffering & Reassembly

  • Accumulates incoming data until complete PostgreSQL messages are received
  • Protocol-aware parsing identifies message boundaries:
    • Startup messages: [length: 4 bytes][protocol version][parameters]
    • Regular messages: [type: 1 byte][length: 4 bytes][payload]
  • Only complete messages are sent to PGlite

Robust Error Handling

  • Removed connection-level exclusive locking that could get stuck
  • Added proper cleanup for ECONNRESET errors (normal behavior with connection pooling)
  • Wrapped all async operations to prevent unhandled promise rejections
  • Server continues running even when clients disconnect abruptly

Multiple Concurrent Connections

  • Implemented query-level queuing instead of connection-level blocking
  • Multiple clients can maintain simultaneous connections
  • Queries from all connections are serialized through a shared queue
  • Connections persist after queries complete (supports both pooled and persistent connection patterns)
  • Proper cleanup when connections drop mid-query

Architecture Changes

Before: One connection at a time, connection-level locking

Client A → [Wait for lock] → [Hold lock for entire connection] → PGlite
Client B → [Blocked until A disconnects]

After: Multiple connections, query-level queuing

Client A ─┬→ [Handler A] ─┐
Client B ─┼→ [Handler B] ─┼→ [Query Queue] → PGlite
Client C ─┴→ [Handler C] ─┘   (queries processed sequentially)

Testing

  • ✅ Tested with bulk inserts of large documents (>100KB total)
  • ✅ Verified multiple simultaneous connections (Server Queries via Drizzle + Drizzle Studio)
  • ✅ Confirmed graceful handling of abrupt disconnections
  • ✅ Validated both connection patterns:
    • Pooled connections (connect → query → disconnect)
    • Persistent connections (connect → multiple queries → idle → more queries)

Configuration Options

New optional configuration parameters:

Parameter Type Default Description
maxConnections number 100 Maximum concurrent connections
idleTimeout number 0 Disconnect idle connections after N milliseconds (0 = disabled)

Example Usage

const server = new PGLiteSocketServer({
  db,
  port: 5432,
  host: '127.0.0.1',
  debug: true,
  maxConnections: 100,  // Allow up to 100 concurrent connections
  idleTimeout: 300000,  // Disconnect after 5 minutes of inactivity
});

Breaking Changes

None - existing configurations continue to work with improved behavior.

Technical Details

Message Buffer Implementation

The handler now maintains a message buffer that accumulates incoming TCP packets until a complete PostgreSQL message is available. This ensures PGlite only receives valid, complete protocol messages.

Query Queue Manager

A new QueryQueueManager class serializes access to PGlite at the query level rather than connection level, allowing multiple connections to coexist while maintaining PGlite's single-threaded execution model.

Connection Lifecycle

Each connection is managed by a PGLiteSocketHandler that:

  • Maintains its own message buffer
  • Submits queries to the shared queue
  • Cleans up pending queries on disconnect
  • Handles both persistent and pooled connection patterns

@nickfujita nickfujita changed the title [pglite-socket] Fix: Handle PostgreSQL messages split across multiple TCP packets [pglite-socket] Fix: Message buffering, connection handling, and concurrent connection support Aug 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant