Skip to content

Releases: erans/pgsqlite

Release v0.0.18

28 Sep 00:40
Compare
Choose a tag to compare

🚀 Major Features

Comprehensive ORM Framework Compatibility

  • Complete Django integration with REST API test application featuring:
    • UUID primary keys and JSONB field support
    • PostgreSQL arrays and GENERATED BY DEFAULT AS IDENTITY columns
    • Complex model relationships and constraints validation
    • Poetry dependency management with comprehensive test suite
  • Rails ActiveRecord integration with full PostgreSQL compatibility:
    • PostgreSQL arrays (TEXT[]) and JSONB operations
    • Identity columns, foreign keys, and complex associations
    • Complete CRUD operations and business logic support
  • Go GORM application with advanced PostgreSQL features:
    • Full GORM models with UUID, arrays, and JSONB support
    • Auto-migrations, scopes, hooks, and associations
    • REST API with Gin framework and comprehensive test coverage

Security & Performance Enhancements

  • Advanced security features including:
    • Comprehensive SQL injection protection with AST-based detection
    • Security audit logging with structured JSON events
    • Rate limiting and DoS protection with circuit breaker patterns
    • Memory-aware caching system with TTL support
  • Protocol improvements:
    • Enhanced binary protocol support for better psycopg3 compatibility
    • Memory-mapped protocol handling for improved performance
    • Protocol fuzzing tests for robustness validation

🛠️ Bug Fixes

Critical CREATE TABLE Translation Fix

  • Fixed IDENTITY column translation bug that was breaking Django migrations
  • Properly handle "GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY" syntax
  • Translate correctly to SQLite "PRIMARY KEY AUTOINCREMENT"
  • Added comprehensive unit tests for IDENTITY translation
  • Fixed regex handling for quoted table names (e.g., "django_migrations")

Protobuf Compatibility

  • Moved ownership_token field from position 8 to position 15 to avoid breaking existing field
    numbering
  • Ensures backward compatibility while adding ownership verification functionality

⚙️ CI/CD Improvements

Build System Optimization

  • Aggressive disk space optimization for GitHub Actions to prevent build failures
  • Optimized parallelism with smart build job configuration
  • Separated build and test jobs for better resource management
  • Enhanced caching strategy with proper target directory sharing
  • Skip problematic benchmark tests that were causing CI instability

Test Infrastructure

  • Added protocol fuzzing tests for security validation
  • Enhanced rate limiter testing
  • Comprehensive security audit test suite
  • Improved SSL/TLS test coverage

📚 Documentation & Developer Experience

Enhanced Documentation

  • Streamlined CLAUDE.md by removing historical content
  • Added comprehensive security documentation
  • Enhanced ORM compatibility guides
  • Updated build and test instructions

Code Quality

  • Added comprehensive code review guidelines
  • Enhanced memory tracking and optimization utilities
  • Improved string utilities for better performance
  • Enhanced query executor with better error handling

🔧 Technical Improvements

Memory & Performance

  • Memory-aware cache manager with intelligent eviction policies
  • TTL-based caching system for improved query performance
  • String optimization utilities for reduced memory allocation
  • Enhanced connection management with better resource cleanup

Query Processing

  • Improved CREATE TABLE translator with better PostgreSQL syntax support
  • Enhanced binary protocol encoding for arrays and complex types
  • Better query executor with improved error handling and logging

Total Changes: 21 commits with significant additions across security, ORM compatibility, and
infrastructure improvements.

Compatibility: This release maintains full backward compatibility while significantly expanding
PostgreSQL feature support and ORM framework integration.

Release v0.0.17

22 Sep 16:17
Compare
Choose a tag to compare

Summary

This release delivers comprehensive PostgreSQL catalog and function support with significant performance optimizations, enhancing compatibility with major ORM frameworks including Django, SQLAlchemy, Rails, and Ecto.

New Features

  • PostgreSQL Session Functions: Complete session information access with current_user(), session_user(), current_database(), current_schema(), and current_schemas() functions for ORM connection management
  • PostgreSQL Permission Functions: Full authorization and access control with enhanced pg_has_role() and has_table_privilege() functions supporting role membership checking and table access control
  • information_schema.triggers: Complete trigger introspection with all 17 PostgreSQL-standard columns for business logic discovery
  • pg_tablespace Catalog: Enterprise tablespace management support with standard PostgreSQL tablespaces (pg_default, pg_global)
  • information_schema.check_constraints: Check constraint metadata support for ORM validation frameworks
  • information_schema.referential_constraints: Foreign key constraint introspection with all 9 PostgreSQL-standard columns
  • information_schema.views: Complete view metadata support with 10 PostgreSQL-standard columns
  • information_schema.routines: Function metadata for 40+ built-in functions across all categories
  • pg_stats Table: Query optimization and performance hints with realistic statistics generation
  • pg_roles and pg_user: Complete user and role management system for enterprise authentication
  • pg_description: Object comment and documentation system with COMMENT DDL integration
  • pg_proc Function Metadata: Comprehensive function introspection with 35+ built-in functions
  • Enhanced pg_attribute: Column metadata with defaults, constraints, identity/SERIAL detection
  • pg_index Support: Complete index management with multi-column index introspection
  • pg_constraint Support: Full constraint introspection for foreign key, primary key, unique, and check constraints
  • pg_depend Support: Rails sequence ownership detection for auto-increment columns

Improvements

  • Performance Optimizations: Session identifier translation with memchr-optimized pattern detection
  • Dual-processor Support: Integrated SessionIdentifierTranslator into both LazyQueryProcessor and UnifiedProcessor
  • Binary Protocol Enhancements: Complete array binary encoding for psycopg3 compatibility
  • Session-aware Architecture: Connection-per-session for proper transaction isolation across all catalog queries
  • ORM Compatibility: Full support for Django inspectdb, SQLAlchemy reflection, Rails introspection, and Ecto schema analysis

Bug Fixes

  • Fixed critical data type issues in referential_constraints (confrelid column read as INTEGER instead of STRING)
  • Fixed arithmetic test failures with proper floating-point division using CAST(... AS REAL)
  • Fixed information_schema binary protocol support with proper field descriptions
  • Fixed clippy warnings improving code quality and maintainability
  • Fixed cast translation issues preventing ARRAY[] syntax from working correctly

Breaking Changes

None - All changes are backward compatible with enhanced functionality

Release v0.0.16

04 Sep 19:50
Compare
Choose a tag to compare

Release Notes for v0.0.16

Summary

This release focuses on PostgreSQL compatibility improvements, fixing critical syntax errors and adding comprehensive support for database object comments and CREATE INDEX operator classes.

New Features

  • Complete PostgreSQL Comment System: Full implementation of COMMENT ON statements and comment functions

    • Added obj_description(oid, catalog_name) and col_description(table_oid, column_number) functions
    • Support for COMMENT ON TABLE, COMMENT ON COLUMN, and COMMENT ON FUNCTION statements
    • Schema migration v12 with __pgsqlite_comments table for persistent comment storage
    • Comprehensive PostgreSQL catalog compatibility for metadata queries
  • CREATE INDEX Operator Class Support: PostgreSQL-compatible index creation syntax

    • Support for varchar_pattern_ops, text_pattern_ops, bpchar_pattern_ops operator classes
    • Automatic translation to SQLite COLLATE BINARY for pattern-optimized indexes
    • Multiple operator classes in single index statements
    • Full Django/SQLAlchemy migration compatibility

Bug Fixes

  • Fixed "near varchar_pattern_ops: syntax error": CREATE INDEX statements with PostgreSQL operator classes now work correctly
  • Fixed "no such function: obj_description": Catalog queries using PostgreSQL comment functions now execute without errors
  • Improved test stability: Reduced concurrent reads test threshold for better CI reliability

Improvements

  • Enhanced PostgreSQL Compatibility: Better support for standard PostgreSQL catalog queries and metadata operations
  • Code Quality: Fixed multiple clippy warnings including manual_strip, if_same_then_else, and unnecessary_unwrap
  • Test Coverage: Added comprehensive integration and unit tests for new comment system and index operator class features

Breaking Changes

None. This release maintains full backward compatibility while adding new PostgreSQL compatibility features.

Release v0.0.15

12 Aug 06:55
Compare
Choose a tag to compare

Release Notes for v0.0.15 - 0.0.15-test

Summary

This release delivers complete PostgreSQL binary protocol support, enabling full compatibility with psycopg3's binary mode while fixing critical protocol issues and improving code quality.

Release v0.0.14

08 Aug 16:37
Compare
Choose a tag to compare

Release Notes for v0.0.14 - 0.0.14-test

Summary

This release introduces full PostgreSQL binary protocol support, delivering massive performance improvements with psycopg3-binary achieving ~139x overhead for SELECT queries (5x better than target). Major SQLAlchemy compatibility fixes ensure all tests pass for both psycopg2 and psycopg3 drivers, making pgsqlite production-ready for Python ORMs.

New Features

  • Binary Protocol Support: Full implementation of PostgreSQL binary format encoders for Numeric/Decimal, UUID, JSONJSONB, Money, and other types
  • psycopg3-binary Driver Support: Native binary encoding provides 19x faster SELECT performance compared to psycopg2
  • Enhanced Type Inference: Proper PostgreSQL type OIDs for aliased columns, aggregates, and empty result sets
  • Multi-Row INSERT RETURNING: Bulk insert operations now return correct row counts with proper rowid range queries

Improvements

  • Performance Breakthrough: SELECT queries now achieve ~139x overhead (0.139ms) with psycopg3-binary, exceeding all performance
    targets
  • SQLAlchemy Full Compatibility: All 8 ORM test scenarios pass including transactions, relationships, and cascade deletes
  • Aggregate Function Types: SUM/AVG/MAX/MIN now return proper NUMERIC types instead of TEXT for better driver compatibility
  • DateTime Handling: Binary timestamp parameters properly converted between PostgreSQL and Unix epochs
  • Connection Isolation: Schema lookups now use session connections for proper transaction visibility

Bug Fixes

  • Fixed json_object_agg returning JSON type instead of TEXT type
  • Fixed timestamp conversion in VALUES clauses and scalar subqueries
  • Fixed parameter cast queries bypassing ultra-fast path optimizations
  • Fixed column alias type inference for table.column AS alias patterns
  • Fixed date function translation creating malformed julianday syntax
  • Fixed empty result sets defaulting all columns to TEXT type
  • Fixed psycopg3 binary parameters returning 0 rows in queries
  • Fixed aggregate aliases like sum_1 returning TEXT instead of NUMERIC

Breaking Changes

None - This release maintains backward compatibility while adding new binary protocol capabilities.

Release v0.0.13

01 Aug 17:26
Compare
Choose a tag to compare

Release Notes for v0.0.13 - 0.0.13-test

Summary

This release focuses on performance optimizations through a new unified query processor architecture and fixes critical SQLAlchemy compatibility issues with aggregate functions. While the new architecture provides a cleaner foundation for future optimizations, significant performance regressions were discovered that require immediate attention.

New Features

  • Unified Query Processor: New streamlined query processing architecture that combines fast path detection and lazy query processing into a single, efficient system - Thread-Local Connection Cache: LRU cache for session connections to reduce mutex contention in high-concurrency scenarios
  • Fast Path Query Detection: Byte-level query detector to skip heavy processing for simple queries
  • RETURNING Clause Optimization: Optimized handling of simple RETURNING clauses in DML operations

Improvements

  • Code Reduction: 54% reduction in query processing code (870 → ~400 lines)
  • Zero-Allocation Fast Path: Simple queries now use Cow::Borrowed to avoid allocations
  • Progressive Complexity Detection: Checks cheap patterns first for better performance
  • Benchmark Suite: Added comprehensive Python benchmarking tools for performance analysis

Bug Fixes

  • SQLAlchemy Aggregate Types: Fixed "Unknown PG numeric type: 25" error when using MAX/MIN on DECIMAL columns
  • Aggregate Type Detection: Added support for SQLAlchemy's aliased aggregate columns (e.g., "max_1")
  • Build Warnings: Fixed all compilation warnings in simple_query_detector and unified_processor modules
  • Unit Tests: All 372 unit tests now pass without warnings

Breaking Changes

None

Performance Notes

⚠️ Critical Performance Regression Detected:

  • SELECT: 4.016ms (599x worse than 0.669ms target)
  • INSERT: 0.163ms (269x worse than 0.060ms target)
  • UPDATE: 0.053ms (90x worse than target)
  • DELETE: 0.033ms (100x worse than target)

The unified processor architecture provides a foundation for future optimizations including SIMD pattern ma

Release v0.0.12

29 Jul 17:25
Compare
Choose a tag to compare

Release Notes for v0.0.12 - 0.0.12-test

Summary

This release delivers critical SQLAlchemy ORM compatibility fixes and introduces PostgreSQL Full-Text Search support. Major architectural improvements include connection-per-session isolation and comprehensive transaction persistence fixes.

New Features

  • PostgreSQL Full-Text Search (FTS): Complete implementation using SQLite FTS5 backend
    • Support for tsvector/tsquery types and @@ match operator
    • Functions: to_tsvector(), to_tsquery(), plainto_tsquery(), phraseto_tsquery(), websearch_to_tsquery()
    • Zero performance impact on non-FTS queries

Improvements

  • Connection-Per-Session Architecture: Each PostgreSQL client gets its own SQLite connection

    • Proper transaction isolation matching PostgreSQL behavior
    • Fixes SQLAlchemy transaction persistence issues with WAL mode
    • Eliminates transaction visibility problems between sessions
  • SQLAlchemy ORM Compatibility: Full support for Python SQLAlchemy 2.0+

    • Fixed PostgreSQL type OID mapping for psycopg2 compatibility
    • Multi-row INSERT VALUES pattern translation for bulk operations
    • JOIN query type inference for proper relationship handling
    • DateTime column alias resolution and INTEGER storage conversion
    • RETURNING clause support for all DML operations

Bug Fixes

  • Fixed duplicate UUID generation in cached queries with gen_random_uuid()
  • Fixed NOW() function returning epoch time instead of current timestamp
  • Fixed AT TIME ZONE operator encoding issues with simple_query protocol
  • Fixed NUMERIC binary encoding causing Unicode decode errors
  • Fixed information_schema.tables returning incorrect table names
  • Fixed DELETE ... USING syntax translation for batch operations
  • Resolved "Migration lock held by process" errors in concurrent tests

Breaking Changes

None - This release maintains backward compatibility while adding new features.

Release v0.0.10

22 Jul 08:12
Compare
Choose a tag to compare
Release v0.0.10 Pre-release
Pre-release

Release Notes for v0.0.10 - 0.0.10-test

Summary

This release brings significant performance improvements through connection pooling, enhanced PostgreSQL compatibility with comprehensive function support, and critical bug fixes for prepared statements. The query optimization system has been completely overhauled, delivering up to 21% performance improvements for SELECT queries.

New Features

  • Connection Pooling with Read/Write Separation: Production-ready connection pooling infrastructure with automatic query routing, health checks, and transaction affinity
  • PostgreSQL Functions: Implemented 35+ PostgreSQL-compatible functions including string functions (split_part, string_agg, translate), math functions (trunc with precision, trigonometric functions), and more
  • Enhanced psql Support: Full \d tablename command support with populated pg_constraint, pg_attrdef, and pg_index catalog tables
  • Comprehensive Query Optimization: New optimization infrastructure featuring read-only optimizer, enhanced statement caching, and intelligent query plan caching

Improvements

  • SELECT Performance: 21% improvement (from ~369x to ~291x overhead) through query optimization
  • Cache Effectiveness: Maintained 1.6x speedup for cached queries
  • Connection Pool Performance: Baseline 95,961 QPS single-thread, 124,380 QPS with 8 concurrent tasks
  • Statement Caching: Enhanced caching system now handles 200+ cached query plans with priority-based eviction
  • Query Router: Intelligent classification and routing of SELECT vs DML queries for optimal resource utilization

Bug Fixes

  • BIT Type Cast Issues: Fixed prepared statements returning empty strings for BIT type casts
  • SQL Parser Enhancement: Resolved parser errors with parameterized BIT types (::bit(8), ::varbit(10))
  • DateTime Roundtrip: Fixed binary encoding issues for TIME values and proper microsecond precision
  • Performance Regression: Eliminated 34% performance regression through fast-path optimizations

Breaking Changes

None - all changes maintain backward compatibility

Release v0.0.9

16 Jul 20:34
Compare
Choose a tag to compare
Release v0.0.9 Pre-release
Pre-release

Release Notes for v0.0.9 - 0.0.9-test

Summary

This release completes the array and JSON feature sets for pgsqlite, adding full support for PostgreSQL-compatible array literals, operators, and functions, along with comprehensive JSON manipulation capabilities. The release also includes significant codebase improvements and test infrastructure enhancements.

New Features

  • ARRAY Literal Syntax: Support for ARRAY[1,2,3] syntax with automatic translation to JSON format
  • ALL Operator: Fixed and enhanced ALL operator with proper balanced parentheses parsing
  • unnest() WITH ORDINALITY: PostgreSQL-compatible implementation with 1-based indexing
  • JSON Row Conversion: Complete row_to_json() function for converting table rows to JSON objects
  • JSON Manipulation Functions:
    • jsonb_insert() - Insert values into JSON objects/arrays
    • jsonb_delete() - Delete values from JSON by path
    • jsonb_pretty() - Pretty-print JSON with formatting
  • JSON Table Functions: json_each_text() and jsonb_each_text() for text-based key-value expansion
  • JSON Aggregation: json_populate_record() and json_to_record() for record conversions

Improvements

  • Test Infrastructure: Reorganized SQL integration tests into categorized subdirectories for better maintainability
  • Array Performance: Zero performance impact from array enhancements - maintained ~280x SELECT overhead
  • Code Organization: Cleaned up root directory by removing temporary files
  • Test Coverage: Comprehensive CI/CD validation suite for all JSON and array functions

Bug Fixes

  • Fixed simple query detector to ensure array queries use the translation pipeline
  • Resolved array concatenation operator (||) detection for ARRAY[] syntax patterns
  • Fixed JSON path operator conflicts with SQL parser $ character handling
  • Corrected wire protocol conversion for JSON arrays to PostgreSQL format

Breaking Changes

None

Release v0.0.7

13 Jul 16:54
Compare
Choose a tag to compare
Release v0.0.7 Pre-release
Pre-release

Release Notes for v0.0.7 - 0.0.7-test

Summary

This release enhances array type support in pgsqlite, providing better compatibility with PostgreSQL array operations and fixing critical issues with array value handling.

New Features

  • Enhanced Array Support: Improved PostgreSQL array type compatibility with better handling of array literals and operations
  • Array Wire Protocol: Fixed array value conversion between JSON storage format and PostgreSQL wire protocol format

Improvements

  • Better array literal parsing for both ARRAY[...] and {...} syntax formats
  • Enhanced array type validation and constraint checking
  • Improved performance for array operations through optimized conversion paths

Bug Fixes

  • Fixed wire protocol conversion for JSON arrays to properly return PostgreSQL-formatted array strings
  • Resolved array value handling in multi-row INSERT statements
  • Corrected array type registration in the pg_type system catalog

Breaking Changes

None