Releases: erans/pgsqlite
Release v0.0.18
🚀 Major Features
Comprehensive ORM Framework Compatibility
- Complete Django integration with REST API test application featuring:
- UUID primary keys and JSONB field support
- PostgreSQL arrays and GENERATED BY DEFAULT AS IDENTITY columns
- Complex model relationships and constraints validation
- Poetry dependency management with comprehensive test suite
- Rails ActiveRecord integration with full PostgreSQL compatibility:
- PostgreSQL arrays (TEXT[]) and JSONB operations
- Identity columns, foreign keys, and complex associations
- Complete CRUD operations and business logic support
- Go GORM application with advanced PostgreSQL features:
- Full GORM models with UUID, arrays, and JSONB support
- Auto-migrations, scopes, hooks, and associations
- REST API with Gin framework and comprehensive test coverage
Security & Performance Enhancements
- Advanced security features including:
- Comprehensive SQL injection protection with AST-based detection
- Security audit logging with structured JSON events
- Rate limiting and DoS protection with circuit breaker patterns
- Memory-aware caching system with TTL support
- Protocol improvements:
- Enhanced binary protocol support for better psycopg3 compatibility
- Memory-mapped protocol handling for improved performance
- Protocol fuzzing tests for robustness validation
🛠️ Bug Fixes
Critical CREATE TABLE Translation Fix
- Fixed IDENTITY column translation bug that was breaking Django migrations
- Properly handle "GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY" syntax
- Translate correctly to SQLite "PRIMARY KEY AUTOINCREMENT"
- Added comprehensive unit tests for IDENTITY translation
- Fixed regex handling for quoted table names (e.g., "django_migrations")
Protobuf Compatibility
- Moved ownership_token field from position 8 to position 15 to avoid breaking existing field
numbering - Ensures backward compatibility while adding ownership verification functionality
⚙️ CI/CD Improvements
Build System Optimization
- Aggressive disk space optimization for GitHub Actions to prevent build failures
- Optimized parallelism with smart build job configuration
- Separated build and test jobs for better resource management
- Enhanced caching strategy with proper target directory sharing
- Skip problematic benchmark tests that were causing CI instability
Test Infrastructure
- Added protocol fuzzing tests for security validation
- Enhanced rate limiter testing
- Comprehensive security audit test suite
- Improved SSL/TLS test coverage
📚 Documentation & Developer Experience
Enhanced Documentation
- Streamlined CLAUDE.md by removing historical content
- Added comprehensive security documentation
- Enhanced ORM compatibility guides
- Updated build and test instructions
Code Quality
- Added comprehensive code review guidelines
- Enhanced memory tracking and optimization utilities
- Improved string utilities for better performance
- Enhanced query executor with better error handling
🔧 Technical Improvements
Memory & Performance
- Memory-aware cache manager with intelligent eviction policies
- TTL-based caching system for improved query performance
- String optimization utilities for reduced memory allocation
- Enhanced connection management with better resource cleanup
Query Processing
- Improved CREATE TABLE translator with better PostgreSQL syntax support
- Enhanced binary protocol encoding for arrays and complex types
- Better query executor with improved error handling and logging
Total Changes: 21 commits with significant additions across security, ORM compatibility, and
infrastructure improvements.
Compatibility: This release maintains full backward compatibility while significantly expanding
PostgreSQL feature support and ORM framework integration.
Release v0.0.17
Summary
This release delivers comprehensive PostgreSQL catalog and function support with significant performance optimizations, enhancing compatibility with major ORM frameworks including Django, SQLAlchemy, Rails, and Ecto.
New Features
- PostgreSQL Session Functions: Complete session information access with
current_user()
,session_user()
,current_database()
,current_schema()
, andcurrent_schemas()
functions for ORM connection management - PostgreSQL Permission Functions: Full authorization and access control with enhanced
pg_has_role()
andhas_table_privilege()
functions supporting role membership checking and table access control - information_schema.triggers: Complete trigger introspection with all 17 PostgreSQL-standard columns for business logic discovery
- pg_tablespace Catalog: Enterprise tablespace management support with standard PostgreSQL tablespaces (pg_default, pg_global)
- information_schema.check_constraints: Check constraint metadata support for ORM validation frameworks
- information_schema.referential_constraints: Foreign key constraint introspection with all 9 PostgreSQL-standard columns
- information_schema.views: Complete view metadata support with 10 PostgreSQL-standard columns
- information_schema.routines: Function metadata for 40+ built-in functions across all categories
- pg_stats Table: Query optimization and performance hints with realistic statistics generation
- pg_roles and pg_user: Complete user and role management system for enterprise authentication
- pg_description: Object comment and documentation system with COMMENT DDL integration
- pg_proc Function Metadata: Comprehensive function introspection with 35+ built-in functions
- Enhanced pg_attribute: Column metadata with defaults, constraints, identity/SERIAL detection
- pg_index Support: Complete index management with multi-column index introspection
- pg_constraint Support: Full constraint introspection for foreign key, primary key, unique, and check constraints
- pg_depend Support: Rails sequence ownership detection for auto-increment columns
Improvements
- Performance Optimizations: Session identifier translation with memchr-optimized pattern detection
- Dual-processor Support: Integrated SessionIdentifierTranslator into both LazyQueryProcessor and UnifiedProcessor
- Binary Protocol Enhancements: Complete array binary encoding for psycopg3 compatibility
- Session-aware Architecture: Connection-per-session for proper transaction isolation across all catalog queries
- ORM Compatibility: Full support for Django inspectdb, SQLAlchemy reflection, Rails introspection, and Ecto schema analysis
Bug Fixes
- Fixed critical data type issues in referential_constraints (confrelid column read as INTEGER instead of STRING)
- Fixed arithmetic test failures with proper floating-point division using CAST(... AS REAL)
- Fixed information_schema binary protocol support with proper field descriptions
- Fixed clippy warnings improving code quality and maintainability
- Fixed cast translation issues preventing ARRAY[] syntax from working correctly
Breaking Changes
None - All changes are backward compatible with enhanced functionality
Release v0.0.16
Release Notes for v0.0.16
Summary
This release focuses on PostgreSQL compatibility improvements, fixing critical syntax errors and adding comprehensive support for database object comments and CREATE INDEX operator classes.
New Features
-
Complete PostgreSQL Comment System: Full implementation of
COMMENT ON
statements and comment functions- Added
obj_description(oid, catalog_name)
andcol_description(table_oid, column_number)
functions - Support for
COMMENT ON TABLE
,COMMENT ON COLUMN
, andCOMMENT ON FUNCTION
statements - Schema migration v12 with
__pgsqlite_comments
table for persistent comment storage - Comprehensive PostgreSQL catalog compatibility for metadata queries
- Added
-
CREATE INDEX Operator Class Support: PostgreSQL-compatible index creation syntax
- Support for
varchar_pattern_ops
,text_pattern_ops
,bpchar_pattern_ops
operator classes - Automatic translation to SQLite
COLLATE BINARY
for pattern-optimized indexes - Multiple operator classes in single index statements
- Full Django/SQLAlchemy migration compatibility
- Support for
Bug Fixes
- Fixed "near varchar_pattern_ops: syntax error": CREATE INDEX statements with PostgreSQL operator classes now work correctly
- Fixed "no such function: obj_description": Catalog queries using PostgreSQL comment functions now execute without errors
- Improved test stability: Reduced concurrent reads test threshold for better CI reliability
Improvements
- Enhanced PostgreSQL Compatibility: Better support for standard PostgreSQL catalog queries and metadata operations
- Code Quality: Fixed multiple clippy warnings including manual_strip, if_same_then_else, and unnecessary_unwrap
- Test Coverage: Added comprehensive integration and unit tests for new comment system and index operator class features
Breaking Changes
None. This release maintains full backward compatibility while adding new PostgreSQL compatibility features.
Release v0.0.15
Release Notes for v0.0.15 - 0.0.15-test
Summary
This release delivers complete PostgreSQL binary protocol support, enabling full compatibility with psycopg3's binary mode while fixing critical protocol issues and improving code quality.
Release v0.0.14
Release Notes for v0.0.14 - 0.0.14-test
Summary
This release introduces full PostgreSQL binary protocol support, delivering massive performance improvements with psycopg3-binary achieving ~139x overhead for SELECT queries (5x better than target). Major SQLAlchemy compatibility fixes ensure all tests pass for both psycopg2 and psycopg3 drivers, making pgsqlite production-ready for Python ORMs.
New Features
- Binary Protocol Support: Full implementation of PostgreSQL binary format encoders for Numeric/Decimal, UUID, JSONJSONB, Money, and other types
- psycopg3-binary Driver Support: Native binary encoding provides 19x faster SELECT performance compared to psycopg2
- Enhanced Type Inference: Proper PostgreSQL type OIDs for aliased columns, aggregates, and empty result sets
- Multi-Row INSERT RETURNING: Bulk insert operations now return correct row counts with proper rowid range queries
Improvements
- Performance Breakthrough: SELECT queries now achieve ~139x overhead (0.139ms) with psycopg3-binary, exceeding all performance
targets - SQLAlchemy Full Compatibility: All 8 ORM test scenarios pass including transactions, relationships, and cascade deletes
- Aggregate Function Types: SUM/AVG/MAX/MIN now return proper NUMERIC types instead of TEXT for better driver compatibility
- DateTime Handling: Binary timestamp parameters properly converted between PostgreSQL and Unix epochs
- Connection Isolation: Schema lookups now use session connections for proper transaction visibility
Bug Fixes
- Fixed json_object_agg returning JSON type instead of TEXT type
- Fixed timestamp conversion in VALUES clauses and scalar subqueries
- Fixed parameter cast queries bypassing ultra-fast path optimizations
- Fixed column alias type inference for
table.column AS alias
patterns - Fixed date function translation creating malformed julianday syntax
- Fixed empty result sets defaulting all columns to TEXT type
- Fixed psycopg3 binary parameters returning 0 rows in queries
- Fixed aggregate aliases like
sum_1
returning TEXT instead of NUMERIC
Breaking Changes
None - This release maintains backward compatibility while adding new binary protocol capabilities.
Release v0.0.13
Release Notes for v0.0.13 - 0.0.13-test
Summary
This release focuses on performance optimizations through a new unified query processor architecture and fixes critical SQLAlchemy compatibility issues with aggregate functions. While the new architecture provides a cleaner foundation for future optimizations, significant performance regressions were discovered that require immediate attention.
New Features
- Unified Query Processor: New streamlined query processing architecture that combines fast path detection and lazy query processing into a single, efficient system - Thread-Local Connection Cache: LRU cache for session connections to reduce mutex contention in high-concurrency scenarios
- Fast Path Query Detection: Byte-level query detector to skip heavy processing for simple queries
- RETURNING Clause Optimization: Optimized handling of simple RETURNING clauses in DML operations
Improvements
- Code Reduction: 54% reduction in query processing code (870 → ~400 lines)
- Zero-Allocation Fast Path: Simple queries now use
Cow::Borrowed
to avoid allocations - Progressive Complexity Detection: Checks cheap patterns first for better performance
- Benchmark Suite: Added comprehensive Python benchmarking tools for performance analysis
Bug Fixes
- SQLAlchemy Aggregate Types: Fixed "Unknown PG numeric type: 25" error when using MAX/MIN on DECIMAL columns
- Aggregate Type Detection: Added support for SQLAlchemy's aliased aggregate columns (e.g., "max_1")
- Build Warnings: Fixed all compilation warnings in simple_query_detector and unified_processor modules
- Unit Tests: All 372 unit tests now pass without warnings
Breaking Changes
None
Performance Notes
- SELECT: 4.016ms (599x worse than 0.669ms target)
- INSERT: 0.163ms (269x worse than 0.060ms target)
- UPDATE: 0.053ms (90x worse than target)
- DELETE: 0.033ms (100x worse than target)
The unified processor architecture provides a foundation for future optimizations including SIMD pattern ma
Release v0.0.12
Release Notes for v0.0.12 - 0.0.12-test
Summary
This release delivers critical SQLAlchemy ORM compatibility fixes and introduces PostgreSQL Full-Text Search support. Major architectural improvements include connection-per-session isolation and comprehensive transaction persistence fixes.
New Features
- PostgreSQL Full-Text Search (FTS): Complete implementation using SQLite FTS5 backend
- Support for tsvector/tsquery types and @@ match operator
- Functions: to_tsvector(), to_tsquery(), plainto_tsquery(), phraseto_tsquery(), websearch_to_tsquery()
- Zero performance impact on non-FTS queries
Improvements
-
Connection-Per-Session Architecture: Each PostgreSQL client gets its own SQLite connection
- Proper transaction isolation matching PostgreSQL behavior
- Fixes SQLAlchemy transaction persistence issues with WAL mode
- Eliminates transaction visibility problems between sessions
-
SQLAlchemy ORM Compatibility: Full support for Python SQLAlchemy 2.0+
- Fixed PostgreSQL type OID mapping for psycopg2 compatibility
- Multi-row INSERT VALUES pattern translation for bulk operations
- JOIN query type inference for proper relationship handling
- DateTime column alias resolution and INTEGER storage conversion
- RETURNING clause support for all DML operations
Bug Fixes
- Fixed duplicate UUID generation in cached queries with gen_random_uuid()
- Fixed NOW() function returning epoch time instead of current timestamp
- Fixed AT TIME ZONE operator encoding issues with simple_query protocol
- Fixed NUMERIC binary encoding causing Unicode decode errors
- Fixed information_schema.tables returning incorrect table names
- Fixed DELETE ... USING syntax translation for batch operations
- Resolved "Migration lock held by process" errors in concurrent tests
Breaking Changes
None - This release maintains backward compatibility while adding new features.
Release v0.0.10
Release Notes for v0.0.10 - 0.0.10-test
Summary
This release brings significant performance improvements through connection pooling, enhanced PostgreSQL compatibility with comprehensive function support, and critical bug fixes for prepared statements. The query optimization system has been completely overhauled, delivering up to 21% performance improvements for SELECT queries.
New Features
- Connection Pooling with Read/Write Separation: Production-ready connection pooling infrastructure with automatic query routing, health checks, and transaction affinity
- PostgreSQL Functions: Implemented 35+ PostgreSQL-compatible functions including string functions (split_part, string_agg, translate), math functions (trunc with precision, trigonometric functions), and more
- Enhanced psql Support: Full \d tablename command support with populated pg_constraint, pg_attrdef, and pg_index catalog tables
- Comprehensive Query Optimization: New optimization infrastructure featuring read-only optimizer, enhanced statement caching, and intelligent query plan caching
Improvements
- SELECT Performance: 21% improvement (from ~369x to ~291x overhead) through query optimization
- Cache Effectiveness: Maintained 1.6x speedup for cached queries
- Connection Pool Performance: Baseline 95,961 QPS single-thread, 124,380 QPS with 8 concurrent tasks
- Statement Caching: Enhanced caching system now handles 200+ cached query plans with priority-based eviction
- Query Router: Intelligent classification and routing of SELECT vs DML queries for optimal resource utilization
Bug Fixes
- BIT Type Cast Issues: Fixed prepared statements returning empty strings for BIT type casts
- SQL Parser Enhancement: Resolved parser errors with parameterized BIT types (::bit(8), ::varbit(10))
- DateTime Roundtrip: Fixed binary encoding issues for TIME values and proper microsecond precision
- Performance Regression: Eliminated 34% performance regression through fast-path optimizations
Breaking Changes
None - all changes maintain backward compatibility
Release v0.0.9
Release Notes for v0.0.9 - 0.0.9-test
Summary
This release completes the array and JSON feature sets for pgsqlite, adding full support for PostgreSQL-compatible array literals, operators, and functions, along with comprehensive JSON manipulation capabilities. The release also includes significant codebase improvements and test infrastructure enhancements.
New Features
- ARRAY Literal Syntax: Support for
ARRAY[1,2,3]
syntax with automatic translation to JSON format - ALL Operator: Fixed and enhanced ALL operator with proper balanced parentheses parsing
- unnest() WITH ORDINALITY: PostgreSQL-compatible implementation with 1-based indexing
- JSON Row Conversion: Complete
row_to_json()
function for converting table rows to JSON objects - JSON Manipulation Functions:
jsonb_insert()
- Insert values into JSON objects/arraysjsonb_delete()
- Delete values from JSON by pathjsonb_pretty()
- Pretty-print JSON with formatting
- JSON Table Functions:
json_each_text()
andjsonb_each_text()
for text-based key-value expansion - JSON Aggregation:
json_populate_record()
andjson_to_record()
for record conversions
Improvements
- Test Infrastructure: Reorganized SQL integration tests into categorized subdirectories for better maintainability
- Array Performance: Zero performance impact from array enhancements - maintained ~280x SELECT overhead
- Code Organization: Cleaned up root directory by removing temporary files
- Test Coverage: Comprehensive CI/CD validation suite for all JSON and array functions
Bug Fixes
- Fixed simple query detector to ensure array queries use the translation pipeline
- Resolved array concatenation operator (
||
) detection for ARRAY[] syntax patterns - Fixed JSON path operator conflicts with SQL parser $ character handling
- Corrected wire protocol conversion for JSON arrays to PostgreSQL format
Breaking Changes
None
Release v0.0.7
Release Notes for v0.0.7 - 0.0.7-test
Summary
This release enhances array type support in pgsqlite, providing better compatibility with PostgreSQL array operations and fixing critical issues with array value handling.
New Features
- Enhanced Array Support: Improved PostgreSQL array type compatibility with better handling of array literals and operations
- Array Wire Protocol: Fixed array value conversion between JSON storage format and PostgreSQL wire protocol format
Improvements
- Better array literal parsing for both
ARRAY[...]
and{...}
syntax formats - Enhanced array type validation and constraint checking
- Improved performance for array operations through optimized conversion paths
Bug Fixes
- Fixed wire protocol conversion for JSON arrays to properly return PostgreSQL-formatted array strings
- Resolved array value handling in multi-row INSERT statements
- Corrected array type registration in the pg_type system catalog
Breaking Changes
None