Searchable drizzle api #264
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add new query encryption API for searchable encryption: - encryptQuery(): Single value query encryption with index type control - createQuerySearchTerms(): Bulk query encryption with mixed index types - createJsonSearchTerms(): JSON path and containment query encryption Features: - Support for all index types: ore, match, unique, ste_vec - Lock context support for all query operations - SEM-only payloads (no ciphertext) optimized for database queries - Path queries (dot notation and array format) - Containment queries (contains/contained_by) Test coverage includes: - Lock context integration tests - Boundary conditions (empty strings, Unicode, emoji, large numbers) - Deep JSON nesting (5+ levels) - Bulk operation edge cases - Error handling scenarios
Remove public API additions that diverged from requirements: - Requirements specified using existing createSearchTerms function - Requirements specified NOT changing the existing protectjs public API Removed: - encryptQuery(), createQuerySearchTerms(), createJsonSearchTerms() methods - Public type exports for query-specific types - Test files for removed public API Internal operation files remain for potential future use.
- Revert package.json from local link to published 0.19.0 - Define IndexTypeName and QueryOpName locally in types.ts - These types will be available from FFI once 0.20.0 is released
Add 32 tests covering JsonSearchTermsOperation including: - Path queries (string/array paths, deep paths, path-only) - Containment queries (simple/nested objects, multiple keys) - Bulk operations (mixed queries, multiple columns) - Lock context integration - Edge cases (unicode, deep nesting, special chars) - Error handling (missing ste_vec index) - Selector generation verification
Add missing public methods to ProtectClient: - encryptQuery: encrypt single value with explicit index type - createQuerySearchTerms: bulk query term encryption - createJsonSearchTerms: JSON path/containment query encryption Update tests to use public API instead of unsafe internal access. Export new operation types and search term types.
Updates README.md, schema reference, and searchable encryption guides to include details on the new JSON search capabilities (path and containment queries).
…operations Covers encryptQuery and createQuerySearchTerms with unique, ORE, and match indexes, as well as composite-literal return types and lock context integration.
SearchTerm is now a union of SimpleSearchTerm, JsonPathSearchTerm, and JsonContainmentSearchTerm, enabling createSearchTerms to accept all search term types in a single call. - Add SimpleSearchTerm type alias for original behavior - Update SearchTerm to union type - Export SimpleSearchTerm from public API
SearchTermsOperation.execute() now handles JSON search terms: - Partitions terms by type (simple, JSON path, JSON containment) - Encrypts simple terms with encryptBulk (original behavior) - Encrypts JSON terms with encryptQueryBulk (ste_vec index) - Reassembles results in original order - Supports mixed batches of simple and JSON terms Also includes: - Type guards for SearchTerm variants - Helper functions (pathToSelector, buildNestedObject, flattenJson) - withLockContext support for JSON terms - Extracted shared logic into encryptSearchTermsHelper to reduce duplication
Tests for: - JSON path search term via createSearchTerms - JSON containment search term via createSearchTerms - Mixed simple and JSON search terms in single call
Add @deprecated JSDoc tag to guide users toward createSearchTerms. Implementation unchanged to avoid breaking existing code.
Remove the deprecated createJsonSearchTerms function and supporting code, consolidating JSON search functionality into the unified createSearchTerms API. - Remove createJsonSearchTerms method from ProtectClient - Delete json-search-terms.ts operation file - Remove JsonSearchTermsOperation export from index - Migrate comprehensive tests to search-terms.test.ts - Update documentation examples to use createSearchTerms
Add missing lock context integration tests for JSON search terms and refactor test file to use shared beforeAll client for efficiency.
Remove __RESOLVE_AT_BUILD__ placeholder in favor of inferring the ste_vec prefix from table/column context when not explicitly set. Changes: - searchableJson() now sets empty ste_vec object - ProtectTable.build() and buildEncryptConfig() infer prefix when missing - Simplified error checks in search-terms.ts - Enabled previously commented test for ste_vec index
Add tests to prevent regressions based on code review feedback: - Selector prefix resolution test verifying table/column prefix - encryptQuery(null) null handling verification - escaped-composite-literal return type for createQuerySearchTerms - ste_vec index with default queryOp for JSON object encryption
Set temporary column name prefix in searchableJson() to satisfy type requirements, then always overwrite with full table/column prefix during build. Update search-terms.ts to always derive prefix from table/column names rather than relying on column.build() which may have incomplete prefix. This fixes the DTS build error where prefix was required by the type but not set until table build time.
- Add major version changeset for @cipherstash/protect and @cipherstash/schema - Clarify searchableJson() exclusivity is enforced by backend, not TypeScript - Update nested objects section to reference searchableJson() alternative - Fix parameter tables to include returnType field - Add SQL equivalent comments to JSON query examples - Consolidate duplicate JSON search sections into single cohesive section - Enhance TSDoc for encryptQuery() and createQuerySearchTerms() with usage examples - Add platform docs links to IndexTypeName type definition
- Remove duplicate TSDoc from internal operation classes, mark as @internal - Add cross-references to public interface (ProtectClient methods) - Clarify SQL comments to show plaintext equivalent queries
- Remove incorrect returnType property from JSON query term docs (returnType only applies to ScalarQueryTerm, not JSON terms) - Export IndexTypeName and QueryOpName types from index.ts (enables consumers to fully utilize the type system) Found via dual-verification review.
…ching values - Rename IndexTypeName to QueryTypeName - Change values: ore → orderAndRange, match → freeTextSearch, unique → equality, ste_vec → searchableJson - Add queryTypes constant for convenient import - Update JSDoc examples to use new API - Add work files to .gitignore
Export ColumnInfo for use in json-operators module.
Add searchableJson option to enable JSON path and containment queries on encrypted JSON columns.
Normalizes JSONPath format ($.user.email) to dot notation (user.email). Handles both formats for user convenience.
When extractProtectSchema encounters a column with searchableJson: true, it now calls .searchableJson() on the ProtectColumn builder.
JsonPathBuilder holds column, path, and config for JSON operations. Will provide chainable methods for comparison and value extraction.
LazyJsonOperator extends the lazy operator pattern for JSON operations. Includes type guard isLazyJsonOperator for batching logic.
Returns a LazyJsonOperator for JSON path equality comparison. Integrates with the lazy operator pattern for batching support.
Complete the comparison method set for JSON path operations. All methods return lazy operators for batching support.
pathExtract() and pathExtractFirst() encrypt the current path to get a selector, then use EQL v2 functions for extraction. Includes WithSelector variants for pre-encrypted selectors.
get() returns SQL expression for SELECT clauses. arrayLength() returns new builder in array-length mode for comparison chaining. Array-length on root path needs no encryption. Array-length on nested path encrypts the path selector (not the comparison value).
Entry point for JSON path operations on encrypted columns. Validates searchableJson config and returns JsonPathBuilder.
Array expansion methods for SELECT clauses using EQL v2 functions. Root path: direct expansion. Nested path: path extraction then expansion. Includes sync variants for pre-encrypted selectors.
Exports JsonPathBuilder, isLazyJsonOperator, normalizePath and types for advanced usage patterns.
Detects LazyJsonOperator instances and encrypts them using the JSON-specific encryption logic before combining with regular conditions.
Add test to verify JSON operators are properly batched with regular operators in and() calls. Test verifies detection and SQL generation.
Mirrors and() implementation for JSON operator detection and encryption.
Produces correct EQL v2 SQL for all JSON operator types: - json_eq/json_ne: jsonb_path_match - json_contains/json_contained_by: jsonb_contains/contained_by - json_array_length_*: jsonb_array_length with comparison operators Exports createJsonOperatorExecute factory for testing and manual operator construction. Array-length operators now correctly include the comparison value in the generated SQL.
Documents all JSON path operations with examples covering: - Path comparisons (eq, ne) - Containment checks (contains, containedBy) - Array operations (arrayLength with comparisons) - Value and path extraction - Integration with other operators
Tests now mock encryptQuery and createSearchTerms methods on protectClient to verify JSON operator handling without requiring a full encryption environment.
🦋 Changeset detectedLatest commit: cf28220 The changes in this PR will be included in the next version bump. This PR includes changesets to release 10 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
coderdan
reviewed
Jan 27, 2026
| } | ||
|
|
||
| expect(result.data).toHaveLength(2) | ||
| expect(result.data[0]).toHaveProperty('hm') // unique returns HMAC |
Contributor
There was a problem hiding this comment.
We should check the second term as well.
coderdan
reviewed
Jan 27, 2026
| } | ||
|
|
||
| expect(result.data).toHaveLength(1) | ||
| expect(result.data[0]).toHaveProperty('s', 'json_users/metadata/user/email') |
Contributor
There was a problem hiding this comment.
The selector isn't encrypted.
coderdan
reviewed
Jan 27, 2026
| } | ||
|
|
||
| expect(result.data).toHaveLength(1) | ||
| expect(result.data[0]).toEqual({ s: 'json_users/metadata/user/role' }) |
coderdan
reviewed
Jan 27, 2026
Comment on lines
107
to
111
| expect(result.data).toHaveLength(1) | ||
| expect(result.data[0]).toHaveProperty('sv') | ||
| const sv = (result.data[0] as any).sv | ||
| expect(sv).toHaveLength(1) | ||
| expect(sv[0]).toHaveProperty('s', 'json_users/metadata/role') |
Contributor
There was a problem hiding this comment.
It doesn't look like this is encrypting anything.
coderdan
reviewed
Jan 27, 2026
Comment on lines
195
to
201
| it('should encrypt query with lock context', async () => { | ||
| const userJwt = process.env.USER_JWT | ||
|
|
||
| if (!userJwt) { | ||
| console.log('Skipping lock context test - no USER_JWT provided') | ||
| return | ||
| } |
Contributor
There was a problem hiding this comment.
Lock context is meaningless for queries.
coderdan
requested changes
Jan 27, 2026
Contributor
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
My guess is that this hasn't been run before opening the PR. None of the operations actually encrypt anything. I'm not sure what to do with this tbh.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.