Searchable json query api #257

tobyhede · 2026-01-16T02:41:19Z

No description provided.

…eries

Add new query encryption API for searchable encryption: - encryptQuery(): Single value query encryption with index type control - createQuerySearchTerms(): Bulk query encryption with mixed index types - createJsonSearchTerms(): JSON path and containment query encryption Features: - Support for all index types: ore, match, unique, ste_vec - Lock context support for all query operations - SEM-only payloads (no ciphertext) optimized for database queries - Path queries (dot notation and array format) - Containment queries (contains/contained_by) Test coverage includes: - Lock context integration tests - Boundary conditions (empty strings, Unicode, emoji, large numbers) - Deep JSON nesting (5+ levels) - Bulk operation edge cases - Error handling scenarios

changeset-bot · 2026-01-16T02:41:22Z

🦋 Changeset detected

Latest commit: b65022d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages

Name	Type
@cipherstash/protect	Major
@cipherstash/schema	Major
@cipherstash/drizzle	Major
@cipherstash/protect-dynamodb	Major
@cipherstash/basic-example	Patch
@cipherstash/dynamo-example	Patch
nest	Patch
next-drizzle-mysql	Patch
@cipherstash/nextjs-clerk-example	Patch
@cipherstash/typeorm-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Remove public API additions that diverged from requirements: - Requirements specified using existing createSearchTerms function - Requirements specified NOT changing the existing protectjs public API Removed: - encryptQuery(), createQuerySearchTerms(), createJsonSearchTerms() methods - Public type exports for query-specific types - Test files for removed public API Internal operation files remain for potential future use.

- Revert package.json from local link to published 0.19.0 - Define IndexTypeName and QueryOpName locally in types.ts - These types will be available from FFI once 0.20.0 is released

Add 32 tests covering JsonSearchTermsOperation including: - Path queries (string/array paths, deep paths, path-only) - Containment queries (simple/nested objects, multiple keys) - Bulk operations (mixed queries, multiple columns) - Lock context integration - Edge cases (unicode, deep nesting, special chars) - Error handling (missing ste_vec index) - Selector generation verification

Add missing public methods to ProtectClient: - encryptQuery: encrypt single value with explicit index type - createQuerySearchTerms: bulk query term encryption - createJsonSearchTerms: JSON path/containment query encryption Update tests to use public API instead of unsafe internal access. Export new operation types and search term types.

Updates README.md, schema reference, and searchable encryption guides to include details on the new JSON search capabilities (path and containment queries).

…operations Covers encryptQuery and createQuerySearchTerms with unique, ORE, and match indexes, as well as composite-literal return types and lock context integration.

SearchTerm is now a union of SimpleSearchTerm, JsonPathSearchTerm, and JsonContainmentSearchTerm, enabling createSearchTerms to accept all search term types in a single call. - Add SimpleSearchTerm type alias for original behavior - Update SearchTerm to union type - Export SimpleSearchTerm from public API

SearchTermsOperation.execute() now handles JSON search terms: - Partitions terms by type (simple, JSON path, JSON containment) - Encrypts simple terms with encryptBulk (original behavior) - Encrypts JSON terms with encryptQueryBulk (ste_vec index) - Reassembles results in original order - Supports mixed batches of simple and JSON terms Also includes: - Type guards for SearchTerm variants - Helper functions (pathToSelector, buildNestedObject, flattenJson) - withLockContext support for JSON terms - Extracted shared logic into encryptSearchTermsHelper to reduce duplication

Tests for: - JSON path search term via createSearchTerms - JSON containment search term via createSearchTerms - Mixed simple and JSON search terms in single call

@deprecated

Add @deprecated JSDoc tag to guide users toward createSearchTerms. Implementation unchanged to avoid breaking existing code.

Remove the deprecated createJsonSearchTerms function and supporting code, consolidating JSON search functionality into the unified createSearchTerms API. - Remove createJsonSearchTerms method from ProtectClient - Delete json-search-terms.ts operation file - Remove JsonSearchTermsOperation export from index - Migrate comprehensive tests to search-terms.test.ts - Update documentation examples to use createSearchTerms

Add missing lock context integration tests for JSON search terms and refactor test file to use shared beforeAll client for efficiency.

Remove __RESOLVE_AT_BUILD__ placeholder in favor of inferring the ste_vec prefix from table/column context when not explicitly set. Changes: - searchableJson() now sets empty ste_vec object - ProtectTable.build() and buildEncryptConfig() infer prefix when missing - Simplified error checks in search-terms.ts - Enabled previously commented test for ste_vec index

Add tests to prevent regressions based on code review feedback: - Selector prefix resolution test verifying table/column prefix - encryptQuery(null) null handling verification - escaped-composite-literal return type for createQuerySearchTerms - ste_vec index with default queryOp for JSON object encryption

Set temporary column name prefix in searchableJson() to satisfy type requirements, then always overwrite with full table/column prefix during build. Update search-terms.ts to always derive prefix from table/column names rather than relying on column.build() which may have incomplete prefix. This fixes the DTS build error where prefix was required by the type but not set until table build time.

packages/protect/src/types.ts

calvinbrewer

This looks great @tobyhede good stuff

packages/protect/src/types.ts

calvinbrewer · 2026-01-20T22:06:09Z

Note: you don't have a changeset in here - might be worth bumping the major version just to be safe to account for all the different typescript targets

docs/reference/schema.md

docs/reference/searchable-encryption-postgres.md

packages/protect/__tests__/query-search-terms.test.ts

packages/protect/src/types.ts

packages/protect/src/ffi/operations/search-terms.ts

packages/protect/src/ffi/operations/query-search-terms.ts

packages/protect/src/ffi/operations/search-terms.ts

packages/protect/src/ffi/operations/query-search-terms.ts

packages/protect/src/ffi/operations/encrypt-query.ts

packages/protect/src/ffi/operations/batch-encrypt-query.ts

- Encrypt ste_vec selectors using the 'ste_vec_selector' operation. - Normalize JSON paths to standard '$' prefixed JSONPath strings for FFI compatibility. - Remove 'withLockContext' from all search-related operations (SearchTermsOperation, QuerySearchTermsOperation, EncryptQueryOperation, BatchEncryptQueryOperation) as it is not applicable to queries. - Update test suite to expect hex string tokens for selectors and remove now-unsupported LockContext tests. - Address review feedback regarding naming conventions and result construction.

- Move schema documentation to packages/schema/src/index.ts. - Move configuration and initialization documentation to packages/protect/src/index.ts. - Move model operation documentation to packages/protect/src/ffi/index.ts. - Move searchable encryption and PostgreSQL integration documentation to packages/protect/src/types.ts. - Move Supabase and composite type helper documentation to packages/protect/src/helpers/index.ts. - Add integration tips such as the ::jsonb cast requirement for Supabase/PostgreSQL. - Wired in shared test helpers into batch-encrypt-query.test.ts and search-terms.test.ts.

…helpers Replace weak assertions (toBeDefined, toHaveProperty, Object.keys.length) with specific validation helpers that verify encrypted payloads contain expected fields like selectors, ciphertext, and sv arrays. Add expectBasicEncryptedPayload helper and complete migration of all tests to use consistent assertion patterns.

Leverage FFI automatic mode inference to simplify JSON query encryption. The FFI now infers query mode from plaintext type (object → StoreMode, string → SteVecSelector), eliminating separate selector encryption and manual assembly. - Remove flattenJson and pathToSelector from json-path-utils - Simplify batch-encrypt-query.ts and search-terms.ts - Path+value queries now return { sv: [...] } format (same as containment) - Update tests to expect new output format

Add programmatic error handling support by: - Re-export FfiProtectError class and ProtectErrorCode type from FFI - Add optional code field to ProtectError interface - Preserve FFI error codes in encrypt-query, batch-encrypt-query, and search-terms operations when wrapping errors Consumers can now distinguish specific error conditions like UNKNOWN_COLUMN, MISSING_INDEX, INVALID_JSON_PATH, etc.

Add tests verifying FFI type inference behavior: - String JSON path plaintext → selector-only output (s field) - Object/array plaintext → containment output (sv array) - Explicit ste_vec_selector and ste_vec_term queryOp respected Add tests verifying error code propagation: - UNKNOWN_COLUMN for non-existent columns - MISSING_INDEX for columns without required index - Error codes preserved alongside error messages

Switch @cipherstash/protect-ffi from local file link to github:cipherstash/protectjs-ffi#add-stevec-type-inference

Update all NEW API examples to use the current naming convention: - indexType: 'unique' → queryType: 'equality' - indexType: 'match' → queryType: 'freeTextSearch' Deprecated/old API examples retain indexType for migration reference.

- Add TIP noting queryType is optional for single-index columns - Clarify ScalarQueryTerm valid queryType values in type table

Switch from git branch reference to npm release. Fixes ste_vec query encryption returning wrong format.

Update assertion to match new error message format from protect-ffi. The library now returns 'Cannot convert X to Float' instead of 'Unsupported conversion'.

Add tests for JSON extraction operators (->, jsonb_path_query, jsonb_path_query_first, jsonb_array_length) ensuring parity with proxy functionality. - Verify Equality, Order, and Range operations on extracted values. - Verify array elements filtering via wildcard paths. - Ensure encrypted output (HMAC/ORE) and absence of plaintext.

- Update indexType to queryType parameter in code examples - Replace deprecated createSearchTerms with encryptQuery - Fix containment query to use contains property pattern - Fix broken link to composite types doc (use Drizzle guide) - Fix incorrect npm-lockfile-v3 filename in link - Add searchable JSON to documentation index - Note createSearchTerms deprecation in AGENTS.md

tobyhede added 3 commits January 15, 2026 16:02

feat(schema): enable searchableJson() method for SteVec indexing

e941688

feat(protect): add JSON search term types for containment and path qu…

6a9208c

…eries

tobyhede added 14 commits January 16, 2026 13:57

chore: update protect-ffi to 0.20.0

a112a44

fix: use local type definitions until protect-ffi 0.20.0 release

bedda29

- Revert package.json from local link to published 0.19.0 - Define IndexTypeName and QueryOpName locally in types.ts - These types will be available from FFI once 0.20.0 is released

chore: update protect-ffi to 0.20.0

b0c00d2

docs: add documentation for searchable encrypted JSON

b660611

Updates README.md, schema reference, and searchable encryption guides to include details on the new JSON search capabilities (path and containment queries).

test(protect): add comprehensive tests for explicit query encryption …

f5e4793

…operations Covers encryptQuery and createQuerySearchTerms with unique, ORE, and match indexes, as well as composite-literal return types and lock context integration.

test(protect): add JSON support tests for createSearchTerms

6be18b4

Tests for: - JSON path search term via createSearchTerms - JSON containment search term via createSearchTerms - Mixed simple and JSON search terms in single call

deprecate(protect): mark createJsonSearchTerms as deprecated

2d7a40d

Add @deprecated JSDoc tag to guide users toward createSearchTerms. Implementation unchanged to avoid breaking existing code.

test(protect): add lock context tests and optimize client initialization

3f9aed2

Add missing lock context integration tests for JSON search terms and refactor test file to use shared beforeAll client for efficiency.

tobyhede force-pushed the searchable-json-query-api branch from 66227cd to c4f5d8c Compare January 20, 2026 04:29

tobyhede added 3 commits January 20, 2026 15:46

tobyhede commented Jan 20, 2026

View reviewed changes

packages/protect/src/types.ts Show resolved Hide resolved

calvinbrewer approved these changes Jan 20, 2026

View reviewed changes

packages/protect/src/types.ts Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/schema.md Outdated Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/schema.md Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/searchable-encryption-postgres.md Outdated Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/searchable-encryption-postgres.md Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/searchable-encryption-postgres.md Outdated Show resolved Hide resolved