feat(tokenization): implement atomic batch operations

conductor/archive/batch_tokenization_20260309/index.md

@@ -0,0 +1,5 @@
+# Track batch_tokenization_20260309 Context
+
+- [Specification](./spec.md)
+- [Implementation Plan](./plan.md)
+- [Metadata](./metadata.json)

conductor/archive/batch_tokenization_20260309/metadata.json

@@ -0,0 +1,8 @@
+{
+  "track_id": "batch_tokenization_20260309",
+  "type": "feature",
+  "status": "new",
+  "created_at": "2026-03-09T14:30:00Z",
+  "updated_at": "2026-03-09T14:30:00Z",
+  "description": "Add Batch Tokenize/Detokenize Endpoints (wrap existing single-item logic in a loop with transaction)"
+}

conductor/archive/batch_tokenization_20260309/plan.md

@@ -0,0 +1,49 @@
+# Implementation Plan: Batch Tokenize/Detokenize Endpoints
+
+This plan outlines the steps to implement batch tokenization and detokenization endpoints in the Secrets manager.
+
+## Phase 1: Domain and Repository Layer
+
+- [x] Task: Define Batch Tokenization Interfaces [e8a8cee]
+    - [ ] Add `CreateBatch` to `TokenRepository` interface in `internal/tokenization/domain/token.go`.
+    - [ ] Add `GetBatchByTokens` to `TokenRepository` interface in `internal/tokenization/domain/token.go`.
+- [x] Task: Implement Batch Repository Methods (PostgreSQL) [517777c]
+    - [ ] Implement `CreateBatch` in `internal/tokenization/repository/postgresql/token_repository.go`.
+    - [ ] Implement `GetBatchByTokens` in `internal/tokenization/repository/postgresql/token_repository.go`.
+    - [ ] Write integration tests for these methods (tagged with `//go:build integration`).
+- [x] Task: Implement Batch Repository Methods (MySQL) [cc03816]
+    - [ ] Implement `CreateBatch` in `internal/tokenization/repository/mysql/token_repository.go`.
+    - [ ] Implement `GetBatchByTokens` in `internal/tokenization/repository/mysql/token_repository.go`.
+    - [ ] Write integration tests for these methods (tagged with `//go:build integration`).
+- [x] Task: Implement Batch Usecase Logic [191cb29]
+    - [ ] Add `TokenizeBatch` to `TokenizationUsecase` in `internal/tokenization/usecase/tokenization_usecase.go`.
+    - [ ] Add `DetokenizeBatch` to `TokenizationUsecase` in `internal/tokenization/usecase/tokenization_usecase.go`.
+    - [ ] Ensure both methods use `TxManager` for atomicity.
+    - [ ] Implement the loop over existing single-item logic.
+    - [ ] Write unit tests for the new usecase methods.
+- [x] Task: Conductor - User Manual Verification 'Phase 1: Domain and Repository Layer' (Protocol in workflow.md)
+
+## Phase 2: HTTP Layer
+
+- [x] Task: Define Request/Response DTOs [cc85bfe]
+    - [ ] Create `TokenizeBatchRequest` and `TokenizeBatchResponse` in `internal/tokenization/http/dto.go` (or equivalent).
+    - [ ] Create `DetokenizeBatchRequest` and `DetokenizeBatchResponse` in `internal/tokenization/http/dto.go`.
+    - [ ] Implement validation rules (e.g., max 100 items).
+- [x] Task: Implement HTTP Handlers [ee3290b]
+    - [ ] Implement `TokenizeBatch` handler in `internal/tokenization/http/tokenization_handler.go`.
+    - [ ] Implement `DetokenizeBatch` handler in `internal/tokenization/http/tokenization_handler.go`.
+    - [ ] Write unit tests for the new handlers in `internal/tokenization/http/tokenization_handler_test.go`.
+- [x] Task: Register Routes [ee3290b]
+    - [ ] Add the new batch routes to the router in `internal/tokenization/http/tokenization_handler.go` (or `internal/app/di_tokenization.go`).
+- [x] Task: Conductor - User Manual Verification 'Phase 2: HTTP Layer' (Protocol in workflow.md)
+
+## Phase 3: Documentation and Integration Testing
+
+- [x] Task: Update Integration Flow Tests [efa3c2c]
+    - [ ] Add batch operation test cases to `test/integration/tokenization_flow_test.go`.
+    - [ ] Verify atomicity by intentionally failing one item in a batch.
+- [x] Task: Update OpenAPI Specification [fa50f71]
+    - [ ] Add the new batch endpoints to `docs/openapi.yaml`.
+- [x] Task: Update Engine Documentation [9faab2e]
+    - [ ] Update `docs/engines/tokenization.md` with examples of batch requests and responses.
+- [x] Task: Conductor - User Manual Verification 'Phase 3: Documentation and Integration Testing' (Protocol in workflow.md)

conductor/archive/batch_tokenization_20260309/spec.md

@@ -0,0 +1,39 @@
+# Specification: Batch Tokenize/Detokenize Endpoints
+
+## Overview
+This track introduces batch processing capabilities to the Tokenization Engine. Currently, tokenization and detokenization are performed on a single item at a time. This feature will add new endpoints to allow clients to tokenize or detokenize multiple items in a single request, wrapped in a database transaction for atomicity.
+
+## Functional Requirements
+- **New Endpoints:**
+  - `POST /v1/tokenization/keys/:name/tokenize-batch`: Batch tokenize a list of values using a named key.
+  - `POST /v1/tokenization/detokenize-batch`: Batch detokenize a list of tokens.
+- **Batch Limit:** A configurable limit of 100 items per batch request will be enforced to ensure performance and prevent resource exhaustion.
+- **Atomicity:** Both batch endpoints MUST be atomic. If any single item in the batch fails (e.g., validation error, database failure), the entire request MUST fail, and any database changes MUST be rolled back.
+- **Request/Response Formats:**
+  - `tokenize-batch`:
+    - Request: `{"values": ["val1", "val2", ...]}`
+    - Response: `{"tokens": ["token1", "token2", ...]}`
+  - `detokenize-batch`:
+    - Request: `{"tokens": ["token1", "token2", ...]}`
+    - Response: `{"values": ["val1", "val2", ...]}`
+- **Documentation:**
+  - Update `docs/engines/tokenization.md` to include batch operations.
+  - Update `docs/openapi.yaml` with the new endpoint definitions.
+
+## Non-Functional Requirements
+- **Performance:** Batch processing should be more efficient than multiple single-item calls by reducing network round-trips and utilizing a single database transaction.
+- **Security:** Standard capability validation (`tokenize` or `detokenize`) must be enforced for the batch operations.
+
+## Acceptance Criteria
+- [ ] Clients can successfully tokenize up to 100 values in a single call.
+- [ ] Clients can successfully detokenize up to 100 tokens in a single call.
+- [ ] If any value in a `tokenize-batch` request is invalid, the entire request returns an error (400 Bad Request) and no tokens are created.
+- [ ] If any token in a `detokenize-batch` request is invalid, the entire request returns an error (400 Bad Request) and no values are returned.
+- [ ] The batch limit is enforced and returns a 400 Bad Request if exceeded.
+- [ ] Unit tests cover new domain logic, usecase methods, and HTTP handlers.
+- [ ] Integration tests in `test/integration/tokenization_flow_test.go` cover batch operations for both PostgreSQL and MySQL.
+- [ ] Documentation (`docs/engines/tokenization.md`) and OpenAPI spec (`docs/openapi.yaml`) are updated.
+
+## Out of Scope
+- Partial success/failure handling for batch requests.
+- Asynchronous batch processing.

conductor/product.md

@@ -14,7 +14,7 @@ To provide a secure, developer-friendly, and lightweight secrets management plat
 ## Core Features
 - **Secret Management (Storage):** Versioned, envelope-encrypted storage with support for arbitrary key-value pairs and strict path validation.
 - **Transit Engine (EaaS):** On-the-fly encryption/decryption of application data without database storage.
-- **Tokenization Engine:** Format-preserving tokens for sensitive data types like credit card numbers.
+- **Tokenization Engine:** Format-preserving tokens for sensitive data types like credit card numbers, with support for atomic batch processing.
 - **Auth Token Revocation:** Immediate invalidation of authentication tokens (single or client-wide) with full state management.
 - **Client Secret Rotation:** Self-service and administrative rotation of client secrets with automatic auth token revocation.
 - **Audit Logs:** HMAC-signed audit trails capturing every access attempt and policy evaluation, with support for advanced filtering by client and date range.

docs/engines/tokenization.md

@@ -74,6 +74,27 @@ Example response (`201 Created`):
 }
 ```
 
+### Tokenize Data (Batch)
+
+- **Endpoint**: `POST /v1/tokenization/keys/:name/tokenize-batch`
+- **Capability**: `encrypt`
+- **Body**: `items` (array of objects with `plaintext`, `metadata`, `ttl`).
+- **Limit**: Maximum 100 items per batch.
+
+Generates tokens for multiple plaintext values in a single atomic operation. If any item fails (e.g., invalid format), the entire batch is rejected.
+
+```bash
+curl -X POST http://localhost:8080/v1/tokenization/keys/payment-cards/tokenize-batch \
+  -H "Authorization: Bearer <token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "items": [
+      { "plaintext": "NDUzMjAxNTExMjgzMDM2Ng==", "metadata": { "index": 1 } },
+      { "plaintext": "NTQ5ODAxNTExMjgzMDM2Nw==", "metadata": { "index": 2 } }
+    ]
+  }'
+```
+
 ### Detokenize Data
 
 - **Endpoint**: `POST /v1/tokenization/detokenize`
@@ -91,6 +112,24 @@ Example response (`200 OK`):
 }
 ```
 
+### Detokenize Data (Batch)
+
+- **Endpoint**: `POST /v1/tokenization/detokenize-batch`
+- **Capability**: `decrypt`
+- **Body**: `{"tokens": ["string", "string"]}`
+- **Limit**: Maximum 100 tokens per batch.
+
+Retrieves original plaintext values for multiple tokens in a single atomic operation.
+
+```bash
+curl -X POST http://localhost:8080/v1/tokenization/detokenize-batch \
+  -H "Authorization: Bearer <token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "tokens": ["4532015112830366", "5498015112830367"]
+  }'
+```
+
 ### Validate and Revoke
 
 - `POST /v1/tokenization/validate` (Capability: `read`) - Check if token is valid without returning plaintext.

docs/openapi.yaml

@@ -911,6 +911,46 @@ paths:
           $ref: "#/components/responses/ValidationError"
         "429":
           $ref: "#/components/responses/TooManyRequests"
+  /v1/tokenization/keys/{name}/tokenize-batch:
+    post:
+      tags: [tokenization]
+      summary: Tokenize multiple plaintexts in batch
+      description: Generates tokens for multiple plaintext values in a single atomic operation.
+      security:
+        - bearerAuth: []
+      parameters:
+        - name: name
+          in: path
+          required: true
+          schema:
+            type: string
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: "#/components/schemas/TokenizeBatchRequest"
+      responses:
+        "201":
+          description: Tokens created
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/TokenizeBatchResponse"
+        "401":
+          $ref: "#/components/responses/Unauthorized"
+        "403":
+          $ref: "#/components/responses/Forbidden"
+        "404":
+          description: Tokenization key not found
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/ErrorResponse"
+        "422":
+          $ref: "#/components/responses/ValidationError"
+        "429":
+          $ref: "#/components/responses/TooManyRequests"
   /v1/tokenization/detokenize:
     post:
       tags: [tokenization]
@@ -944,6 +984,40 @@ paths:
           $ref: "#/components/responses/ValidationError"
         "429":
           $ref: "#/components/responses/TooManyRequests"
+  /v1/tokenization/detokenize-batch:
+    post:
+      tags: [tokenization]
+      summary: Detokenize multiple tokens in batch
+      description: Retrieves original plaintext values for multiple tokens in a single atomic operation.
+      security:
+        - bearerAuth: []
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: "#/components/schemas/DetokenizeBatchRequest"
+      responses:
+        "200":
+          description: Plaintexts resolved
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/DetokenizeBatchResponse"
+        "401":
+          $ref: "#/components/responses/Unauthorized"
+        "403":
+          $ref: "#/components/responses/Forbidden"
+        "404":
+          description: One or more tokens not found
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/ErrorResponse"
+        "422":
+          $ref: "#/components/responses/ValidationError"
+        "429":
+          $ref: "#/components/responses/TooManyRequests"
   /v1/tokenization/validate:
     post:
       tags: [tokenization]
@@ -1455,6 +1529,16 @@ components:
           type: integer
           minimum: 1
       required: [plaintext]
+    TokenizeBatchRequest:
+      type: object
+      properties:
+        items:
+          type: array
+          minItems: 1
+          maxItems: 100
+          items:
+            $ref: "#/components/schemas/TokenizeRequest"
+      required: [items]
     TokenizeResponse:
       type: object
       properties:
@@ -1471,12 +1555,30 @@ components:
           format: date-time
           nullable: true
       required: [token, created_at]
+    TokenizeBatchResponse:
+      type: object
+      properties:
+        items:
+          type: array
+          items:
+            $ref: "#/components/schemas/TokenizeResponse"
+      required: [items]
     DetokenizeRequest:
       type: object
       properties:
         token:
           type: string
       required: [token]
+    DetokenizeBatchRequest:
+      type: object
+      properties:
+        tokens:
+          type: array
+          minItems: 1
+          maxItems: 100
+          items:
+            type: string
+      required: [tokens]
     DetokenizeResponse:
       type: object
       properties:
@@ -1487,6 +1589,14 @@ components:
           type: object
           additionalProperties: true
       required: [plaintext]
+    DetokenizeBatchResponse:
+      type: object
+      properties:
+        items:
+          type: array
+          items:
+            $ref: "#/components/schemas/DetokenizeResponse"
+      required: [items]
     ValidateTokenRequest:
       type: object
       properties:

internal/app/di_tokenization.go

@@ -249,6 +249,11 @@ func (c *Container) initTokenizationKeyUseCase(
 func (c *Container) initTokenizationUseCase(
 	ctx context.Context,
 ) (tokenizationUseCase.TokenizationUseCase, error) {
+	txManager, err := c.TxManager(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get tx manager for tokenization use case: %w", err)
+	}
+
 	tokenizationKeyRepository, err := c.TokenizationKeyRepository(ctx)
 	if err != nil {
 		return nil, fmt.Errorf(
@@ -279,6 +284,7 @@ func (c *Container) initTokenizationUseCase(
 	}
 
 	baseUseCase := tokenizationUseCase.NewTokenizationUseCase(
+		txManager,
 		tokenizationKeyRepository,
 		tokenRepository,
 		dekRepository,

internal/http/server.go

@@ -408,6 +408,12 @@ func (s *Server) registerTokenizationRoutes(
 				authHTTP.AuthorizationMiddleware(authDomain.EncryptCapability, auditLogUseCase, s.logger),
 				tokenizationHandler.TokenizeHandler,
 			)
+
+			// Tokenize batch of plaintexts with tokenization key
+			keys.POST("/:name/tokenize-batch",
+				authHTTP.AuthorizationMiddleware(authDomain.EncryptCapability, auditLogUseCase, s.logger),
+				tokenizationHandler.TokenizeBatchHandler,
+			)
 		}
 
 		// Detokenize token to retrieve plaintext
@@ -416,6 +422,12 @@ func (s *Server) registerTokenizationRoutes(
 			tokenizationHandler.DetokenizeHandler,
 		)
 
+		// Detokenize batch of tokens to retrieve plaintexts
+		tokenization.POST("/detokenize-batch",
+			authHTTP.AuthorizationMiddleware(authDomain.DecryptCapability, auditLogUseCase, s.logger),
+			tokenizationHandler.DetokenizeBatchHandler,
+		)
+
 		// Validate token existence and validity
 		tokenization.POST("/validate",
 			authHTTP.AuthorizationMiddleware(authDomain.ReadCapability, auditLogUseCase, s.logger),