Dynamic Indexing Feature Design

Dynamic Indexing Feature Design

Executive Summary

This document presents implementation options for dynamic indexing in Redis OM Spring, leveraging existing Spring Data Redis infrastructure and Redis OM Spring's current capabilities. All solutions guarantee 100% backward compatibility - existing applications continue working without any changes.

Problem Statement

Community Requirements by Theme

Multi-Tenancy & Security

• Dynamic Index Names (#634, #638): Runtime index name generation for tenant isolation and ACL control
• Custom Key Prefixes (#634): Tenant-specific prefixes for data isolation
• ACL Integration (#634): Index naming patterns that work with Redis ACLs for fine-grained access control

DevOps & Index Lifecycle Management

• Index Versioning (#26): Support for versioned indexes during schema evolution
• Index Aliasing (#590, #26): Enable blue-green deployments and zero-downtime migrations
• Migration Tools (#26): Maven/Gradle tasks for index management and migration
• Index Maintenance (#26): Tools to export, review, and manage index definitions

Flexibility & Advanced Patterns

• Ephemeral Indexes (#376): Temporary indexes with TTL for transient data
• Index Decoupling (#638): Separate index definition from document definition
• Config Reuse (#332): Use single entity class with multiple index configurations
• CQRS Support (#590): Different indexes for read and write operations

Related issues:

• #634 - Custom index names for ACL control
• #638 - Decouple index and document definitions
• #590 - Blue-green deployment support
• #376 - Ephemeral indexes
• #332 - Multiple configs with single class
• #26 - Index maintenance and migrations

Current Infrastructure Analysis

Spring Data Redis Index Infrastructure

Spring Data Redis provides an index infrastructure in org.springframework.data.redis.core.index:

// Core interfaces we can leverage
public interface IndexDefinition {
    String getKeyspace();
    Collection<Condition<?>> getConditions();
    IndexValueTransformer valueTransformer();
    String getIndexName();
}

public interface IndexDefinitionProvider {
    boolean hasIndexFor(Serializable keyspace);
    Set<IndexDefinition> getIndexDefinitionsFor(Serializable keyspace);
}

public interface ConfigurableIndexDefinitionProvider
    extends IndexDefinitionProvider, IndexDefinitionRegistry {
    // Allows dynamic registration of index definitions
}

Redis OM Spring Current Implementation

Redis OM Spring has its own parallel indexing system:

1. RediSearchIndexer: Manages RediSearch indexes independently
2. IndexingOptions: Controls index creation behavior
3. IndexCreationMode: SKIP_IF_EXIST,DROP_AND_RECREATE,SKIP_ALWAYS
4. Minimal Integration: Uses empty IndexConfiguration() in repository setup

Implementation Options

Option 1: Bridge RediSearchIndexer with Spring Data Redis Infrastructure

Design

Create a bridge between Redis OM's RediSearchIndexer and Spring Data Redis's ConfigurableIndexDefinitionProvider:

// NEW: Bridge class implementing Spring Data Redis interface
@Component
public class RediSearchIndexDefinitionProvider
    implements ConfigurableIndexDefinitionProvider {

    private final RediSearchIndexer indexer;
    private final Map<String, Set<RediSearchIndexDefinition>> definitions;

    @Override
    public Set<IndexDefinition> getIndexDefinitionsFor(Serializable keyspace) {
        // Bridge to RediSearchIndexer's index definitions
        Class<?> entityClass = indexer.getEntityClassForKeyspace(keyspace.toString());
        if (entityClass == null) return Collections.emptySet();

        // Convert RediSearch schema to IndexDefinition
        return convertToIndexDefinitions(entityClass);
    }

    @Override
    public void addIndexDefinition(IndexDefinition definition) {
        // Allow runtime registration of new index definitions
        if (definition instanceof RediSearchIndexDefinition) {
            registerDynamicIndex((RediSearchIndexDefinition) definition);
        }
    }
}

// NEW: Custom IndexDefinition for RediSearch
public class RediSearchIndexDefinition implements IndexDefinition {
    private final String indexName;
    private final String keyspace;
    private final List<SchemaField> schemaFields;
    private final IndexNamingStrategy namingStrategy;

    @Override
    public String getIndexName() {
        // Can be dynamic based on context
        RedisIndexContext context = RedisIndexContextHolder.getContext();
        return namingStrategy != null && context != null
            ? namingStrategy.getIndexName(keyspace, context)
            : indexName;
    }
}

Pros

• ✅ Integrates with Spring Data Redis infrastructure
• ✅ Allows runtime index registration
• ✅ Maintains RediSearch-specific features
• ✅ Zero breaking changes

Cons

• ❌ Complexity of bridging two systems
• ❌ Potential performance overhead

Option 2: Enhance IndexingOptions with Dynamic Resolution

Design

Extend the existing IndexingOptions annotation to support dynamic resolution:

// ENHANCE: Existing annotation with new capabilities
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface IndexingOptions {
    String indexName() default "";

    // NEW: Support for SpEL expressions
    String dynamicIndexName() default "";

    // NEW: Support for custom naming strategy
    Class<? extends IndexNamingStrategy> namingStrategy() default DefaultNamingStrategy.class;

    // Existing
    IndexCreationMode creationMode() default IndexCreationMode.SKIP_IF_EXIST;

    // NEW: Support for runtime index creation
    boolean allowRuntimeCreation() default false;
}

// NEW: Interface for dynamic naming
public interface IndexNamingStrategy {
    String resolveIndexName(Class<?> entityClass, String baseIndexName);
    String resolveKeyPrefix(Class<?> entityClass, String basePrefix);
}

// ENHANCE: RediSearchIndexer to support dynamic resolution
public class RediSearchIndexer {

    private final Map<Class<?>, IndexNamingStrategy> namingStrategies = new ConcurrentHashMap<>();

    public void createIndexFor(Class<?> cl) {
        IndexingOptions options = cl.getAnnotation(IndexingOptions.class);

        // Dynamic index name resolution
        String indexName;
        if (options != null && !options.dynamicIndexName().isEmpty()) {
            // Evaluate SpEL expression
            indexName = evaluateExpression(options.dynamicIndexName(), cl);
        } else if (options != null && options.namingStrategy() != DefaultNamingStrategy.class) {
            // Use custom naming strategy
            IndexNamingStrategy strategy = getOrCreateStrategy(options.namingStrategy());
            indexName = strategy.resolveIndexName(cl, options.indexName());
        } else {
            // Current behavior
            indexName = getCurrentIndexName(cl, options);
        }

        // Rest remains the same...
    }
}

Pros

• ✅ Minimal changes to existing code
• ✅ Leverages existing annotation infrastructure
• ✅ Clear migration path
• ✅ SpEL support for simple cases

Cons

• ❌ Limited to compile-time configuration
• ❌ Cannot change strategy at runtime

Option 3: Context-Aware RediSearchIndexer

Design

Enhance RediSearchIndexer to be context-aware while maintaining backward compatibility:

// NEW: Context for index resolution
public class RedisIndexContext {
    private final Map<String, Object> attributes;
    private final String tenantId;
    private final String environment;

    // ThreadLocal management
    private static final ThreadLocal<RedisIndexContext> CONTEXT = new ThreadLocal<>();

    public static void setContext(RedisIndexContext context) {
        CONTEXT.set(context);
    }

    public static RedisIndexContext getContext() {
        return CONTEXT.get();
    }
}

// ENHANCE: RediSearchIndexer with context support
@Component
public class RediSearchIndexer {

    // NEW: Index resolver for dynamic resolution
    private IndexResolver indexResolver = new DefaultIndexResolver();

    public String getIndexName(Class<?> entityClass) {
        // Check for context first
        RedisIndexContext context = RedisIndexContext.getContext();
        if (context != null) {
            // Dynamic resolution based on context
            return indexResolver.resolveIndexName(entityClass, context);
        }

        // Fall back to current behavior
        return entityClassToIndexName.get(entityClass);
    }

    // NEW: Support for runtime index creation
    public void createIndexForContext(Class<?> entityClass, RedisIndexContext context) {
        String contextualIndexName = indexResolver.resolveIndexName(entityClass, context);

        // Check if index already exists for this context
        if (!indexExistsForContext(entityClass, context)) {
            // Create index with context-specific name and prefix
            createContextualIndex(entityClass, context, contextualIndexName);
        }
    }
}

// NEW: Interface for index resolution
public interface IndexResolver {
    String resolveIndexName(Class<?> entityClass, RedisIndexContext context);
    String resolveKeyPrefix(Class<?> entityClass, RedisIndexContext context);
}

Pros

• ✅ True runtime dynamism
• ✅ Clean separation of concerns
• ✅ Thread-safe context handling
• ✅ Extensible resolver pattern

Cons

• ❌ Requires context management
• ❌ More complex than annotation-based

Index Lifecycle Management

Index Versioning Strategy

Support versioned indexes for zero-downtime schema migrations:

@Document("users")
@IndexingOptions(
    indexName = "users_idx",
    versioningStrategy = IndexVersioningStrategy.TIMESTAMP,
    aliasName = "users_current"  // Alias always points to active version
)
public class User {
    // When schema changes, new index created as users_idx_20240115_1420
    // Alias 'users_current' switches after data migration
}

Migration Workflow

@Component
public class IndexMigrationService {

    public void migrateIndex(Class<?> entityClass, MigrationStrategy strategy) {
        // 1. Create new versioned index
        String newIndexName = createVersionedIndex(entityClass);

        // 2. Migrate data (configurable strategy)
        switch (strategy) {
            case DUAL_WRITE:
                enableDualWrite(entityClass, newIndexName);
                reindexInBackground(entityClass, newIndexName);
                break;
            case BLUE_GREEN:
                reindexToNewIndex(entityClass, newIndexName);
                switchAlias(entityClass, newIndexName);
                break;
        }

        // 3. Verify and cleanup
        verifyIndexIntegrity(newIndexName);
        scheduleOldIndexCleanup(entityClass);
    }
}

Index Maintenance Tools

Maven Plugin

<plugin>
    <groupId>com.redis.om</groupId>
    <artifactId>redis-om-spring-maven-plugin</artifactId>
    <executions>
        <execution>
            <goals>
                <goal>export-indexes</goal>  <!-- Export index definitions -->
                <goal>validate-indexes</goal> <!-- Validate against Redis -->
                <goal>migrate-indexes</goal>  <!-- Run migrations -->
            </goals>
        </execution>
    </executions>
</plugin>

Gradle Task

task exportIndexes(type: RedisIndexExportTask) {
    outputDir = file("$buildDir/redis-indexes")
    format = 'json' // or 'redis-cli'
}

task migrateIndexes(type: RedisIndexMigrationTask) {
    migrationDir = file("src/main/resources/redis/migrations")
    strategy = 'blue-green'
}

Security & Access Control Integration

ACL-Compatible Index Naming

Support index naming patterns that work with Redis ACLs:

public class AclAwareIndexNamingStrategy implements IndexNamingStrategy {

    // Generate: <env>:<service>:<tenant>:<entity>:<version>
    // Example: prod:user-service:tenant123:users:v1
    @Override
    public String resolveIndexName(Class<?> entityClass, RedisIndexContext context) {
        return String.format("%s:%s:%s:%s:v%d",
            context.getEnvironment(),      // prod, staging, dev
            context.getServiceName(),       // microservice name
            context.getTenantId(),          // tenant identifier
            entityClass.getSimpleName().toLowerCase(),
            context.getSchemaVersion()
        );
    }
}

// Redis ACL configuration
// ACL SETUSER tenant123 ~prod:user-service:tenant123:* &prod:user-service:tenant123:* +@read +@write

Per-Tenant Index Isolation

@Configuration
public class TenantIndexConfiguration {

    @Bean
    public IndexNamingStrategy tenantAwareStrategy() {
        return new TenantAwareIndexNamingStrategy();
    }

    @Bean
    public TenantContextResolver tenantResolver() {
        return () -> {
            // Extract from JWT/OAuth2 token
            Authentication auth = SecurityContextHolder.getContext().getAuthentication();
            return ((OAuth2Authentication) auth).getTenantId();
        };
    }
}

Recommended Approach: Hybrid Solution

Combine the best aspects of each option:

Phase 1: Enhanced IndexingOptions (Quick Win)

@Document("users")
@IndexingOptions(
    indexName = "users_idx",
    dynamicIndexName = "#{T(com.example.TenantContext).currentTenant + '_users_idx'}",
    creationMode = IndexCreationMode.SKIP_IF_EXIST
)
public class User {
    // ...
}

Phase 2: Context-Aware Indexer

// For programmatic control
@Service
public class TenantAwareService {
    @Autowired
    private RediSearchIndexer indexer;

    public void setupTenantIndex(String tenantId) {
        RedisIndexContext context = RedisIndexContext.builder()
            .tenantId(tenantId)
            .build();

        // Create tenant-specific index
        indexer.createIndexForContext(User.class, context);
    }
}

Phase 3: Spring Data Redis Bridge (Long-term)

Implement ConfigurableIndexDefinitionProvider to fully integrate with Spring Data Redis ecosystem.

Backward Compatibility Guarantee

Default Behavior Unchanged

// Existing code works exactly as today
@Document("products")
public class Product {
    @Id private String id;
    @Indexed private String name;
}

// No changes needed to repositories
@Repository
public interface ProductRepository extends RedisDocumentRepository<Product, String> {
    List<Product> findByName(String name);
}

Opt-in Dynamic Features

// Only when explicitly configured
@Document("orders")
@IndexingOptions(
    dynamicIndexName = "#{@tenantService.currentTenant + '_orders_idx'}"
)
public class Order {
    // Dynamic index only when SpEL expression is present
}

Implementation Plan

Phase 1: Core Enhancements (2 weeks)

• Enhance IndexingOptions with SpEL support
• Add IndexNamingStrategy interface
• Update RediSearchIndexer to evaluate dynamic expressions

Phase 2: Context Support (2 weeks)

• Implement RedisIndexContext
• Add context-aware methods to RediSearchIndexer
• Create IndexResolver interface and implementations

Phase 3: Runtime Index Management (3 weeks)

• Support for runtime index creation
• Index aliasing support
• Migration tools

Phase 4: Spring Data Redis Integration (4 weeks)

• Implement ConfigurableIndexDefinitionProvider
• Bridge RediSearch indexes with Spring Data Redis
• Full integration testing

Testing Strategy

@Test
public void existingCodeWorksUnchanged() {
    // All existing tests must pass without modification
}

@Test
public void dynamicIndexNameResolution() {
    // Test SpEL evaluation in IndexingOptions
}

@Test
public void contextAwareIndexing() {
    // Test context-based index resolution
}

@Test
public void multiTenantScenario() {
    // Test isolation between tenants
}

Comparison with Other Spring Data Projects

Feature	Redis OM Spring	MongoDB	Elasticsearch	JPA
Dynamic Collection/Index Names	Via SpEL + Context	Template methods	SpEL in @Document	N/A
Multi-tenancy	Context-based	Database switching	Index per tenant	Schema/Discriminator
Runtime Creation	Supported	Supported	Supported	Limited

Real-World Usage Examples

Example 1: Multi-Tenant SaaS Application

@Document("users")
@IndexingOptions(
    namingStrategy = TenantAwareIndexNamingStrategy.class,
    creationMode = IndexCreationMode.SKIP_IF_EXIST
)
public class User {
    @Id private String id;
    @Indexed private String tenantId;
    @Indexed private String email;
    @Searchable private String name;
}

@RestController
public class UserController {
    @Autowired
    private UserRepository userRepository;

    @GetMapping("/users/{id}")
    public User getUser(@PathVariable String id,
                        @AuthenticationPrincipal OAuth2User principal) {
        // Context automatically set from OAuth2 token
        String tenantId = principal.getAttribute("tenant_id");

        try (var context = RedisIndexContext.withTenant(tenantId)) {
            // Repository uses tenant-specific index: tenant123_users_idx
            return userRepository.findById(id).orElseThrow();
        }
    }
}

Example 2: Blue-Green Deployment

@Service
public class DeploymentService {
    @Autowired
    private IndexMigrationService migrationService;

    public void performBlueGreenDeployment() {
        // 1. Create new "green" index with updated schema
        String greenIndex = "products_idx_green_v2";
        createIndexWithNewSchema(Product.class, greenIndex);

        // 2. Dual-write to both indexes
        enableDualWrite("products_idx_blue_v1", greenIndex);

        // 3. Backfill green index
        reindexInBackground(Product.class, greenIndex);

        // 4. Switch alias atomically
        switchAlias("products_current", greenIndex);

        // 5. Cleanup old index after verification
        scheduleCleanup("products_idx_blue_v1", Duration.ofHours(24));
    }
}

Example 3: Ephemeral Index for Batch Processing

@Component
public class BatchProcessor {
    @Autowired
    private RediSearchIndexer indexer;

    public void processBatch(List<Order> orders) {
        // Create temporary index for batch
        String tempIndex = String.format("batch_%s_idx", UUID.randomUUID());

        RedisIndexContext context = RedisIndexContext.builder()
            .attribute("indexName", tempIndex)
            .attribute("ttl", Duration.ofHours(2))
            .build();

        // Create ephemeral index
        indexer.createIndexForContext(Order.class, context);

        try {
            // Process batch with dedicated index
            processBatchWithIndex(orders, tempIndex);
        } finally {
            // Auto-cleanup after TTL or manual cleanup
            indexer.dropIndex(tempIndex);
        }
    }
}

Example 4: CQRS with Separate Read/Write Indexes

@Document("events")
@IndexingOptions(
    indexName = "events_write_idx",
    readIndexName = "events_read_idx",  // Optimized for queries
    cqrsMode = true
)
public class Event {
    @Id private String id;
    @Indexed private Instant timestamp;
    @Indexed private String type;
    @Searchable private String description;
}

@Service
public class EventService {
    @Autowired
    private EventRepository repository;

    public void writeEvent(Event event) {
        // Writes go to write-optimized index
        repository.save(event);

        // Async projection to read index
        projectToReadIndex(event);
    }

    public List<Event> searchEvents(String query) {
        // Queries use read-optimized index with materialized views
        return repository.searchFromReadIndex(query);
    }
}

Conclusion

This design addresses all community requirements by:

1. Multi-tenancy & Security: Dynamic index naming with ACL support (#634, #638)
2. DevOps & Lifecycle: Versioning, aliasing, and migration tools (#26, #590)
3. Flexibility: Ephemeral indexes and config reuse (#376, #332)
4. Spring Integration: Leveraging existing Spring Data Redis infrastructure

The solution provides:

• 100% backward compatibility - existing applications work unchanged
• Gradual adoption - features can be enabled incrementally
• Production-ready patterns - blue-green deployments, CQRS, multi-tenancy
• DevOps friendly - Maven/Gradle plugins for index management

All changes build upon Redis OM Spring's existing infrastructure (IndexingOptions, IndexCreationMode, RediSearchIndexer) while adding the dynamic capabilities the community needs.