Skip to content

[8] Create XML to JSON Migration Tool #159

@michaelbeale-IL

Description

@michaelbeale-IL

Estimate: 2 days
Sprint: Week 3-4
Assignee: [Developer]


Description

Build a command-line tool to migrate existing XML configuration files to JSON format, using the schemas created in Ticket #7.

Context

  • Users have existing XML configurations
  • Need automated migration to avoid manual conversion
  • Should validate output against JSON schemas

AI Prompt

Create a ConfigMigrationTool console application that:

1. Discovers all XML configuration files in a directory
2. Identifies the schema type of each XML file
3. Converts to equivalent JSON using the POCOs from Ticket #7
4. Validates converted JSON against schema
5. Writes JSON files to output directory
6. Generates migration report with:
   - Files processed
   - Successful conversions
   - Errors/warnings
   - Manual review needed

Include:
- Progress bar during migration
- Dry-run mode (preview changes)
- Backup original files
- Rollback capability
- Detailed logging

Use System.CommandLine for CLI interface.

Tasks

  • Create new console project ACAT.ConfigMigrationTool
  • AI generates migration logic
  • Add XML → POCO deserializers
  • Add POCO → JSON serializers
  • Implement validation
  • Add CLI arguments and help
  • Test with real XML files
  • Create user documentation

Acceptance Criteria

  • ✅ Tool successfully converts all test XML files
  • ✅ JSON validates against schemas
  • ✅ No data loss in conversion
  • ✅ Migration report generated
  • ✅ Dry-run mode works
  • ✅ Backup files created
  • ✅ User documentation complete

CLI Interface

# Dry run (preview)
ConfigMigrationTool.exe migrate --input "C:\ACAT\Config" --dry-run

# Actual migration
ConfigMigrationTool.exe migrate --input "C:\ACAT\Config" --output "C:\ACAT\ConfigJson" --backup

# Validate only
ConfigMigrationTool.exe validate --input "C:\ACAT\ConfigJson"

# Rollback
ConfigMigrationTool.exe rollback --backup "C:\ACAT\Config.backup"

Example Implementation

// Program.cs
using System.CommandLine;

var rootCommand = new RootCommand("ACAT Configuration Migration Tool");

var migrateCommand = new Command("migrate", "Migrate XML configs to JSON");
var inputOption = new Option<string>("--input", "Input directory") { IsRequired = true };
var outputOption = new Option<string>("--output", "Output directory") { IsRequired = true };
var dryRunOption = new Option<bool>("--dry-run", "Preview changes without applying");
var backupOption = new Option<bool>("--backup", "Backup original files");

migrateCommand.AddOption(inputOption);
migrateCommand.AddOption(outputOption);
migrateCommand.AddOption(dryRunOption);
migrateCommand.AddOption(backupOption);

migrateCommand.SetHandler(async (string input, string output, bool dryRun, bool backup) =>
{
    var migrator = new ConfigurationMigrator();
    var result = await migrator.MigrateAsync(input, output, dryRun, backup);
    Console.WriteLine(result.GenerateReport());
}, inputOption, outputOption, dryRunOption, backupOption);

rootCommand.AddCommand(migrateCommand);
await rootCommand.InvokeAsync(args);

// ConfigurationMigrator.cs
public class ConfigurationMigrator
{
    public async Task<MigrationResult> MigrateAsync(
        string inputDir, 
        string outputDir, 
        bool dryRun, 
        bool backup)
    {
        var result = new MigrationResult();
        var xmlFiles = Directory.GetFiles(inputDir, "*.xml", SearchOption.AllDirectories);
        
        foreach (var xmlFile in xmlFiles)
        {
            try
            {
                // Detect schema type
                var schemaType = DetectSchemaType(xmlFile);
                
                // Convert XML → POCO → JSON
                var json = ConvertToJson(xmlFile, schemaType);
                
                // Validate
                var isValid = ValidateJson(json, schemaType);
                
                if (!dryRun && isValid)
                {
                    if (backup)
                    {
                        File.Copy(xmlFile, xmlFile + ".backup", true);
                    }
                    
                    var jsonPath = Path.Combine(outputDir, 
                        Path.GetFileNameWithoutExtension(xmlFile) + ".json");
                    await File.WriteAllTextAsync(jsonPath, json);
                }
                
                result.SuccessCount++;
            }
            catch (Exception ex)
            {
                result.Errors.Add((xmlFile, ex.Message));
            }
        }
        
        return result;
    }
}

Testing

# Test with sample files
ConfigMigrationTool.exe migrate --input "test-data/xml" --output "test-data/json" --dry-run

# Verify output
cat test-data/json/ActuatorSettings.json
# Should be valid JSON

# Validate against schema
ConfigMigrationTool.exe validate --input "test-data/json"
# Should report all valid

Dependencies

Package Requirements

<PackageReference Include="System.CommandLine" Version="2.0.0-beta4.22272.1" />
<PackageReference Include="System.Text.Json" Version="8.0.0" />

🎫 Ticket #9: Update ACAT to Load JSON Configurations

Labels: phase-1-foundation, configuration, P1-High, refactor
Estimate: 2 days
Assignee: [Developer]
Sprint: Week 4

Description

Update ACAT's configuration loading code to read JSON files instead of XML, using the POCOs and validators from Ticket #7.

Context

  • Currently loads XML via custom parsers
  • Need to switch to JSON deserialization
  • Must maintain backward compatibility during transition

Tasks

  • Identify all XML configuration loading code
  • Create JsonConfigurationLoader<T> utility
  • Update each configuration loader to use JSON
  • Add validation on load
  • Handle missing/corrupted files gracefully
  • Add fallback to defaults if config missing
  • Test with real JSON files

Acceptance Criteria

  • ✅ All configuration types load from JSON
  • ✅ Validation runs on load
  • ✅ Invalid config shows user-friendly error
  • ✅ Missing config falls back to defaults
  • ✅ No references to XML loading remain
  • ✅ Application runs with JSON configs
  • ✅ All existing features work

Implementation Example

// JsonConfigurationLoader.cs
public class JsonConfigurationLoader<T> where T : class
{
    private readonly ILogger<JsonConfigurationLoader<T>> _logger;
    private readonly IValidator<T> _validator;
    
    public JsonConfigurationLoader(
        ILogger<JsonConfigurationLoader<T>> logger,
        IValidator<T> validator = null)
    {
        _logger = logger;
        _validator = validator;
    }
    
    public async Task<T> LoadAsync(string filePath)
    {
        try
        {
            _logger.LogInformation("Loading configuration from {FilePath}", filePath);
            
            if (!File.Exists(filePath))
            {
                _logger.LogWarning("Configuration file not found: {FilePath}", filePath);
                return null;
            }
            
            var json = await File.ReadAllTextAsync(filePath);
            var config = JsonSerializer.Deserialize<T>(json, new JsonSerializerOptions
            {
                PropertyNameCaseInsensitive = true,
                AllowTrailingCommas = true,
                ReadCommentHandling = JsonCommentHandling.Skip
            });
            
            if (config == null)
            {
                throw new InvalidOperationException("Failed to deserialize configuration");
            }
            
            // Validate if validator provided
            if (_validator != null)
            {
                var validationResult = await _validator.ValidateAsync(config);
                if (!validationResult.IsValid)
                {
                    var errors = string.Join(", ", validationResult.Errors.Select(e => e.ErrorMessage));
                    throw new ValidationException($"Configuration validation failed: {errors}");
                }
            }
            
            _logger.LogInformation("Configuration loaded and validated successfully");
            return config;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to load configuration from {FilePath}", filePath);
            throw;
        }
    }
    
    public T LoadOrDefault(string filePath, Func<T> defaultFactory)
    {
        try
        {
            return LoadAsync(filePath).GetAwaiter().GetResult();
        }
        catch
        {
            _logger.LogWarning("Using default configuration");
            return defaultFactory();
        }
    }
}

// Usage in ActuatorManager
public class ActuatorManager
{
    private readonly JsonConfigurationLoader<ActuatorSettings> _configLoader;
    
    public ActuatorManager(JsonConfigurationLoader<ActuatorSettings> configLoader)
    {
        _configLoader = configLoader;
    }
    
    public async Task InitializeAsync()
    {
        var configPath = Path.Combine(
            FileUtils.GetUserConfigDir(), 
            "ActuatorSettings.json");
        
        var settings = await _configLoader.LoadAsync(configPath) 
            ?? ActuatorSettings.CreateDefault();
        
        // Use settings...
    }
}

Migration Path

// Temporary: Support both XML and JSON during transition
public async Task<ActuatorSettings> LoadSettingsAsync()
{
    var jsonPath = "ActuatorSettings.json";
    var xmlPath = "ActuatorSettings.xml";
    
    // Try JSON first
    if (File.Exists(jsonPath))
    {
        return await _jsonLoader.LoadAsync(jsonPath);
    }
    
    // Fallback to XML if JSON doesn't exist
    if (File.Exists(xmlPath))
    {
        _logger.LogWarning("XML config found. Please migrate to JSON using ConfigMigrationTool");
        var xmlSettings = LoadXml(xmlPath); // Old method
        
        // Auto-migrate
        var jsonSettings = ConvertToJson(xmlSettings);
        await File.WriteAllTextAsync(jsonPath, JsonSerializer.Serialize(jsonSettings));
        
        return jsonSettings;
    }
    
    // Neither exists - use defaults
    return ActuatorSettings.CreateDefault();
}

Testing

  • Unit tests for JsonConfigurationLoader
  • Integration tests with real JSON files
  • Test validation errors
  • Test missing file handling
  • Test corrupted JSON handling
  • End-to-end test: Launch ACAT with JSON configs

Dependencies


🎫 Ticket #10: Update Documentation for JSON Configuration

Labels: phase-1-foundation, configuration, P2-Medium, docs
Estimate: 1 day
Assignee: [Developer]
Sprint: Week 4

Description

Create comprehensive documentation for the new JSON configuration system, including migration guide, schema reference, and examples.

Tasks

  • Create user migration guide
  • Document JSON schemas
  • Create configuration examples
  • Update developer documentation
  • Create troubleshooting guide
  • Add IntelliSense setup instructions

Acceptance Criteria

  • ✅ Migration guide complete and tested
  • ✅ All JSON schemas documented
  • ✅ Example files for each config type
  • ✅ Developer guide updated
  • ✅ Troubleshooting common issues documented
  • ✅ VS Code IntelliSense setup documented

Documentation Structure

# ACAT JSON Configuration Guide

## Migration from XML
[Instructions for using ConfigMigrationTool]

## Configuration Files

### ActuatorSettings.json
**Location:** `%APPDATA%\ACAT\Config\ActuatorSettings.json`

**Schema Reference:**
- `actuators` (array, required): List of input devices
  - `name` (string, required): Device name
  - `enabled` (boolean): Enable/disable device
  - `switches` (array): Button/key mappings

**Example:**
```json
{
  "actuators": [
    {
      "name": "Keyboard",
      "enabled": true,
      "switches": [
        { "name": "Enter", "action": "Select" }
      ]
    }
  ]
}

VS Code IntelliSense Setup

[Instructions for associating schemas]

Troubleshooting

[Common issues and solutions]


### Dependencies
- Ticket #7, #8, #9 (System complete)

---

## 🎫 Ticket #11: Phase 1 Integration Testing

**Labels:** `phase-1-foundation`, `test`, `P1-High`, `integration`  
**Estimate:** 1 day  
**Assignee:** [Developer]  
**Sprint:** Week 4

### Description
Comprehensive integration testing of Phase 1 deliverables: logging and configuration systems working together.

### Test Scenarios
- [ ] **Fresh Install**
  - Clean user directory
  - Launch ACAT
  - Verify default configs created (JSON)
  - Verify logs created
  
- [ ] **XML Migration**
  - User has existing XML configs
  - Run migration tool
  - Launch ACAT with new JSON configs
  - Verify all settings preserved
  
- [ ] **Logging in Production**
  - Configure production log levels
  - Run ACAT for 10 minutes
  - Check log files created
  - Verify performance impact minimal
  
- [ ] **Configuration Validation**
  - Create invalid JSON config
  - Launch ACAT
  - Verify user-friendly error message
  - Verify fallback to defaults

### Acceptance Criteria
- ✅ All scenarios pass
- ✅ No regressions in existing features
- ✅ Log files created correctly
- ✅ JSON configs load correctly
- ✅ Migration tool works end-to-end
- ✅ Performance impact < 5%
- ✅ Test results documented

### Performance Validation
```csharp
[TestMethod]
public void LoggingPerformanceTest()
{
    var logger = GetLogger();
    var stopwatch = Stopwatch.StartNew();
    
    for (int i = 0; i < 10_000; i++)
    {
        logger.LogInformation("Test message {Index}", i);
    }
    
    stopwatch.Stop();
    
    // Should be < 100ms for 10K logs
    Assert.IsTrue(stopwatch.ElapsedMilliseconds < 100, 
        $"Logging too slow: {stopwatch.ElapsedMilliseconds}ms");
}

Dependencies

  • All Phase 1 tickets complete

🎫 Ticket #12: Phase 1 Documentation & Handoff

Labels: phase-1-foundation, docs, P1-High
Estimate: 0.5 days
Assignee: [Developer]
Sprint: Week 4

Description

Final documentation, knowledge transfer, and preparation for Phase 2.

Tasks

  • Update ACAT_MODERNIZATION_PLAN.md with Phase 1 completion
  • Document lessons learned
  • Create Phase 1 retrospective
  • Generate metrics report
  • Prepare Phase 2 kickoff materials

Phase 1 Metrics Report

# Phase 1 Completion Report

## Objectives Met
- ✅ Logging modernized (3,891 calls converted)
- ✅ JSON configuration system implemented
- ✅ Migration tool created
- ✅ All tests passing

## Metrics
- Files modified: [count]
- Lines of code changed: [count]
- Tests added: [count]
- AI acceleration: 2.3x (estimate vs actual)

## Performance Impact
- Startup time: [before][after]
- Logging overhead: [before][after]
- Configuration load time: [before][after]

## Lessons Learned
[Document what worked well, what didn't]

## Risks for Phase 2
[Any issues discovered during Phase 1]

## Recommendations
[Suggestions for Phase 2]

Deliverables

  • ✅ Completion report
  • ✅ Updated modernization plan
  • ✅ Retrospective notes
  • ✅ Phase 2 kickoff deck
  • ✅ Knowledge transfer session scheduled

Dependencies

  • All Phase 1 tickets complete

📊 Phase 1 Summary

Timeline

Dependencies Graph

#1 (Logging Setup)
  ├─→ #2 (Log Migration)
  │    └─→ #3 (DI Entry Points)
  │         └─→ #4 (Tests)
  │              └─→ #5 (Cleanup)
  │
#6 (XML Analysis)
  └─→ #7 (JSON Schemas)
       └─→ #8 (Migration Tool)
            └─→ #9 (JSON Loading)
                 └─→ #10 (Documentation)
                      └─→ #11 (Integration Tests)
                           └─→ #12 (Handoff)

Success Criteria

  • ✅ All logging uses ILogger
  • ✅ All config files migrated to JSON
  • ✅ No XML parsing for configuration
  • ✅ Structured logging working
  • ✅ Migration tool available for users
  • ✅ Documentation complete
  • ✅ All tests passing
  • ✅ Ready for Phase 2

🚀 Getting Started with Phase 1

Day 1 Checklist

  • Create GitHub milestone: "Phase 1: Foundation"
  • Create all 12 issues from this document
  • Assign to developers
  • Set up project board
  • Schedule kickoff meeting
  • Review AI prompts with team
  • Set up development environment

Team Assignments (Example)

Weekly Standups

  • Monday: Plan week, assign tickets
  • Wednesday: Mid-week check-in, unblock issues
  • Friday: Demo completed work, retrospective

End of Phase 1 Tickets

Ready to copy/paste into GitHub Issues or Azure DevOps!

Sub-issues

Metadata

Metadata

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions