From 09bcb54d83bf18ea902cd730b4b268dcbd2198d5 Mon Sep 17 00:00:00 2001 From: tarekelgindy Date: Mon, 5 Jan 2026 17:27:36 -0700 Subject: [PATCH] Adding more thorough documentation --- API.md | 472 ++++++++++++++++++++++++++++++++++++++++++++++++ ARCHITECTURE.md | 386 +++++++++++++++++++++++++++++++++++++++ CONTRIBUTING.md | 437 ++++++++++++++++++++++++++++++++++++++++++++ README.md | 199 +++++++++++++++++--- 4 files changed, 1470 insertions(+), 24 deletions(-) create mode 100644 API.md create mode 100644 ARCHITECTURE.md create mode 100644 CONTRIBUTING.md diff --git a/API.md b/API.md new file mode 100644 index 0000000..c310b5d --- /dev/null +++ b/API.md @@ -0,0 +1,472 @@ +# DiTTo API Reference + +This document provides detailed API documentation for DiTTo's readers and writers. + +## Table of Contents + +- [Readers](#readers) + - [OpenDSS Reader](#opendss-reader) + - [CIM/IEC 61968-13 Reader](#cimiec-61968-13-reader) +- [Writers](#writers) + - [OpenDSS Writer](#opendss-writer) +- [GDM DistributionSystem](#gdm-distributionsystem) +- [Utilities](#utilities) + +--- + +## Readers + +### OpenDSS Reader + +**Module**: `ditto.readers.opendss.reader` + +The OpenDSS reader parses OpenDSS model files and creates a GDM DistributionSystem. + +#### Class: `Reader` + +```python +from ditto.readers.opendss.reader import Reader +``` + +##### Constructor + +```python +Reader( + master_file: Path, + crs: str | None = None +) +``` + +**Parameters**: +| Parameter | Type | Description | +|-----------|------|-------------| +| `master_file` | `Path` | Path to the OpenDSS master file (.dss) | +| `crs` | `str \| None` | Coordinate Reference System for bus coordinates (optional) | + +##### Methods + +###### `get_system() -> DistributionSystem` + +Parses the OpenDSS model and returns a populated DistributionSystem. + +```python +from pathlib import Path +from ditto.readers.opendss.reader import Reader + +reader = Reader(Path("IEEE13NODE.dss")) +system = reader.get_system() +``` + +**Returns**: `DistributionSystem` - The populated distribution system model + +**Raises**: +- `FileNotFoundError` - If the master file doesn't exist +- `ValidationError` - If the model contains invalid components + +#### Supported OpenDSS Elements + +| Element Type | GDM Component | Notes | +|--------------|---------------|-------| +| `Bus` | `DistributionBus` | Includes coordinates if available | +| `Line` | `DistributionBranch` | With line geometry or matrix impedance | +| `Transformer` | `DistributionTransformer` | Multi-winding support | +| `Load` | `DistributionLoad` | All load models supported | +| `Capacitor` | `DistributionCapacitor` | Shunt capacitors | +| `PVSystem` | `DistributionSolar` | Solar PV systems | +| `Storage` | `DistributionStorage` | Battery storage | +| `Vsource` | `DistributionVoltageSource` | Voltage sources | +| `Fuse` | `MatrixImpedanceFuse` | Fuse elements | +| `RegControl` | `DistributionRegulatorController` | Regulator controls | +| `LoadShape` | `LoadProfile` | Time-series data | + +#### Example Usage + +```python +from pathlib import Path +from ditto.readers.opendss.reader import Reader + +# Basic usage +reader = Reader(Path("path/to/model.dss")) +system = reader.get_system() + +# With coordinate reference system +reader = Reader( + Path("path/to/model.dss"), + crs="EPSG:4326" +) +system = reader.get_system() + +# Access components +for bus in system.get_buses(): + print(f"Bus: {bus.name}") + +for load in system.get_loads(): + print(f"Load: {load.name}, kW: {load.kw}") +``` + +--- + +### CIM/IEC 61968-13 Reader + +**Module**: `ditto.readers.cim_iec_61968_13.reader` + +The CIM reader parses CIM/IEC 61968-13 XML files using RDF graph queries. + +#### Class: `Reader` + +```python +from ditto.readers.cim_iec_61968_13.reader import Reader +``` + +##### Constructor + +```python +Reader( + cim_file: str | Path +) +``` + +**Parameters**: +| Parameter | Type | Description | +|-----------|------|-------------| +| `cim_file` | `str \| Path` | Path to the CIM XML file | + +##### Methods + +###### `read() -> None` + +Parses the CIM XML file and populates the internal DistributionSystem. + +```python +reader = Reader("ieee13_cim.xml") +reader.read() +``` + +###### `get_system() -> DistributionSystem` + +Returns the populated DistributionSystem. + +```python +system = reader.get_system() +``` + +**Returns**: `DistributionSystem` - The populated distribution system model + +#### Supported CIM Elements + +| CIM Element | GDM Component | +|-------------|---------------| +| `TopologicalNode` | `DistributionBus` | +| `PowerTransformer` | `DistributionTransformer` | +| `ACLineSegment` | `DistributionBranch` | +| `EnergyConsumer` | `DistributionLoad` | +| `LinearShuntCompensator` | `DistributionCapacitor` | +| `SynchronousMachine` | `DistributionVoltageSource` | +| `LoadBreakSwitch` | `MatrixImpedanceSwitch` | +| `RatioTapChanger` | `DistributionRegulatorController` | + +#### Example Usage + +```python +from pathlib import Path +from ditto.readers.cim_iec_61968_13.reader import Reader + +# Read CIM file +reader = Reader("path/to/ieee13_cim.xml") +reader.read() +system = reader.get_system() + +# Process components +print(f"System name: {system.name}") +print(f"Number of buses: {len(list(system.get_buses()))}") +``` + +--- + +## Writers + +### OpenDSS Writer + +**Module**: `ditto.writers.opendss.write` + +The OpenDSS writer exports a DistributionSystem to OpenDSS format files. + +#### Class: `Writer` + +```python +from ditto.writers.opendss.write import Writer +``` + +##### Constructor + +```python +Writer( + system: DistributionSystem +) +``` + +**Parameters**: +| Parameter | Type | Description | +|-----------|------|-------------| +| `system` | `DistributionSystem` | The distribution system to export | + +##### Methods + +###### `write(output_path, separate_substations, separate_feeders) -> None` + +Writes the distribution system to OpenDSS files. + +```python +writer = Writer(system) +writer.write( + output_path=Path("./output"), + separate_substations=True, + separate_feeders=False +) +``` + +**Parameters**: +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `output_path` | `Path` | Required | Directory for output files | +| `separate_substations` | `bool` | `False` | Organize by substation | +| `separate_feeders` | `bool` | `False` | Organize by feeder | + +#### Output File Structure + +When `separate_substations=False` and `separate_feeders=False`: +``` +output/ +├── Master.dss +├── Buses/ +│ └── BusCoords.dss +├── Lines.dss +├── LineCodes.dss +├── Transformers.dss +├── Loads.dss +├── Capacitors.dss +├── Regulators.dss +├── RegControllers.dss +├── Solar.dss +├── Switches.dss +├── SwitchCodes.dss +├── Fuses.dss +├── FuseCodes.dss +├── LoadShapes.dss +├── WireData.dss +└── LineGeometry.dss +``` + +When `separate_substations=True`: +``` +output/ +├── Substation_1/ +│ ├── Master.dss +│ ├── ... +├── Substation_2/ +│ ├── Master.dss +│ ├── ... +``` + +#### Supported Components + +| GDM Component | OpenDSS Element | +|---------------|-----------------| +| `DistributionBus` | Bus coordinates | +| `DistributionBranch` | `Line` | +| `DistributionTransformer` | `Transformer` | +| `DistributionLoad` | `Load` | +| `DistributionCapacitor` | `Capacitor` | +| `DistributionSolar` | `PVSystem` | +| `DistributionVoltageSource` | `Vsource` | +| `DistributionRegulator` | `Transformer` + `RegControl` | +| `MatrixImpedanceBranch` | `Line` with `LineCode` | +| `MatrixImpedanceSwitch` | `Line` (switch mode) | +| `MatrixImpedanceFuse` | `Fuse` | +| `GeometryBranch` | `Line` with `LineGeometry` | +| `SequenceImpedanceBranch` | `Line` with sequence model | + +#### Example Usage + +```python +from pathlib import Path +from ditto.readers.opendss.reader import Reader +from ditto.writers.opendss.write import Writer + +# Read a model +reader = Reader(Path("IEEE13NODE.dss")) +system = reader.get_system() + +# Modify the system if needed +# ... + +# Write to new location +writer = Writer(system) +writer.write( + output_path=Path("./modified_model"), + separate_substations=False, + separate_feeders=False +) +``` + +--- + +## GDM DistributionSystem + +The `DistributionSystem` class from Grid-Data-Models is the central data container in DiTTo. + +### Import + +```python +from gdm import DistributionSystem +``` + +### Key Methods + +#### Serialization + +```python +# Save to JSON +system.to_json(Path("model.json"), overwrite=True) + +# Load from JSON +system = DistributionSystem.from_json(Path("model.json")) +``` + +#### Component Access + +```python +# Get all components of a type +buses = system.get_buses() +loads = system.get_loads() +transformers = system.get_transformers() +branches = system.get_branches() +capacitors = system.get_capacitors() + +# Get component by name +bus = system.get_component_by_name("bus1") + +# Get all components +all_components = system.get_components() +``` + +#### Component Addition + +```python +from gdm import DistributionBus, DistributionLoad + +# Add a bus +bus = DistributionBus(name="new_bus", ...) +system.add_component(bus) + +# Add a load +load = DistributionLoad(name="new_load", bus=bus, ...) +system.add_component(load) +``` + +--- + +## Utilities + +### Phase Mapping + +**Module**: `ditto.readers.opendss.common.phase_mapper` + +Maps between OpenDSS phase notation and GDM Phase enum. + +```python +from ditto.readers.opendss.common.phase_mapper import map_phases + +# OpenDSS phases to GDM +phases = map_phases(".1.2.3") # Returns [Phase.A, Phase.B, Phase.C] +phases = map_phases(".1.2") # Returns [Phase.A, Phase.B] +phases = map_phases(".1") # Returns [Phase.A] +``` + +### Unit Conversion + +**Module**: `ditto.readers.opendss.common.units` + +Convert between different length units. + +```python +from ditto.readers.opendss.common.units import convert_length + +# Convert miles to meters +length_m = convert_length(1.0, "mi", "m") + +# Convert feet to kilometers +length_km = convert_length(5280, "ft", "km") +``` + +### OpenDSS File Types + +**Module**: `ditto.enumerations` + +Enumeration of standard OpenDSS file names. + +```python +from ditto.enumerations import OpenDSSFileTypes + +print(OpenDSSFileTypes.MASTER) # "Master.dss" +print(OpenDSSFileTypes.LINES) # "Lines.dss" +print(OpenDSSFileTypes.LOADS) # "Loads.dss" +``` + +--- + +## Error Handling + +DiTTo uses standard Python exceptions along with validation from Pydantic. + +### Common Exceptions + +```python +from pydantic import ValidationError + +try: + reader = Reader(Path("model.dss")) + system = reader.get_system() +except FileNotFoundError: + print("Model file not found") +except ValidationError as e: + print(f"Model validation failed: {e}") +except Exception as e: + print(f"Error reading model: {e}") +``` + +### Logging + +DiTTo uses `loguru` for logging: + +```python +from loguru import logger + +# Enable debug logging +logger.enable("ditto") + +# Disable logging +logger.disable("ditto") +``` + +--- + +## Type Hints + +DiTTo is fully typed. Use type hints for better IDE support: + +```python +from pathlib import Path +from gdm import DistributionSystem +from ditto.readers.opendss.reader import Reader +from ditto.writers.opendss.write import Writer + +def convert_model(input_path: Path, output_path: Path) -> DistributionSystem: + reader: Reader = Reader(input_path) + system: DistributionSystem = reader.get_system() + + writer: Writer = Writer(system) + writer.write(output_path) + + return system +``` diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..3711370 --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,386 @@ +# DiTTo Architecture Guide + +This document provides a detailed overview of DiTTo's architecture, design patterns, and component structure. + +## Table of Contents + +- [Overview](#overview) +- [Core Design: Many-to-One-to-Many](#core-design-many-to-one-to-many) +- [Component Architecture](#component-architecture) +- [Readers](#readers) +- [Writers](#writers) +- [Data Flow](#data-flow) +- [Extending DiTTo](#extending-ditto) + +## Overview + +DiTTo (Distribution Transformation Tool) is designed around a modular, extensible architecture that enables conversion between various distribution system model formats. The core principle is separation of concerns: reading, representation, and writing are handled by distinct, independent modules. + +## Core Design: Many-to-One-to-Many + +The fundamental architecture pattern in DiTTo is **many-to-one-to-many**: + +``` + ┌──────────────────────────────────────┐ + │ │ + ┌─────────────┐ │ GDM DistributionSystem │ ┌─────────────┐ + │ OpenDSS │──┐ │ (Intermediate Representation) │ ┌──│ OpenDSS │ + └─────────────┘ │ │ │ │ └─────────────┘ + ┌─────────────┐ │ │ ┌────────┐ ┌────────┐ ┌────────┐ │ │ ┌─────────────┐ + │ CIM/IEC │──┼──▶│ │ Bus │ │ Load │ │ Xfmr │ │──┼──▶│ JSON │ + └─────────────┘ │ │ └────────┘ └────────┘ └────────┘ │ │ └─────────────┘ + ┌─────────────┐ │ │ ┌────────┐ ┌────────┐ ┌────────┐ │ │ ┌─────────────┐ + │ CYME │──┘ │ │ PV │ │ Cap │ │ Branch │ │ └──│ CYME │ + └─────────────┘ │ └────────┘ └────────┘ └────────┘ │ └─────────────┘ + │ │ + READERS └──────────────────────────────────────┘ WRITERS +``` + +### Benefits of This Design + +1. **Modularity**: Each reader/writer is independent; adding a new format doesn't affect existing code +2. **Consistency**: All formats convert to the same intermediate representation +3. **Validation**: Validation logic is centralized in the GDM layer +4. **Extensibility**: New readers and writers can be added without modifying existing ones +5. **Maintainability**: Changes to one format's handling don't ripple through the system + +## Component Architecture + +### Directory Structure + +``` +src/ditto/ +├── __init__.py # Package initialization, version +├── enumerations.py # Shared enumerations (OpenDSSFileTypes) +│ +├── readers/ # Input format parsers +│ ├── reader.py # AbstractReader base class +│ ├── opendss/ # OpenDSS format reader +│ │ ├── reader.py # Main reader class +│ │ ├── common/ # Shared utilities +│ │ ├── components/ # Component parsers +│ │ ├── equipment/ # Equipment parsers +│ │ └── controllers/ # Controller parsers +│ ├── cim_iec_61968_13/ # CIM/IEC reader +│ │ ├── reader.py # Main reader class +│ │ ├── queries.py # SPARQL queries +│ │ ├── components/ # Component parsers +│ │ ├── equipment/ # Equipment parsers +│ │ └── controllers/ # Controller parsers +│ └── cyme/ # CYME reader (in progress) +│ +└── writers/ # Output format exporters + ├── abstract_writer.py # AbstractWriter base class + └── opendss/ # OpenDSS format writer + ├── write.py # Main writer class + ├── opendss_mapper.py # Mapping utilities + ├── components/ # Component mappers + ├── equipment/ # Equipment mappers + └── controllers/ # Controller mappers +``` + +### Grid-Data-Models (GDM) Integration + +DiTTo uses [Grid-Data-Models](https://github.com/NREL-Distribution-Suites/grid-data-models) as its intermediate representation. The core class is `DistributionSystem`, which serves as a container for all distribution system components. + +```python +from gdm import DistributionSystem + +# The DistributionSystem holds all components +system = DistributionSystem(name="my_system") + +# Components are accessed through typed getters +buses = system.get_buses() +loads = system.get_loads() +transformers = system.get_transformers() +``` + +## Readers + +Readers are responsible for parsing input files and populating a `DistributionSystem` object. + +### AbstractReader Base Class + +All readers inherit from `AbstractReader`: + +```python +from abc import ABC, abstractmethod +from gdm import DistributionSystem + +class AbstractReader(ABC): + """Base class for all DiTTo readers.""" + + def __init__(self): + self.system = DistributionSystem() + + @abstractmethod + def read(self) -> None: + """Parse input files and populate self.system.""" + pass + + def get_system(self) -> DistributionSystem: + """Return the populated distribution system.""" + return self.system +``` + +### OpenDSS Reader + +**Location**: `src/ditto/readers/opendss/` + +The OpenDSS reader uses `opendssdirect.py` to interface with the OpenDSS engine: + +```python +from ditto.readers.opendss.reader import Reader + +reader = Reader(Path("model.dss")) +system = reader.get_system() +``` + +**Architecture**: +- Uses OpenDSS's COM interface via `opendssdirect` +- Component parsers in `components/` handle specific element types +- Equipment parsers in `equipment/` handle equipment definitions +- Common utilities provide phase mapping, unit conversion + +**Component Modules**: +| Module | Purpose | +|--------|---------| +| `buses.py` | Parse bus/node data | +| `branches.py` | Parse line segments | +| `transformers.py` | Parse transformer data | +| `loads.py` | Parse load data | +| `capacitors.py` | Parse capacitor banks | +| `pv_systems.py` | Parse solar/PV systems | +| `storage.py` | Parse energy storage | +| `sources.py` | Parse voltage sources | +| `fuses.py` | Parse fuse elements | +| `loadshapes.py` | Parse time-series profiles | + +### CIM/IEC 61968-13 Reader + +**Location**: `src/ditto/readers/cim_iec_61968_13/` + +The CIM reader parses XML files using RDF graph queries: + +```python +from ditto.readers.cim_iec_61968_13.reader import Reader + +reader = Reader("model.xml") +reader.read() +system = reader.get_system() +``` + +**Architecture**: +- Uses `rdflib` for RDF/XML graph parsing +- SPARQL-like queries defined in `queries.py` +- Component mappers convert CIM objects to GDM components + +## Writers + +Writers export a `DistributionSystem` to a specific output format. + +### AbstractWriter Base Class + +```python +from abc import ABC, abstractmethod +from pathlib import Path +from gdm import DistributionSystem + +class AbstractWriter(ABC): + """Base class for all DiTTo writers.""" + + def __init__(self, system: DistributionSystem): + self.system = system + + @abstractmethod + def write(self, output_path: Path, **kwargs) -> None: + """Write the system to output files.""" + pass +``` + +### OpenDSS Writer + +**Location**: `src/ditto/writers/opendss/` + +The OpenDSS writer generates complete DSS model files: + +```python +from ditto.writers.opendss.write import Writer + +writer = Writer(system) +writer.write( + output_path=Path("./output"), + separate_substations=True, + separate_feeders=True +) +``` + +**Architecture**: +- Mapper pattern: Each GDM component type has a corresponding mapper +- Uses `altdss_schema` for DSS output validation +- Generates organized file structure + +**Output Files Generated**: +``` +output/ +├── Master.dss # Main entry point +├── Buses/ +│ └── BusCoords.dss # Bus coordinates +├── Lines.dss # Line definitions +├── LineCodes.dss # Line code specifications +├── Transformers.dss # Transformer definitions +├── Loads.dss # Load definitions +├── Capacitors.dss # Capacitor banks +├── Regulators.dss # Voltage regulators +├── RegControllers.dss # Regulator controllers +├── Solar.dss # PV systems +└── LoadShapes.dss # Time-series profiles +``` + +**Writer Options**: +| Option | Description | +|--------|-------------| +| `separate_substations` | Organize output by substation | +| `separate_feeders` | Organize output by feeder | + +## Data Flow + +### Complete Conversion Flow + +``` +1. INPUT FILE(S) + │ + ▼ +2. READER + ├── Parse input format + ├── Create GDM components + ├── Validate components + └── Populate DistributionSystem + │ + ▼ +3. GDM DistributionSystem (in memory) + ├── Buses + ├── Branches/Lines + ├── Transformers + ├── Loads + ├── Capacitors + ├── PV Systems + ├── Storage + ├── Controllers + └── Profiles + │ + ▼ +4. WRITER + ├── Query system components + ├── Map to output format + ├── Generate output files + └── Write to disk + │ + ▼ +5. OUTPUT FILE(S) +``` + +### Example: OpenDSS to JSON Conversion + +```python +from pathlib import Path +from ditto.readers.opendss.reader import Reader + +# Step 1-2: Read OpenDSS model +reader = Reader(Path("IEEE13NODE.dss")) + +# Step 3: Get GDM DistributionSystem +system = reader.get_system() + +# Step 4-5: Write to JSON (built into GDM) +system.to_json(Path("IEEE13NODE.json"), overwrite=True) +``` + +## Extending DiTTo + +### Adding a New Reader + +1. Create a new directory under `src/ditto/readers/` +2. Implement a `reader.py` with a class inheriting from `AbstractReader` +3. Implement component parsers as needed + +```python +# src/ditto/readers/myformat/reader.py +from ditto.readers.reader import AbstractReader +from gdm import DistributionSystem + +class Reader(AbstractReader): + def __init__(self, input_file): + super().__init__() + self.input_file = input_file + + def read(self): + # Parse your format and populate self.system + # Create GDM components and add to system + pass + + def get_system(self) -> DistributionSystem: + self.read() + return self.system +``` + +### Adding a New Writer + +1. Create a new directory under `src/ditto/writers/` +2. Implement a `write.py` with a class inheriting from `AbstractWriter` +3. Implement component mappers as needed + +```python +# src/ditto/writers/myformat/write.py +from pathlib import Path +from ditto.writers.abstract_writer import AbstractWriter +from gdm import DistributionSystem + +class Writer(AbstractWriter): + def __init__(self, system: DistributionSystem): + super().__init__(system) + + def write(self, output_path: Path, **kwargs): + # Convert GDM components to your format + # Write output files + pass +``` + +### Component Mapping Pattern + +When implementing readers/writers, use a consistent pattern for component handling: + +```python +# Reader pattern: Format-specific -> GDM +def parse_load(self, raw_data) -> Load: + return Load( + name=raw_data["name"], + bus=self.get_bus(raw_data["bus"]), + phases=self.map_phases(raw_data["phases"]), + kw=raw_data["kw"], + kvar=raw_data["kvar"] + ) + +# Writer pattern: GDM -> Format-specific +def write_load(self, load: Load) -> str: + return f"New Load.{load.name} bus={load.bus.name} kW={load.kw} kvar={load.kvar}" +``` + +## Key Design Decisions + +### Why GDM as Intermediate Format? + +1. **Standardization**: GDM provides a well-defined, validated component model +2. **Rich Type System**: Pydantic-based validation ensures data integrity +3. **Serialization**: Built-in JSON serialization/deserialization +4. **Extensibility**: GDM is actively maintained with new component types +5. **Shared Ecosystem**: Other NREL tools use GDM, enabling interoperability + +### Why Separate Component/Equipment/Controller Directories? + +This organization mirrors the logical structure of distribution systems: +- **Components**: Physical network elements (buses, branches, loads) +- **Equipment**: Equipment specifications (line codes, transformer ratings) +- **Controllers**: Control logic (regulator controllers, capacitor controllers) + +This separation makes it easier to find and maintain related code. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..c2b4926 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,437 @@ +# Contributing to DiTTo + +Thank you for your interest in contributing to DiTTo! This document provides guidelines and instructions for contributing. + +## Table of Contents + +- [Code of Conduct](#code-of-conduct) +- [Getting Started](#getting-started) +- [Development Setup](#development-setup) +- [Making Changes](#making-changes) +- [Code Style](#code-style) +- [Testing](#testing) +- [Submitting Changes](#submitting-changes) +- [Adding New Readers/Writers](#adding-new-readerswriters) + +## Code of Conduct + +Please be respectful and constructive in all interactions. We welcome contributors of all experience levels. + +## Getting Started + +### Types of Contributions + +We welcome many types of contributions: + +- **Bug fixes**: Found a bug? Submit a fix! +- **Documentation**: Improve or add documentation +- **New features**: Add new functionality +- **New readers/writers**: Add support for new distribution system formats +- **Tests**: Improve test coverage +- **Code quality**: Refactoring and code improvements + +### Finding Issues + +- Check the [GitHub Issues](https://github.com/NREL-Distribution-Suites/ditto/issues) for open issues +- Look for issues labeled `good first issue` for beginner-friendly tasks +- Feel free to ask questions on any issue + +## Development Setup + +### Prerequisites + +- Python 3.10, 3.11, or 3.12 +- Git +- A virtual environment tool (venv, conda, etc.) + +### Setup Steps + +1. **Fork the repository** + + Click the "Fork" button on the [DiTTo GitHub page](https://github.com/NREL-Distribution-Suites/ditto) + +2. **Clone your fork** + + ```bash + git clone https://github.com/YOUR_USERNAME/ditto.git + cd ditto + ``` + +3. **Create a virtual environment** + + ```bash + # Using venv + python -m venv venv + + # Activate on Windows + venv\Scripts\activate + + # Activate on macOS/Linux + source venv/bin/activate + ``` + +4. **Install development dependencies** + + ```bash + pip install -e ".[dev]" + ``` + +5. **Install pre-commit hooks** + + ```bash + pip install pre-commit + pre-commit install + ``` + +6. **Verify setup** + + ```bash + pytest + ``` + +## Making Changes + +### Branch Workflow + +1. **Create a feature branch** + + ```bash + git checkout -b feature/your-feature-name + ``` + + Use descriptive branch names: + - `feature/add-cyme-writer` + - `fix/transformer-phase-mapping` + - `docs/improve-api-reference` + +2. **Make your changes** + + - Write clear, focused commits + - Keep commits atomic (one logical change per commit) + +3. **Write tests** + + Add tests for new functionality or bug fixes + +4. **Run tests locally** + + ```bash + pytest + ``` + +5. **Run linting** + + ```bash + ruff check src/ + ruff format src/ + ``` + +### Commit Messages + +Write clear, descriptive commit messages: + +``` +Add CYME reader support for transformer elements + +- Implement transformer parsing from CYME equipment files +- Add phase mapping for CYME transformer configurations +- Include unit tests for transformer conversion +``` + +## Code Style + +DiTTo uses [Ruff](https://github.com/astral-sh/ruff) for linting and formatting. + +### Style Guidelines + +- **Line length**: 99 characters maximum +- **Indentation**: 4 spaces (no tabs) +- **Quotes**: Double quotes for strings +- **Type hints**: Use type hints for function parameters and return values + +### Running Ruff + +```bash +# Check for issues +ruff check src/ + +# Auto-fix issues +ruff check --fix src/ + +# Format code +ruff format src/ +``` + +### Code Example + +```python +from pathlib import Path +from typing import List, Optional + +from gdm import DistributionSystem, DistributionBus +from loguru import logger + + +def parse_buses( + input_file: Path, + crs: Optional[str] = None +) -> List[DistributionBus]: + """Parse bus data from input file. + + Args: + input_file: Path to the input file + crs: Coordinate reference system (optional) + + Returns: + List of parsed distribution buses + + Raises: + FileNotFoundError: If input file doesn't exist + """ + if not input_file.exists(): + raise FileNotFoundError(f"Input file not found: {input_file}") + + buses = [] + logger.info(f"Parsing buses from {input_file}") + + # Implementation... + + return buses +``` + +## Testing + +### Running Tests + +```bash +# Run all tests +pytest + +# Run with verbose output +pytest -v + +# Run specific test file +pytest tests/test_opendss/test_opendss_reader.py + +# Run specific test +pytest tests/test_opendss/test_opendss_reader.py::test_read_ieee13 + +# Run with coverage +pytest --cov=ditto +``` + +### Writing Tests + +Tests are located in the `tests/` directory. Follow existing patterns: + +```python +# tests/test_opendss/test_new_feature.py +import pytest +from pathlib import Path + +from ditto.readers.opendss.reader import Reader + + +class TestNewFeature: + """Tests for the new feature.""" + + def test_basic_functionality(self, tmp_path): + """Test that basic functionality works.""" + # Arrange + input_file = Path("tests/data/opendss_circuit_models/IEEE13/IEEE13Nodeckt.dss") + + # Act + reader = Reader(input_file) + system = reader.get_system() + + # Assert + assert system is not None + assert len(list(system.get_buses())) > 0 + + @pytest.mark.parametrize("model_name", ["IEEE13", "ckt7", "ckt24"]) + def test_multiple_models(self, model_name): + """Test feature works across multiple models.""" + input_file = Path(f"tests/data/opendss_circuit_models/{model_name}") + # Test implementation... +``` + +### Test Data + +Test data is located in `tests/data/`: +- `opendss_circuit_models/` - OpenDSS test models +- `cim_iec_61968_13/` - CIM test files +- `cyme_test_cases/` - CYME test models +- `GDM_testing/` - Serialized GDM models + +## Submitting Changes + +### Pull Request Process + +1. **Push your branch** + + ```bash + git push origin feature/your-feature-name + ``` + +2. **Create a Pull Request** + + - Go to the [DiTTo GitHub page](https://github.com/NREL-Distribution-Suites/ditto) + - Click "Pull requests" > "New pull request" + - Select your fork and branch + - Fill out the PR template + +3. **PR Description** + + Include: + - What changes were made + - Why the changes were made + - How to test the changes + - Any related issues (use `Fixes #123` to auto-close) + +4. **Code Review** + + - Address reviewer feedback + - Push additional commits as needed + - Keep the PR focused and reasonably sized + +### PR Checklist + +Before submitting: + +- [ ] Tests pass locally (`pytest`) +- [ ] Code is formatted (`ruff format src/`) +- [ ] Linting passes (`ruff check src/`) +- [ ] New code has tests +- [ ] Documentation is updated if needed +- [ ] Commit messages are clear + +## Adding New Readers/Writers + +### Adding a New Reader + +1. **Create directory structure** + + ``` + src/ditto/readers/myformat/ + ├── __init__.py + ├── reader.py + ├── common/ + │ └── __init__.py + ├── components/ + │ └── __init__.py + └── equipment/ + └── __init__.py + ``` + +2. **Implement the reader** + + ```python + # src/ditto/readers/myformat/reader.py + from pathlib import Path + from typing import Optional + + from gdm import DistributionSystem + from ditto.readers.reader import AbstractReader + + + class Reader(AbstractReader): + """Reader for MyFormat distribution system models.""" + + def __init__( + self, + input_file: Path, + crs: Optional[str] = None + ): + """Initialize the reader. + + Args: + input_file: Path to input file + crs: Coordinate reference system + """ + super().__init__() + self.input_file = input_file + self.crs = crs + + def read(self) -> None: + """Parse the input file and populate the system.""" + # Implementation... + pass + + def get_system(self) -> DistributionSystem: + """Return the populated distribution system.""" + self.read() + return self.system + ``` + +3. **Add component parsers** + + Create separate modules for each component type in `components/` + +4. **Add tests** + + ```python + # tests/test_myformat/test_myformat_reader.py + ``` + +5. **Add documentation** + + Update `API.md` with the new reader documentation + +### Adding a New Writer + +1. **Create directory structure** + + ``` + src/ditto/writers/myformat/ + ├── __init__.py + ├── write.py + ├── components/ + │ └── __init__.py + └── equipment/ + └── __init__.py + ``` + +2. **Implement the writer** + + ```python + # src/ditto/writers/myformat/write.py + from pathlib import Path + + from gdm import DistributionSystem + from ditto.writers.abstract_writer import AbstractWriter + + + class Writer(AbstractWriter): + """Writer for MyFormat distribution system models.""" + + def __init__(self, system: DistributionSystem): + """Initialize the writer. + + Args: + system: The distribution system to export + """ + super().__init__(system) + + def write(self, output_path: Path, **kwargs) -> None: + """Write the system to output files. + + Args: + output_path: Directory for output files + **kwargs: Additional options + """ + # Implementation... + pass + ``` + +3. **Add component mappers** + + Create mapper modules for each component type + +4. **Add tests and documentation** + +## Questions? + +- Open an issue on GitHub +- Contact [Tarek Elgindy](mailto:tarek.elgindy@nrel.gov) + +Thank you for contributing to DiTTo! diff --git a/README.md b/README.md index 59fe5ea..2253f33 100644 --- a/README.md +++ b/README.md @@ -1,57 +1,208 @@ +# DiTTo - Distribution Transformation Tool -# DiTTo +[![PyPI version](https://badge.fury.io/py/NREL-ditto.svg)](https://pypi.org/project/NREL-ditto/) +[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -DiTTo is the _Distribution Transformation Tool_. It is an open-source tool to convert and modify electrical distribution system models. The most common domain of electrical distribution systems is from substations to customers. +DiTTo is an open-source tool developed by NREL's Distribution Suites team for converting and modifying electrical distribution system models. It enables seamless conversion between different distribution network formats, with the primary domain being substations to customers. -## How it Works -Flexible representations for power system components are defined in [Grid-Data-Models (GDM)](https://github.com/NREL-Distribution-Suites/grid-data-models) format. -DiTTo implements a _many-to-one-to-many_ parsing framework, making it modular and robust. The [reader modules](https://github.com/NREL-Distribution-Suites/ditto/tree/main/src/ditto/readers) parse data files of distribution system format (e.g. OpenDSS) and create an object for each electrical component. These objects are stored in a [GDM DistributionSystem](https://github.com/NREL-Distribution-Suites/grid-data-models/blob/main/src/gdm/distribution/distribution_system.py) instance. The [writer modules](https://github.com/NREL-Distribution-Suites/ditto/tree/main/src/ditto/writers) are then used to export the data stored in memory to a selected output distribution system format (e.g. OpenDSS) which are written to disk. +## Key Features -Additional documentation is currently under development and will e made available soon. -## Quick Start +- **Multi-format Support**: Read and write models from OpenDSS, CIM/IEC 61968-13, and more +- **Robust Architecture**: Many-to-one-to-many parsing framework ensures modularity and extensibility +- **GDM Integration**: Built on [Grid-Data-Models (GDM)](https://github.com/NREL-Distribution-Suites/grid-data-models) for flexible power system component representation +- **Validation**: Thorough model validation during conversion +- **Serialization**: Full JSON serialization/deserialization support for converted models + +## How It Works + +DiTTo implements a **many-to-one-to-many** parsing framework: + +``` +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ +│ OpenDSS │ │ │ │ OpenDSS │ +├─────────────┤ │ GDM │ ├─────────────┤ +│ CIM/IEC │ ──▶ │ Distribution│ ──▶ │ CYME │ +├─────────────┤ │ System │ ├─────────────┤ +│ CYME │ │ │ │ JSON │ +└─────────────┘ └─────────────┘ └─────────────┘ + READERS INTERMEDIATE WRITERS +``` + +1. **Readers** parse distribution system files and create component objects +2. All components are stored in a **GDM DistributionSystem** instance (intermediate format) +3. **Writers** export the data to the desired output format -### Install DiTTo +## Installation + +### From PyPI (Recommended) + +```bash +pip install nrel-ditto +``` + +### From Source ```bash git clone https://github.com/NREL-Distribution-Suites/ditto.git +cd ditto +pip install -e . ``` -Navigate to the clone directory and use the following command to install +### Optional Dependencies ```bash -pip install -e. +# For documentation building +pip install nrel-ditto[doc] + +# For development (includes pytest, ruff) +pip install nrel-ditto[dev] ``` -### Basic Usage +## Quick Start + +### Reading an OpenDSS Model + +```python +from pathlib import Path +from ditto.readers.opendss.reader import Reader + +# Read an OpenDSS model +opendss_file = Path("path/to/IEEE13NODE.dss") +reader = Reader(opendss_file) +system = reader.get_system() + +# Access components +print(f"Loaded {len(list(system.get_buses()))} buses") +``` -The most basic capability of DiTTo is converting a distribution system from one format to another. -To convert a cyme model represented in ASCII format with network.txt, equipment.txt and load.txt files, the following python script can be run to perform the conversion +### Converting CIM to OpenDSS ```python +from pathlib import Path from ditto.readers.cim_iec_61968_13.reader import Reader from ditto.writers.opendss.write import Writer -cim_reader = Reader(ieee13_node_xml_file) +# Read CIM model +cim_reader = Reader("path/to/ieee13_cim.xml") cim_reader.read() system = cim_reader.get_system() + +# Write to OpenDSS format writer = Writer(system) -new_dss_file = Path(__file__).parent / "model" -writer.write(output_path=new_dss_file, separate_substations=False, separate_feeders=False) +writer.write( + output_path=Path("./output"), + separate_substations=False, + separate_feeders=False +) +``` + +### Serializing to JSON + +```python +from pathlib import Path +from ditto.readers.opendss.reader import Reader +# Read and serialize +reader = Reader(Path("IEEE13NODE.dss")) +system = reader.get_system() +system.to_json(Path("IEEE13NODE.json"), overwrite=True) ``` -The required input files for each reader format are defined in the folder of each reader +### Loading from JSON + +```python +from pathlib import Path +from gdm import DistributionSystem + +# Deserialize a saved model +system = DistributionSystem.from_json(Path("IEEE13NODE.json")) +``` + +## Supported Formats + +### Readers (Input) + +| Format | Status | Description | +|--------|--------|-------------| +| OpenDSS | ✅ Complete | Full support for OpenDSS models | +| CIM/IEC 61968-13 | ✅ Complete | Common Information Model support | +| CYME | 🚧 In Progress | CYME network models | + +### Writers (Output) + +| Format | Status | Description | +|--------|--------|-------------| +| OpenDSS | ✅ Complete | Full DSS file generation | +| JSON/GDM | ✅ Complete | Serialized GDM format | + +## Supported Components + +DiTTo handles a comprehensive set of distribution system components: + +- **Network**: Buses, Lines/Branches, Cables, Conductors +- **Transformers**: Distribution transformers with multiple windings +- **Loads**: Various load types (constant power, impedance, ZIP) +- **Generation**: PV systems, Voltage sources +- **Protection**: Fuses, Regulators with controllers +- **Storage**: Battery/energy storage systems +- **Capacitors**: Shunt capacitors +- **Time-Series**: Load shapes and profiles + +## Project Structure + +``` +ditto/ +├── src/ditto/ +│ ├── readers/ # Format parsers +│ │ ├── opendss/ # OpenDSS reader +│ │ ├── cim_iec_61968_13/ # CIM reader +│ │ └── cyme/ # CYME reader +│ ├── writers/ # Format exporters +│ │ └── opendss/ # OpenDSS writer +│ └── enumerations.py # Shared enumerations +├── tests/ # Test suite +├── docs/ # Documentation +└── pyproject.toml # Project configuration +``` + +## Documentation + +- [Architecture Guide](ARCHITECTURE.md) - System design and components +- [API Reference](API.md) - Reader and writer documentation +- [Examples](EXAMPLES.md) - Detailed usage examples +- [Contributing Guide](CONTRIBUTING.md) - How to contribute + +## Requirements + +- Python 3.10, 3.11, or 3.12 +- Dependencies are automatically installed: + - `grid-data-models` - GDM intermediate representation + - `opendssdirect.py` - OpenDSS interface + - `rdflib` - RDF/XML parsing for CIM + - `NREL-altdss-schema` - DSS output schema ## Contributing -DiTTo is an open-source project and contributions are welcome! Either for a simple typo, a bugfix, or a new parser you want to integrate, feel free to contribute. -To contribute to Ditto in 3 steps: -- Fork the repository (button in the upper right corner of the DiTTo GitHub page). -- Create a feature branch on your local fork and implement your modifications there. -- Once your work is ready to be shared, submit a Pull Request on the DiTTo GitHub page. See the official GitHub documentation on how to do that [here](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) +DiTTo is an open-source project and contributions are welcome! Whether it's a typo fix, bug report, or a new parser, we appreciate your help. + +1. Fork the repository +2. Create a feature branch (`git checkout -b feature/amazing-feature`) +3. Make your changes +4. Run tests (`pytest`) +5. Submit a Pull Request + +See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines. ## Getting Help -If you are having issues using DiTTo, feel free to open an Issue on GitHub [here](https://github.com/NREL/ditto/issues/new) +- **Issues**: [GitHub Issues](https://github.com/NREL-Distribution-Suites/ditto/issues) +- **Questions**: Contact [Tarek Elgindy](mailto:tarek.elgindy@nrel.gov) + +## License + +This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. + +## Acknowledgments -All contributions are welcome. For questions about collaboration please email [Tarek Elgindy](mailto:tarek.elgindy@nrel.gov) +DiTTo is developed and maintained by the [NREL Distribution Suites](https://github.com/NREL-Distribution-Suites) team at the National Renewable Energy Laboratory.