feat(schema): add migration system for schema evolution (#519)#1209
feat(schema): add migration system for schema evolution (#519)#1209pegasus1134 wants to merge 1 commit intozio:mainfrom
Conversation
|
|
| def size: Int = actions.size | ||
| } | ||
|
|
||
| object DynamicMigration { |
There was a problem hiding this comment.
Imperative while loop here instead of foldLeft to avoid allocating intermediate Either values on every iteration. Since migrations can have many actions, this keeps the hot path allocation-free when all actions succeed.
| def execute(action: MigrationAction, value: DynamicValue): Either[SchemaError, DynamicValue] = | ||
| action match { | ||
| case a: AddField => executeAddField(a, value) | ||
| case a: DropField => executeDropField(a, value) |
There was a problem hiding this comment.
MaxPathDepth of 64 guards against circular or deeply nested optic paths. Realistically no schema has 64 levels of nesting, but this prevents stack issues if someone accidentally constructs a cycle.
| action: ApplyMigration, | ||
| value: DynamicValue | ||
| ): Either[SchemaError, DynamicValue] = | ||
| modifyAt(action.at, value, action.at) { nestedValue => |
There was a problem hiding this comment.
TransformElements, TransformKeys, and TransformValues all evaluate the expression per-element inside the foldLeft. This is intentional: the expression may reference the element itself (e.g. a Select on the element's fields), so it must be re-evaluated for each element rather than evaluated once against the parent value.
| val parentPath = DynamicOptic(at.nodes.dropRight(1)) | ||
| Rename(parentPath.field(to), from) | ||
| } | ||
|
|
There was a problem hiding this comment.
Rename.reverse rebuilds the path by replacing the last Field node. This means renaming "name" to "fullName" and then reversing gives a Rename from "fullName" back to "name" at the correct parent path.
|
|
||
| final case class TransformValue(at: DynamicOptic, transform: DynamicSchemaExpr) extends MigrationAction { | ||
| override def reverse: MigrationAction = Irreversible(at, "TransformValue") | ||
| } |
There was a problem hiding this comment.
TransformValue, Join, Split, ChangeType, and the collection transforms all reverse to Irreversible. These are lossy operations where the original value cannot be recovered from the output alone. The issue spec calls this "best-effort" semantic inverse.
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at |
There was a problem hiding this comment.
literalA(implicit Schema[A]) converts typed Scala values to SchemaExpr eagerly, so users write literal[Int](0) instead of manually constructing DynamicValue.Primitive(PrimitiveValue.Int(0)). This matches the API shape requested in the issue review.
| * limitations under the License. | ||
| */ | ||
|
|
||
| package zio.blocks.schema.migration |
There was a problem hiding this comment.
Runtime validation instead of phantom type-level tracking. Uses schema.reflect.asRecord to extract field names from the actual Schema instances and checks that the migration actions fully bridge source to target. This keeps the builder API simple (no 4+ type parameter phantom types) while still catching missing/invalid field operations at .build time.
| case class FieldNode(name: String) extends PathNode | ||
| case object ElementsNode extends PathNode | ||
| case object MapKeysNode extends PathNode | ||
| case object MapValuesNode extends PathNode |
There was a problem hiding this comment.
The macro walks the Term AST recursively, building up path nodes from the leaf back to the root. Pattern matches cover field access, .each, .when[T], .wrapped[T], .at(i), .atKey(k), .eachKey, .eachValue, and selectDynamic for structural types. Unsupported expressions fail at compile time with a clear error.
| sourceSchema, | ||
| targetSchema, | ||
| actions :+ MigrationAction.ChangeType(fromPath, converter.toDynamic) | ||
| ) |
There was a problem hiding this comment.
changeFieldType and transformField both take a target selector. The target selector exists to verify the field is present on the target type at compile time (the macro parses it). Only the source path is stored in the action since both refer to the same field position. The @unused annotation suppresses the warning.
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. |
There was a problem hiding this comment.
DynamicSchemaExpr is the serializable expression ADT. No closures, no functions, just case classes. This is what makes the migration system fully introspectable and suitable for DDL generation, registry storage, etc. Operations are limited to primitives (arithmetic, relational, logical, string concat/length, type conversions).
Path-based migration system for evolving schemas across versions. Cross-compiled for Scala 2.13 and 3.5+. - DynamicMigration: serializable migration operating on DynamicValue - Migration[A, B]: typed wrapper with schema-driven encode/decode - MigrationAction: 16-variant sealed trait with structural reverse - DynamicSchemaExpr: expression ADT for transforms and conversions - MigrationBuilder: macro-validated selectors for both Scala versions - MigrationValidation: build-time field coverage checking - MigrationSelectorSyntax: .each, .when[T], .atKey, .eachKey, .eachValue - 191 tests across 6 spec files, 0 regressions
pegasus1134
force-pushed
the
feat/schema-migration-system-519
branch
from
604344c to
e68ccff
Compare
|
@jdegoes Why? |
3 participants
/claim #519
What this does
Implements the schema migration system specified in #519. Two-layer design: a serializable
DynamicMigrationcore that operates onDynamicValue, and a typedMigration[A, B]wrapper with macro-validated selectors for the user-facing API.Architecture
DynamicMigrationholds aVector[MigrationAction]and applies them sequentially toDynamicValuetreesMigration[A, B]wraps source/target schemas and delegates toDynamicMigrationunder the hoodMigrationBuilder[A, B]provides the fluent API with selector macros that extractDynamicOpticpaths at compile timeDynamicSchemaExpris the expression ADT for value transforms (no closures, fully serializable)MigrationValidationchecks at build time that all source fields are accounted for in the target schemaMigrationAction ADT (16 cases)
Record: AddField, DropField, Rename, TransformValue, Mandate, Optionalize, Join, Split, ChangeType
Enum: RenameCase, TransformCase
Collection: TransformElements, TransformKeys, TransformValues
Meta: ApplyMigration, Irreversible
Every action carries a
DynamicOpticpath and implementsreverse.Selector macros
Scala 3 uses
inline+scala.quotedto parse selector lambdas intoDynamicOpticpaths. Scala 2 useswhitebox.Contextmacros with quasiquotes. Both support field access,.when[T]case selection,.eachcollection traversal,.atKey(k)map access, and.wrapped[T].Literal API
literal[A](value)(implicit Schema[A])converts typed values toDynamicSchemaExpreagerly. No rawDynamicValue.Primitive(...)construction in user code.What changed
20 new files, 3,722 lines. No existing files modified.
Shared (6 files): DynamicMigration, DynamicSchemaExpr, Migration, MigrationAction, MigrationValidation, package object
Scala 3 (4 files): MigrationBuilder, SelectorMacros, MigrationSelectorSyntax, MigrationCompanionVersionSpecific
Scala 2 (4 files): Same as Scala 3 with equivalent macro implementations
Tests (6 files): DynamicMigrationSpec, DynamicSchemaExprSpec, MigrationActionSpec, MigrationSpec, MigrationValidationSpec, MigrationIntegrationSpec
Tests
191 migration tests across 6 test suites. All pass on both Scala 2.13.18 and 3.7.4. Zero regressions across the full test suite (15,909 total tests).
Success criteria from #519