[VL] Parquet writer marks all struct fields as OPTIONAL, breaking Spark variant type

[Zouxxyy]

Backend

VL (Velox)

Bug description

Description

Velox parquet writer unconditionally sets all struct fields as OPTIONAL (nullable), even when Spark schema specifies REQUIRED. This breaks Spark's variant binary format validation.

enableSuite[GlutenVariantSuite]
  // TODO: Velox parquet writer marks all struct fields as OPTIONAL (nullable),
  //  but Spark's variant type requires REQUIRED fields. Needs Velox-side fix.
  .exclude("SPARK-47546: invalid variant binary")
  .exclude("SPARK-47546: valid variant binary")

Root cause:

RowType has no nullability → exportToArrow() sets ARROW_FLAG_NULLABLE on all fields → Arrow Parquet Writer writes OPTIONAL → Spark convertVariantField expects REQUIRED → error.

Affected tests

SPARK-47546: valid variant binary
SPARK-47546: invalid variant binary

Gluten version

No response

Spark version

4.0, 4.1

Spark configurations

No response

System information

No response

Relevant logs

[Zouxxyy]

#11088