[AUDIT][SPARK-51290][SQL] Enable filling default values in DSv2 writes

[cindyyuanjiang]

apache/spark@e17df6e452a supports filling default values in DSv2 writes.

We should check if this will affect the plugin.

[sameerz]

We need to confirm if we fall back to the CPU when a default value is provided. If not, we need to handle this case.

[res-life]

Should be a P1 issue. The Spark-Rapids currently falls back to CPU.
And this should be a low priority if no customer is using default value to insert a column.

Fallback code, in GpuParquetScan.scala

    if (ColumnDefaultValuesShims.hasExistenceDefaultValues(readSchema)) {
      meta.willNotWorkOnGpu("GpuParquetScan does not support default values in schema")
    }

Also in CSV/ORC/json scan, it falls back.

Depends on:

• [FEA] Add support for Spark 4.1.1 [databricks] #14120

In future, please make the following test case to pass:

@allow_non_gpu("ColumnarToRowExec", "FileSourceScanExec")
@pytest.mark.parametrize("file_format", ["parquet", "orc"])
def test_add_column_112(spark_tmp_table_factory, file_format):
    before_gens = [("a", LongGen(min_val=-1, max_val=1,
                                 nullable=not is_databricks_runtime() or is_databricks113_or_later())),
                   ("b", SetValuesGen(StringType(), ["x", "y", "z"])),
                   ("c", long_gen)]
    def testf(spark):
        table_name = spark_tmp_table_factory.get()
        df = gen_df(spark, before_gens, length=16)
        df.write \
            .format(file_format) \
            .partitionBy("a", "b") \
            .saveAsTable(table_name)
        # Add a a1 column with setting default value.
        spark.sql(f"ALTER TABLE {table_name} ADD COLUMN a1 STRING DEFAULT 'initial-text'")
        ret = spark.sql(f"select * from {table_name}").collect()
        print(ret)
    with_gpu_session(testf, conf=_confs)

And add to schema_evolution_test.py

The metadata of a1 column is:
{
"EXISTS_DEFAULT" : "'initial-text'",
"CURRENT_DEFAULT" : "'initial-text'"
}
And EXISTS_DEFAULT config sets hasExistenceDefaultValues to true.

[res-life]

Changed to P1 priority.