[SPARK-53158][WEBUI] Missing metricsProperties in KV Store should be handled correctly

[thejdeep]

[SPARK-53158][WEBUI] Missing metricsProperties in KV Store should be handled correctly

What changes were proposed in this pull request?

When environment property maps (e.g. metricsProperties) are null — often due to LevelDBs created by Spark versions that did not have the property, the Utils.redact(...) function throws a NullPointerException during .map or .toArray operations.

Why are the changes needed?

Without this change, the rest API response of /environment throws a 500 due to the server NullPointerException. The NullPointerException is the following :

Caused by: java.lang.NullPointerException: Cannot invoke "scala.collection.Seq.map(scala.Function1, scala.collection.generic.CanBuildFrom)" because "kvs" is
 null
        at org.apache.spark.util.Utils$.redact(Utils.scala:2923)
        at org.apache.spark.util.Utils$.redact(Utils.scala:2878)
        at org.apache.spark.status.api.v1.AbstractApplicationResource.$anonfun$environmentInfo$1(OneApplicationResource.scala:114)

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Tested this change on a local instance of SHS

Was this patch authored or co-authored using generative AI tooling?

No

[mridulm]

Thanks for the bug fix.

Looks like metricsProperties got added in #36462, and for 'older' event files, this would end up becoming null which would have broken it for history server at that time - will setting it to a default value of Nil remove the issue observed ?

Note that Apache Spark does not support 3.4 anymore - so changes specifically to make master work with older/unsupported versions (and their event files), unless it is an ongoing issue, is unlikely to get merged.

+CC @dongjoon-hyun , @LuciferYang as well for thoughts.

[dongjoon-hyun]

The proposal looks good to me, @thejdeep and @mridulm .

[thejdeep]

@mridulm Current master is compatible (w.r.t metricsProperties) with older versions of Spark since the KVStore DB is generated correctly. However, current master is incompatible with KVStore's generated with older versions of Spark.

This can be an issue, if a user decides to upgrade their Spark version but does not purge the SHS local store directory.

[mridulm]

Can you add a test to validate this works ?
As in, json record without the field - and yet gets deserialized with Nil and not null for ApplicationEnvironmentInfo ?