[FSTORE-1897] DELTA materialization Jobs fail if the feature group contains a struct

[manu-sj]

Issue:
Spark fails with org.apache.spark.sql.avro.IncompatibleSchemaException: Attempting to treat union as a RECORD, but it was: UNION when trying to deserialize an avro schema of the format '["null", {"type": "record", "name": "S_col_object_dict_", "fields": [{"name": "k0", "type": ["null", "long"]}, {"name": "k1", "type": ["null", "long"]}]}]'

Root Cause:
It seems that spark always expects the top level schema (jsonFormatSchema) provided to the from_avro function to be a record if it infers the type as a Struct Type.
From looking at the code it seem that spark always considers a union of null and a records as a Struct Type, and for struct Types spark uses the getRecordWriter function which expects the passed avro type to be a record and throws the exception.

Fix Done
Wrap the the union of struct into a record so that spark can deserialize it. This would result in a deseriialized struct of a struct which is then unnested to get the actual struct.

JIRA Issue: https://hopsworks.atlassian.net/browse/FSTORE-1897

Priority for Review: -

Related PRs: -

How Has This Been Tested?

• Unit Tests
• Integration Tests
• Manual Tests on VM

Checklist For The Assigned Reviewer:

- [ ] Checked if merge conflicts with master exist
- [ ] Checked if stylechecks for Java and Python pass
- [ ] Checked if all docstrings were added and/or updated appropriately
- [ ] Ran spellcheck on docstring
- [ ] Checked if guides & concepts need to be updated
- [ ] Checked if naming conventions for parameters and variables were followed
- [ ] Checked if private methods are properly declared and used
- [ ] Checked if hard-to-understand areas of code are commented
- [ ] Checked if tests are effective
- [ ] Built and deployed changes on dev VM and tested manually
- [x] (Checked if all type annotations were added and/or updated appropriately)