[GLUTEN-11550][UT] Replace GlutenTestsCommonTrait with correct Gluten test traits for Spark 4.0/4.1#11800
Open
baibaichen wants to merge 3 commits intoapache:mainfrom
Open
Conversation
baibaichen
force-pushed
the
fix/gluten-tests-common-trait-spark40-41
branch
from
e613866 to
70962e0
Compare
…SQLTestsBaseTrait for SQL tests Replace GlutenTestsCommonTrait with the correct SQL test trait for 105 test suites in spark40 and spark41 whose Spark parent classes extend QueryTest or SharedSparkSession. This ensures these tests run with GlutenPlugin enabled. - 80 suites -> GlutenSQLTestsTrait (with checkAnswer override) - 25 suites -> GlutenSQLTestsBaseTrait (fallback for inheritance conflicts) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ssion and unit tests Replace GlutenTestsCommonTrait with GlutenTestsTrait (shim.GlutenTestsTrait for spark41) for 73 test suites whose Spark parent classes use ExpressionEvalHelper or are pure SparkFunSuite. This ensures: - Expression tests run checkEvaluation() through Gluten native engine - Unit tests validate GlutenPlugin compatibility via a Gluten-enabled SparkSession Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
fix/gluten-tests-common-trait-spark40-41
branch
3 times, most recently
from
2d0789f to
83008f4
Compare
…enPlugin-enabled traits These suites fail when running with GlutenPlugin loaded (via the new traits). Failures include plan stability mismatches, spark.test.home requirements, and Gluten-incompatible test assumptions. Commented out with TODO: 4.x markers for future investigation. Categories: - 10 plan stability suites (golden file mismatch with Gluten execution plans) - 3 spark.test.home required suites - 27 other suites with various Gluten compatibility issues Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
fix/gluten-tests-common-trait-spark40-41
branch
from
83008f4 to
8dd932b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
GlutenTestsCommonTraitdoes not configureGlutenPluginor create a Gluten-enabledSparkSession. Test suites using this trait were running on vanilla Spark, falsely reporting "offload to gluten".This PR replaces
GlutenTestsCommonTraitwith the correct traits in 157 test files (spark40 + spark41):GlutenSQLTestsTrait/GlutenSQLTestsBaseTraitsparkConfoverrideGlutenTestsTrait(spark41:shim.GlutenTestsTrait)initializeSession()GlutenTestsCommonTraitGlutenTestsCommonTrait47 suites (spark41) / 42 suites (spark40) are disabled with
// TODO: 4.xmarkers after failing under the correct traits. 25 of these are newly discovered failures; the rest were already tracked in #11550.How was this patch tested?
test-compilewith mvnd)run-scala-test.sh --mvnd— all passedNew disabled suites discovered by this PR
These suites previously appeared to pass because they were not loading GlutenPlugin:
catalyst.expressions
GlutenCastWithAnsiOnSuiteGlutenCollationRegexpExpressionsSuiteGlutenExpressionEvalHelperSuiteGlutenObjectExpressionsSuiteGlutenOrderingSuiteGlutenScalaUDFSuiteGlutenToPrettyStringSuiteexecution
GlutenCoalesceShufflePartitionsSuiteGlutenColumnarRulesSuiteGlutenDataSourceScanExecRedactionSuiteGlutenDataSourceV2ScanExecRedactionSuiteGlutenExternalAppendOnlyUnsafeRowArraySuiteGlutenHiveResultSuiteGlutenLogicalPlanTagInSparkPlanSuiteGlutenProjectedOrderingAndPartitioningSuiteGlutenRemoveRedundantProjectsSuiteGlutenSQLExecutionSuiteGlutenSQLJsonProtocolSuiteGlutenShufflePartitionsUtilSuiteGlutenSimpleSQLViewSuiteGlutenUnsafeRowSerializerSuiteexecution.datasources.parquet
GlutenParquetCommitterSuiteorg.apache.spark.sql
GlutenJoinHintSuiteGlutenRandomDataGeneratorSuitestreaming
GlutenTransformWithStateClusterSuite