$expand tries to read much more data, than actually needed #37

guai · 2021-02-17T09:43:54Z

Here is the fix

… so no redundant data loaded from the DB

rzozmann · 2021-02-19T18:45:13Z

Do you have a more concrete example/scenario to explain, when it happens that too much data is read?!

guai · 2021-02-20T08:33:11Z

It happens when data for ManyToOne dependencies being loaded. There is nothing limiting the request so all the data from the dependant table being loaded.
When I'm requesting something like http://foo.bar/odata/ParentEntity?$top=3&$skip=0&$expand=* there is no need to request more than 3 rows for every ManyToOne attribute.
I've added WHERE parent_table.id IN (...) clause.

rzozmann · 2021-03-05T09:19:03Z

So sorry for the delay... but the changes are breaking the test suite, so we have to inspect what happens.. With the current state the PR will have side effects, we have to manage; means still more work...

guai · 2021-03-05T09:28:38Z

I didn't manage to make tests pass even before my changes.
I can look at what I broke if you give me some guides how to launch tests properly

rzozmann · 2021-03-15T06:38:30Z

I hope i will find also time for an deeper look. You can run the complete build + test suite via: clean verify -Duse-eclipselink -Ddisable.jetty=false -Dmaven.source.skip -Dmaven.javadoc.skip started in the root directory of local repository.

guai · 2021-03-16T14:32:31Z

In win 10 with oracle's java build 1.8.0_281-b09 and Apache Maven 3.6.3 the command you provided still does not pass against untouched branch origin/eXXcellent_adaptions from this repo. Can you describe me your env?

[INFO] Reactor Summary for OData-JPA-Adapter Root POM 0.52.1:
[INFO]
[INFO] OData-JPA-Adapter Root POM ......................... SUCCESS [  6.373 s]
[INFO] OData-JPA-Adapter implementation root .............. SUCCESS [  6.773 s]
[INFO] OData-JPA-Adapter data model configuration API ..... SUCCESS [ 13.651 s]
[INFO] OData-JPA-Adapter testmodel ........................ SUCCESS [  9.264 s]
[INFO] OData-JPA-Adapter metadata ......................... SUCCESS [ 22.433 s]
[INFO] OData-JPA-Adapter processor ........................ FAILURE [01:12 min]
[INFO] OData-JPA-Adapter addons ........................... SKIPPED
[INFO] OData-JPA-Adapter excelexport ...................... SKIPPED
[INFO] OData-JPA-Adapter Tooling Parent POM ............... SKIPPED
[INFO] OData-JPA-Adapter client api code generator for Java SKIPPED
[INFO] OData-JPA-Adapter examples ......................... SKIPPED
[INFO] OData-JPA-Adapter UI5 Client Example ............... SKIPPED
[INFO] OData-JPA-Adapter java-client-api-generator-example  SKIPPED
[INFO] OData-JPA-Adapter servlet-example .................. SKIPPED
[INFO] OData-JPA-Adapter coverage report module ........... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project odata-jpa-processor: There are test failures.
[ERROR]
[ERROR] Please refer to D:\olingo-jpa-processor-v4-exxcellent\jpa\odata-jpa-processor\target\surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was cmd.exe /X /C "C:\jdk\jre\bin\java -javaagent:D:\\apache-maven-3.6.3\\repository\\org\\jacoco\\org.jacoco.agent\\0.8.5\\org.jacoco.agent-0.8.5-runtime.jar=destfile=D:\\olingo-jpa-processor-v4-exxcellent\\jpa\\odata-jpa-processor\\target\\jacoco.exec,append=true,inclnolocationclasses=true org.apache.maven.surefire.booter.ForkedBooter C:\Users\guai\AppData\Local\Temp\surefire3902702455596062375 2021-03-16T17-07-16_904-jvmRun1 surefire2822251614539718370tmp surefire_34118265438165058300tmp"
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
[ERROR] org.apache.olingo.jpa.processor.core.query.TestJPACount
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was cmd.exe /X /C "C:\jdk\jre\bin\java -javaagent:D:\\apache-maven-3.6.3\\repository\\org\\jacoco\\org.jacoco.agent\\0.8.5\\org.jacoco.agent-0.8.5-runtime.jar=destfile=D:\\olingo-jpa-processor-v4-exxcellent\\jpa\\odata-jpa-processor\\target\\jacoco.exec,append=true,inclnolocationclasses=true org.apache.maven.surefire.booter.ForkedBooter C:\Users\guai\AppData\Local\Temp\surefire3902702455596062375 2021-03-16T17-07-16_904-jvmRun1 surefire2822251614539718370tmp surefire_34118265438165058300tmp"
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
[ERROR] org.apache.olingo.jpa.processor.core.query.TestJPACount
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282)
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
[ERROR]         at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR]         at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR]         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
[ERROR]         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR]         at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR]         at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
[ERROR]         at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
[ERROR]         at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
[ERROR]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
[ERROR]

guai · 2021-03-16T15:17:07Z

Looks like it was a maven bug can be fixed by adding <forkCount>0</forkCount> to maven-surefire-plugin's <configuration>.
After that there are still some tests that fail:

[ERROR] Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 35.928 s <<< FAILURE! - in org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest
[ERROR] testLoad(org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest)  Time elapsed: 32.941 s  <<< ERROR!
java.io.FileNotFoundException: target\test-load.xlsx (Системе не удается найти указанный путь)
        at org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest.testLoad(ExcelTransformationTest.java:200)

[ERROR] testSuccessfulFullExport(org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest)  Time elapsed: 0.307 s  <<< ERROR!
java.io.FileNotFoundException: target\test-full.xlsx (Системе не удается найти указанный путь)
        at org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest.testSuccessfulFullExport(ExcelTransformationTest.java:83)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ExcelTransformationTest.testLoad:200 ? FileNotFound target\test-load.xlsx (Сис...
[ERROR]   ExcelTransformationTest.testSuccessfulFullExport:83 ? FileNotFound target\test...
[INFO]
[ERROR] Tests run: 15, Failures: 0, Errors: 2, Skipped: 0
[INFO]
[ERROR] There are test failures.

And I'm actually can't find neither test-load.xlsx nor test-full.xlsx in the repo...

rzozmann · 2021-03-17T11:50:09Z

And I'm actually can't find neither test-load.xlsx nor test-full.xlsx in the repo...

These excel-sheets are created by the test classes (testing the excel export), so they are not in the repository. I think your setup is correct (the build + tests are started). That you fail in 'processor' is expected; that is because i telled you that your changes have side effects....
I' normally running a linux environment, checking the Windows capabilities takes more time... What i can see: you have Maven 3 envionment: good; you have used Java 8: also good.
The defaults of Maven should use 1 Thread, so a custom configuration (forkcount) shouldn't be neccessary...
So your problems should be os dependent and no mistake by yourself... but that gives you no solace ;-)

guai · 2021-03-19T11:55:35Z

Are you saying that those errors about *.xlsx files caused by my changes?
They are not. All the above listing I got on unchanged sources from origin/eXXcellent_adaptions.
I wanted to have some baseline first and then look what my changes add to the picture.

rzozmann · 2021-04-06T14:33:22Z

...a-processor/src/main/java/org/apache/olingo/jpa/processor/core/query/EntityQueryBuilder.java

    queryResult.putElementCollectionResults(readElementCollections(elementCollectionMap));

    if (processExpandOption && !intermediateResult.isEmpty()) {
+      String idAttributeName = getQueryResultType().getKeyAttributes(false).get(0).getInternalName();


This line is the reason for some of the test failures: your using the attribute name to get the result value from the tuple, but in the tuple the result is accessible only be the alias... so you have to do:
String idAttributeName = getLastAffectingNavigationKeyBuilder().getNavigationKeyPaths().get(0).getAlias();
But there are more issues... one should be the fact, that you using only one column as key (see ...get(0)), but there are entities with multiple key columns in the world, so you have to bring ALL key columns into effect for the expand query...

Hi
What would be the proper way to get all the key fields?
Anyway, I think not handling multi-column keys is still better than it was before :)

getLastAffectingNavigationKeyBuilder().getNavigationKeyPaths().get(0).getAlias() gives me something else other than key column name. My JPA entities does not work with this change

It seems we cannot use tuples in JPA criteria builder of an inClause. Anyway odata cannot express composite primary keys either

guai added 2 commits February 17, 2021 12:20

new default converter: Character2StringODataAttributeConverter

a52b387

in case when $expand present use 'WHERE parent_table.id IN (?, ?, ?)'…

89add06

… so no redundant data loaded from the DB

rzozmann reviewed Apr 6, 2021

View reviewed changes

$expand tries to read much more data, than actually needed #37

Are you sure you want to change the base?

$expand tries to read much more data, than actually needed #37

Uh oh!

Conversation

guai commented Feb 17, 2021

Uh oh!

rzozmann commented Feb 19, 2021

Uh oh!

guai commented Feb 20, 2021

Uh oh!

rzozmann commented Mar 5, 2021

Uh oh!

guai commented Mar 5, 2021

Uh oh!

rzozmann commented Mar 15, 2021

Uh oh!

guai commented Mar 16, 2021

Uh oh!

guai commented Mar 16, 2021

Uh oh!

rzozmann commented Mar 17, 2021

Uh oh!

guai commented Mar 19, 2021

Uh oh!

rzozmann Apr 6, 2021

Choose a reason for hiding this comment

Uh oh!

guai Apr 13, 2021

Choose a reason for hiding this comment

Uh oh!

guai Apr 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guai Apr 13, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

guai Apr 13, 2021 •

edited

Loading