WIP SRUopener #682

dr0i · 2025-03-28T12:17:49Z

This is a draft and WIP.
@TobiasNx you can use it for functional testing.

Resolves #510.

TobiasNx · 2025-03-28T13:36:41Z

Nice seems to work. +1
The printed logs are a little bit esoteric:

, startRecord=1, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=1001
, startRecord=1001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=2001
, startRecord=2001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=3001
, startRecord=3001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=4001
, startRecord=4001, maximumRecords=1000, istream.length=437

TobiasNx · 2025-04-10T08:55:57Z

@dr0i is still in review?

dr0i · 2025-04-10T08:59:18Z

As we found out in #510 this PR needs a complete redesign.

dr0i · 2025-06-02T14:37:44Z

@TobiasNx can you do functional tests before I go on here? Have a look at the @Description to see how it works (hint: "stream" based, i.e. other than the OAI-PMH opener works atm.)
I've added the class to flux-commands.
[edit]: and ignore the failing editorconfigChecker for now.

TobiasNx · 2025-06-04T09:54:41Z

@dr0i I tried to install the dist: https://metafacture.github.io/metafacture-documentation/docs/flux/Flux-User-Guide.html#build-from-local-distribution to try the runner for functional testing

but it runs into errors:

$ ./gradlew installDist

> Configure project :
HEAD has no annotated tags
No SCM tag found. Making a snapshot build
Feature branch found
Version is feature-510-addSruOpener-SNAPSHOT

[Incubating] Problems report is available at: file:///home/user/git/metafacture-core/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.13/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

When I test the flux.sh then it outputs the following:

$ /home/user/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.metafacture.runner.Flux.main(Flux.java:62)
Caused by: org.metafacture.commons.reflection.ReflectionException: Class not found: org.metafacture.io.
        at org.metafacture.commons.reflection.ReflectionUtil.loadClass(ReflectionUtil.java:70)
        at org.metafacture.commons.reflection.ObjectFactory.loadClassesFromMap(ObjectFactory.java:57)
        at org.metafacture.flux.parser.FluxProgramm.<clinit>(FluxProgramm.java:54)
        ... 1 more
Caused by: java.lang.ClassNotFoundException: org.metafacture.io.
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        at org.metafacture.commons.reflection.ReflectionUtil.loadClass(ReflectionUtil.java:67)
        ... 3 more

Can you help? (I tested the current master to compare, there $ ./gradlew installDist and $ /home/user/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh works)

dr0i · 2025-06-05T07:43:05Z

Ah, I accidently removed the TarReader.
Try again please.

TobiasNx · 2025-06-18T07:08:48Z

Current version stucks in an endless SRU request loop starting by 1 again after finishing all request does not matter if a total number of records is given or not:

e.g.

"https://services.dnb.de/sru/authorities"
| open-sru(recordSchema="MARC21plus-xml", query="WOE%3Dsozialistenkongress%20and%20COD%3Ds",version="1.1",maximumRecords="10")
| object-batch-log(batchsize="10")
| as-records
| write(FLUX_DIR + "result.txt")
;

"https://services.dnb.de/sru/authorities"
| open-sru(recordSchema="MARC21plus-xml", query="WOE%3Dsozialistenkongress%20and%20COD%3Ds",version="1.1",maximumRecords="10",total="10")
| object-batch-log(batchsize="10")
| as-records
| write(FLUX_DIR + "result.txt")
;

Both result in, see that recordPosition 1 is turning up again after the expected last recordPosition 8:

<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">042278333</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20110429135047.0</controlfield>
    <controlfield tag="008">900305n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">4227833-8</subfield>
      <subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
      <subfield code="2">gnd</subfield>
    </datafield>
    <datafield ind1=" " ind2=" " tag="035">
      <subfield code="a">(DE-101)042278333</subfield>
...
    <datafield ind1=" " ind2=" " tag="913">
      <subfield code="S">swd</subfield>
      <subfield code="i">k</subfield>
      <subfield code="a">Internationaler Sozialistenkongress</subfield>
      <subfield code="0">(DE-588c)4021089-3</subfield>
    </datafield>
  </record>
</collection></recordData><recordPosition>8</recordPosition></record></records><echoedSearchRetrieveRequest><version>1.1</version><query>WOE=sozialistenkongress and COD=s</query><xQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/><startRecord>6</startRecord><maximumRecords>5</maximumRecords><recordSchema>MARC21plus-xml</recordSchema></echoedSearchRetrieveRequest></searchRetrieveResponse>
<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">042278333</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20110429135047.0</controlfield>
    <controlfield tag="008">900305n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">4227833-8</subfield>
      <subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
      <subfield code="2">gnd</subfield>
...
    </datafield>
    <datafield ind1=" " ind2=" " tag="913">
      <subfield code="S">swd</subfield>
      <subfield code="i">c</subfield>
      <subfield code="a">Bern / Internationaler Sozialistenkongress &lt;1919&gt;</subfield>
      <subfield code="0">(DE-588c)4227833-8</subfield>
    </datafield>
  </record>
</collection></recordData><recordPosition>1</recordPosition></record><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">1267605979</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20230329111229.0</controlfield>
    <controlfield tag="008">220908n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">1267605979</subfield>
      <subfield code="0">http://d-nb.info/gnd/1267605979</subfield>
      <subfield code="2">gnd</subfield>
...

TobiasNx

It seems that SRU opener stucks in infinite loop. See: #682 (comment)

dr0i · 2025-06-20T09:35:43Z

The inifinite loop should be fixed with b92238b, please try again @TobiasNx .

blackwinter

Made a first pass and left some comments. We should discuss tests after the open questions are resolved.

metafacture-io/build.gradle

blackwinter · 2025-07-22T08:58:31Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

Fails Checkstyle check.

Solved in 415dece besides the ClassFanOutComplexity complain. What to do about it?

Just checkstyle-disable-line it for now.

Fixed in f1980d9.

Please don't mix suppression mechanisms. We're using checkstyle-disable-line.

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

metafacture-io/src/test/java/org/metafacture/io/SruOpenerTest.java

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

See #682 (comment).

Co-authored-by: Jens Wille <jens.wille@hbz-nrw.de>

…-core into 510-addSruOpener

- adjust test query to work with URLencoding Complements 2fd8da6. i#Bitte geben Sie eine Commit-Beschreibung für Ihre Änderungen ein. Zeilen,

blackwinter

Apart from the tests, should be mostly okay. Left some more comments regarding small optimizations and naming improvements.

P.S.: Please let the original commenter resolve any open conversations. They should have the opportunity to decide whether they're satisfied with your proposed solution.

blackwinter · 2025-10-24T11:54:20Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

Just checkstyle-disable-line it for now.

blackwinter · 2025-10-24T11:57:47Z

metafacture-io/src/test/java/org/metafacture/io/SruOpenerTest.java

Fails Checkstyle check.

Fixed wih f1980d9.

... resp. complied to editorconfigChecker with a369df5.

Please make sure that the build passes:

Checkstyle rule violations were found.
Checkstyle files with violations: 1
Checkstyle violations by severity: [error:16]

You might want to consider running at least the basic checks locally before pushing (./gradlew check -x integrationTest).

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

Co-authored-by: Jens Wille <jens.wille@hbz-nrw.de>

- fix call in test from "setTotal" to "setTotalRecords" Resolves #682 (comment).

Resolves #682 (comment).

Resolves #682 (comment) and #682 (comment) and #682 (comment).

Resolves #682 (comment).

blackwinter · 2025-10-28T16:51:39Z

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

Please don't mix suppression mechanisms. We're using checkstyle-disable-line.

metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

blackwinter · 2025-10-28T17:03:45Z

metafacture-io/src/test/java/org/metafacture/io/SruOpenerTest.java

Please make sure that the build passes:

Checkstyle rule violations were found.
Checkstyle files with violations: 1
Checkstyle violations by severity: [error:16]

You might want to consider running at least the basic checks locally before pushing (./gradlew check -x integrationTest).

github-project-automation bot added this to Metafacture Mar 28, 2025

dr0i requested a review from TobiasNx March 28, 2025 12:18

dr0i changed the title ~~WIP SRUopener (#510)~~ WIP SRUopener Mar 28, 2025

dr0i moved this to Review in Metafacture Mar 28, 2025

dr0i added Enhancement Flux Conform To Catmandu labels Mar 28, 2025

TobiasNx approved these changes Mar 28, 2025

View reviewed changes

TobiasNx mentioned this pull request Mar 31, 2025

Add SRU opener / open-sru #510

Open

dr0i force-pushed the 510-addSruOpener branch from ecd9c8c to c3f3ad6 Compare April 10, 2025 13:32

dr0i force-pushed the 510-addSruOpener branch from 84d6845 to 3dc0416 Compare June 2, 2025 14:24

dr0i requested a review from TobiasNx June 2, 2025 14:38

dr0i assigned TobiasNx Jun 2, 2025

TobiasNx assigned dr0i and unassigned TobiasNx Jun 4, 2025

dr0i assigned TobiasNx and unassigned dr0i Jun 5, 2025

dr0i force-pushed the 510-addSruOpener branch from 4a00838 to ac80718 Compare June 10, 2025 13:02

TobiasNx assigned dr0i and unassigned TobiasNx Jun 18, 2025

TobiasNx requested changes Jun 20, 2025

View reviewed changes

dr0i assigned TobiasNx Jun 20, 2025

blackwinter requested changes Jul 22, 2025

View reviewed changes

blackwinter assigned dr0i and unassigned blackwinter Jul 22, 2025

dr0i and others added 12 commits September 11, 2025 16:02

Fix loop when getting data

ecd05fe

Remove unused dependecies

406c5ce

See #682 (comment).

Adjust to fulfill checkstyle conditions

415dece

Fix license

a111aff

Use int in setters

afc756c

Update metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

94da562

Co-authored-by: Jens Wille <jens.wille@hbz-nrw.de>

URLencode query parameter

2fd8da6

Merge branch '510-addSruOpener' of github.com:metafacture/metafacture…

6e7b03c

…-core into 510-addSruOpener

Use try to invoke auto closeable

7f7beab

Inline method thus getting rid of two member variables

1291754

Fix test endless loop

907fdfc

- adjust test query to work with URLencoding Complements 2fd8da6. i#Bitte geben Sie eine Commit-Beschreibung für Ihre Änderungen ein. Zeilen,

Set maximumRecords earlier as it is invariant

aa161ce

dr0i assigned blackwinter and unassigned dr0i Oct 20, 2025

blackwinter requested changes Oct 24, 2025

View reviewed changes

blackwinter assigned dr0i and unassigned blackwinter Oct 24, 2025

dr0i and others added 7 commits October 24, 2025 15:13

Update metafacture-io/src/main/java/org/metafacture/io/SruOpener.java

ad7ae5f

Co-authored-by: Jens Wille <jens.wille@hbz-nrw.de>

Switch back to camelcase

9446b7d

- fix call in test from "setTotal" to "setTotalRecords" Resolves #682 (comment).

Suppress warning "checkstyle:ClassFanOutComplexity"

f1980d9

Resolves #682 (comment).

Instantiate classes only once

3ab46e5

Resolves #682 (comment) and #682 (comment) and #682 (comment).

Build the string once by concatenation

f88c83e

Resolves #682 (comment).

Use a descriptive name

93a3f08

Resolves #682 (comment).

Comply to editorconfigChecker

a369df5

Resolves #682 (comment).

dr0i assigned blackwinter and unassigned dr0i Oct 28, 2025

blackwinter requested changes Oct 28, 2025

View reviewed changes

Uh oh!

WIP SRUopener #682

Are you sure you want to change the base?

WIP SRUopener #682

Uh oh!

Conversation

dr0i commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx commented Mar 28, 2025

Uh oh!

TobiasNx commented Apr 10, 2025

Uh oh!

dr0i commented Apr 10, 2025

Uh oh!

dr0i commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr0i commented Jun 5, 2025

Uh oh!

TobiasNx commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dr0i commented Jun 20, 2025

Uh oh!

blackwinter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blackwinter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

dr0i commented Mar 28, 2025 •

edited

Loading

dr0i commented Jun 2, 2025 •

edited

Loading

TobiasNx commented Jun 4, 2025 •

edited

Loading

TobiasNx commented Jun 18, 2025 •

edited

Loading

TobiasNx left a comment •

edited

Loading