Validator incorrectly indicates error when testing selective harvesting and a datestamp doesn't match the "reference date"

[zerocrates]

The validator seems to test selective harvesting by setting both from and until to the same date and then doing a series of checks.

One of those checks is that every record returned must have a datestamp that matches the date searched for. However, if my reading of the spec is right, this isn't correct behavior:

The selective harvesting section of the spec says that repositories must include any records that were created, modified, or (optionally) deleted within from and until, inclusive. The same section defines the datestamp for a record as a date "which reflects the most recent date and time of the creation, modification, or deletion according to the rules defined above."

So, a record created on the reference date the validator requests must be included in the response, but its datestamp must also be later than that if the record was ever updated.

It seems to me the only check of this type you can validly do here is just ensuring that no datestamps earlier than the reference appear in the response.

[Daniel-KM]

Hi,

I confirm. In fact, the validator doesn't understand that the harvest is done against added and modified dates, but the datestamp of a record is only the newest one.

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management

[olli-gold]

I can also confirm that. Validating our OA repository tub.dok (http://tubdok.tub.tuhh.de/oai/request) also reports an error "No incremental harvesting (day granularity) of ListRecords: Harvest for reference date 2015-09-28 returned no records.". I believe that harvesting tub.dok by date works fine, but in fact no record has been created or modified on Sep-28 2015. So, it is not an error, that the request https://tubdok.tub.tuhh.de/oai/request?verb=ListRecords&metadataPrefix=oai_dc&from=2015-09-28&until=2015-09-28 does not return any matches... https://tubdok.tub.tuhh.de/oai/request?verb=ListRecords&metadataPrefix=oai_dc&from=2015-09-28&until=2015-12-28 is returning many records, which proves, that a date range request is responded correctly. Perhaps its better to check this point with a date range instead of a single date?

[joschirr]

I would split the incremental harvesting check into

• check with 'from' parameter
• check with 'until' parameter
• check with 'from' and 'until' parameter (as it is now)
  Using the same referencedate for 'from' and 'until' in the combined check of 'from' / 'until' meets the OAI-PMH specification

the from argument must be less than or equal to the until argument

but it seems a number of OAI-implementations expect that the 'from' date is different from the 'until' date and therefore return no records if the referencedate is the same.

[olli-gold]

OK, now I understand, why this test has been implemented as it is. But it still might be possible, that the test produces a false-negative as there may be no records on the date specified, so that its ok for the service to return no records (as I explained in my comment earlier).
I guess splitting the incremental harvesting check is a good idea, but the only way to be sure about the conformity of the OAI data provider considering the same date for 'from' and 'until' is in my opinion to take a date, on that one record minimum must have been added to the repository. I'm not sure how to find one date, which is valid for such a request, though...