Enhancement Requests: Exclude fields and give ability to access [@url] and other "bracket" fields

[ddcornwall]

I want to be able to exclude [&extractedText] from my retrievals because it seems to slow down the web app I'm writing. But there does not appear to be a way to exclude fields.

I have been working with the &resultFields= operation you provide. I could have gotten the result I wanted by retrieving objects.object.thumbnail without retrieving objects.object.file. Except when I do that, the file path of the thumbnail changes to the unusable "https://catalog.archives.gov/catalogmedia/".

After trying that I thought that maybe I could get the right thumbnail path to appear if I pulled the objects.object.file["@path"] field. But trying that field, along with any other field in brackets gives me an invalid field error. It would be very helpful to be able to use these fields in &resultFields.

Thanks for considering this idea.

[DominicBM]

Hi @ddcornwall, there's a few different things going on here, so I have separated them out below:

I want to be able to exclude [&extractedText] from my retrievals because it seems to slow down the web app I'm writing. But there does not appear to be a way to exclude fields.

Yes, there are some long fields that can cause slower performance because it's trying to send you a lot of data. Some other fields that can have long blocks of text are the "transcription" field (on objects) and the "scope and content note" field in descriptions, but the extractedText is the main culprit. This is on our radar to improve in future releases, so it's good to hear your user feedback. It's not ideal, but one workaround for now, if you don't need any of the technical metadata fields is to use resultFields to "restrict" the response to all of the fields other than that one. For objects, for example, you can say "&resultFields=num,type,parentDescriptionNaId,objects.object.@id,objects.object.@objectSortNum,objects.object.thumbnail,objects.object.file,publicContributions".

After trying that I thought that maybe I could get the right thumbnail path to appear if I pulled the objects.object.file["@path"] field. But trying that field, along with any other field in brackets gives me an invalid field error. It would be very helpful to be able to use these fields in &resultFields.

This is unfortunately a known issue, and we've been waiting on a fix from our developers. Basically, there is a field name white list which all user inputs are checked against—to be able to identify bad fields with a descriptive error message—but we discovered that the white list is not comprehensive of all fields, which means that using some true fields, like "objects.object.file.@url", will return an INVALID_FIELD error instead of data. We are hoping to have this fixed soon.

I could have gotten the result I wanted by retrieving objects.object.thumbnail without retrieving objects.object.file. Except when I do that, the file path of the thumbnail changes to the unusable "https://catalog.archives.gov/catalogmedia/".

I have reproduced this issue, and will report it immediately. I am not sure we have come across this specific bug, but if the API is returning bad data, that is a major problem! It looks to me like there is something wrong in the filter logic, where the thumbnail url isn't actually an indexed field, but it's computed from other fields (like the file path) that cause it to fail if those fields aren't requested. But this should be fixed.

[ddcornwall]

Hi Dominic - Thanks for the swift and constructive response! It's good to know that there are fixes coming.

I also wanted you know that I tried the set of "resultFields=" you suggested and the API gave me errors:

opaResponse -
header - @status400
time 2018-01-30T14:08:36.967Z
request @actionsearch
@path /api/v1/
error @code INVALID_FIELD
description Invalid result field: objects.object.@objectsortnum
format json
pretty true
action search
rows 10
description.fileUnit.variantControlNumberArray.variantControlNumber%22Alaska%20Digitization%20Project%22
result Types fileUnit
result Fieldsnum,type,parentDescriptionNaId,objects.object.@id,objects.object.@objectsortnum,objects.object.thumbnail,objects.object.file,publicContributions

Really looking forward to seeing the thumbnail fields retrieval fixed. Really appreciate this API overall. You can see the use I've put it to at https://ddcornwall.github.io/nara-alaskana/. It's baby steps, but it's mine -- with your API behind it.

[DominicBM]

Sorry, that was my typo. Field names are case-sensitive, so @objectsortnum should have been @objectSortNum.

This query will work, for example:

• https://catalog.archives.gov/api/v1?type=object&resultFields=num,type,parentDescriptionNaId,objects.object.@id,objects.object.@objectSortNum,objects.object.thumbnail,objects.object.file,publicContributions&sort=titleSort%20asc

I really love seeing how you're using the NARA catalog API! We're still working on making it as useful as it can be, as you can see. It might be good to chat more sometime about your work. Feedback from real users is always useful.

[ddcornwall]

Hi @DominicBM . Thanks! I'd be happy to chat sometime about my work. You could use my bug report form at https://librarianfromalaska.wordpress.com/bug-reports/ to send me your e-mail address. Or open an issue on my repo at https://github.com/ddcornwall/nara-alaskana.

The main objective of my project is provide Alaskans who may not have a lot of comfort in using the larger yet excellent NARA Catalog an interface to just Alaska related digital objects. I was having trouble finding a way to flag descriptions with digital objects, so I started using items in the Seattle Office's "Alaska Digitization Project" as a proxy for Alaska-related records with digital objects.

But now I've found at least one record that is identified as being part of the Alaska Digitization Project but doesn't appear to have any digital items - https://catalog.archives.gov/api/v1?naIds=41027079. Am I somehow missing associated materials, or does "description.fileUnit.variantControlNumberArray.variantControlNumber.number = Alaska Digitization Project" mean something different than I thought I did? Maybe that's a question for the Seattle Field Office.

Thanks again for the swift and effective support of your API.