n-API can return dataset level information

[surchs]

Is there an existing issue for this?

• I have searched the existing issues

Summary

As the query tool (or an interested API user), I want to see dataset level information to display to my user. Especially on

• level of current access of data
• Authors
• contact info
• etc

TODO

• Read in the dataset metadata JSON created in Add entrypoint script to process dataset metadata in JSONLD files recipes#163 on API startup
  • Introduce env var for the datasets metadata JSON path (can be configured for dev, value to be hardcoded in the docker-compose.yml as part of [ENH] Add service to validate and extract dataset info from JSONLDs recipes#164)
    • error out if file not found
  • Add startup event to load datasets metadata JSON into memory
• Expand model for only a datasets-level response from POST /datasets to:
  • add new dataset attributes
  • remove dataset_portal_uri
    

    api/app/api/models.py

    Lines 120 to 131 in 94c41f0

    	class DatasetQueryResponse(BaseModel):
    	"""Data model for metadata of datasets matching a query."""
    	dataset_uuid: str
    	# dataset_file_path: str # TODO: Revisit this field once we have datasets without imaging info/sessions.
    	dataset_name: str
    	dataset_portal_uri: Optional[str]
    	dataset_total_subjects: int
    	records_protected: bool
    	num_matching_subjects: int
    	image_modals: list
    	available_pipelines: dict

• separate out response model for legacy GET /query endpoint so it is not tied to other response models
• Update SPARQL query to no longer return ?dataset_name or ?dataset_portal_uri
• Remove all dataset attributes from /subjects endpoint other than dataset_uuid
• Update /datasets endpoint to append dataset attributes from the dataset metadata JSON to the response object, matching datasets based on the UUID
• Update test data
• [ ] Return node mode --> info already available via records_protected field in /datasets response

Considerations

• SPARQL query for legacy /query endpoint still looks for nb:hasPortalURI which will be deprecated in future JSONLDs, so the dataset_portal_uri field of the response may be always empty
• Separate (unoptimized) SPARQL queries for POST /subjects vs. GET /subjects?

Option 1: rename hasPortalURI edge in SPARQL query but update response models/shaping so that GET /subjects no longer returns dataset-level metadata

• Pro:
  • Minimal changes to query template
  • Full SPARQL query template is now up to date with latest JSONLD model
• Cons:
  • Less optimized SPARQL query since we're returning more info than we need
  • Confusing to maintain since we would be using different internal and exposed field names, etc.
  • Maintains coupling between /subjects and /query via the SPARQL query

Option 2: deprecate /query endpoint entirely

• Pros:
  • We no longer need to maintain endpoint
  • Also simple, just need to update one SPARQL query
• Cons:
  • Bad for any tools consuming the API directly or dedicated, outdated query tool + f-API still using the /query endpoint
  • Need to communicate to nodes (maybe: BrainLife? ReproLake? MicroGigs?)

Option 3: Use slightly different SPARQL queries for each. Can either use two functions or one function with conditionals.

• Pros:
  • We can fully split logic for endpoints - no more interdependency, no rush to retire query endpoint
• Cons:
  • Potential duplication or branching logic = more maintenance
  • Visual clutter

Decision: We will implement Option 1 in this PR as a temporary fix, and deprecate the /query tool endpoint soon in #520.

Documentation update(s) needed

No response

[alyssadai]

@surchs, thoughts on:

1. If we're okay to add all the new dataset metadata to both the /subjects and /datasets endpoint responses?
2. How we should return the node mode? This is set via an env var in the API, so some options I can think of are:
   a. store the value under a new dataset-level key in the query response (though this would be repeated for each dataset)
   b. create a new endpoint that returns only info about whether the node is aggregated/not

[surchs]

If we're okay to add all the new dataset metadata to both the /subjects and /datasets endpoint responses?

What's the pro/con here? Just impact on query time? Not sure I understand the implications of the decision yet.

If costs are low / none then including in both would be great of course!

Edit: sorry, replied before thinking. I think datasets metadata should be limited to datasets query endpoint

How we should return the node mode?

Strong preference for option b. But how important / urgent is this? We'd need it for the UI element about "discovery level", correct? Also feel like I may be missing a bit on the implications here

[alyssadai]

@surchs just as a sanity check: do you agree that it's safe to assume that any given datasets metadata JSON file (containing the dataset attributes for all datasets for the node) loaded into the n-API will always have the same expected keys? i.e. that it must have been generated automatically by our own recipes scripts in neurobagel/recipes#164.

The implication being that the n-API doesn't need to validate that file again/know how to handle weird keys being in that file on startup.

I think the only time this assumption wouldn't hold is:
a. if you're doing local development (fine)
b. if someone is deploying our n-API as a standalone Python app rather than part of our Docker Compose recipe (bad, but I this should essentially never happen so we don't have to worry about it)

[surchs]

Hey @alyssadai , yes, I agree that the n-API doesn't have to validate these files. I also think that in not too long, these JSON files will just be loaded into some simple database by the entrypoint with something like DuckDB or mongoDB. I.e. I'd expect the API to soon not have to handle these files directly