Data Debug Workflow

This is the procedure to follow if your data is not available on Code Ocean. Data goes from the acquisition rig to Code Ocean through a series of steps. The first thing to understand is what step failed, then look at any log information available, and finally contact the appropriate person.

1. "Missing data"

   • Contact the trainer for that session to confirm if the mouse actually ran.
   • If the mouse did run, check to see if there is saved data on the acquisition hard drive

2. Data on the acquisition hard drive

   • ForagingSettings.json specifies where data is saved. This defaults to "C:\behavior_data"
   • Each session is a folder nested under rig name, then mouse.
   • Standard format:
     • behavior - raw.harp, json file (structure of each trial only related to this task), log file for the session
     • behavior-videos - if there's data
     • fib - if there's data
     • metadata-dir - rig.json (AIND rig definition), session.json (high-level info for the session, AIND format)
   • Is the data saved on the acquisition computer? If not, it may be backed up on an on-prem server - contact Kenta.

3. Data moves from the acquisition hard drive to VAST via aind-watchdog-service

   • Problems at this stage show up on the email report as "stalled" or "lingering manifest"
   • Each session generates an upload manifest after the first trial finishes
   • Manifests don't conflict each other - can run multiple sessions from the same mouse within a day.
   • Waiting to upload manifests are here: C:\Users<user name>\Documents\aind_watchdog_service\manifests
   • Completed manifests are here: C:\Users<user name>\Documents\aind_watchdog_service\manifests
   • You can check the watchdog log here (requires VPN, to on-site): http://eng-logtools:8080/?channel_filter=watchdog&hide=location,count
   • If your manifest failed to upload, talk to Patrick in SIPE: "Upload failed, what should I do?"

4. Data moves from VAST to S3 bucket via aind-data-transfer-service

   • Problems at this stage show up on the email report as "No data assets"
   • Failures here either mean the upload to S3 failed, or asset registration on Code Ocean failed
   • You can check the data transfer log here: http://aind-data-transfer-service/jobs
   • Contact Jon Young in Sci.Computing "upload and registration failed, what should I do?"

5. Data gets processed into NWB files

   • Problems at this stage show up on the email report as "No processed data" and "Processing errors"
   • You can see logs under the "Traceback" section of this dashboard grafana dashboard
   • You can also see logs under the "Timeline" tab of the pipeline: Code Ocean Pipeline
   • Ask for credentials if you don't have them
   • If failed at this step, talk to Ahad Bawany@SciComp. "My data uploaded, but did not process into NWBs, what should I do?"

6. Training was inadvertently done under a wrong subjectID

   • Modify the existing watchdog manifest: replace subjectID and file locations thoroughly
   • Modify the corresponding data: names, metadata, behavior-json, etc, change all of them using "replace function". (except for the pythonlog)
   • If the data was already once uploaded under a wrong name, ask Jon Young at SciComp to archive data at s3/CO.
   • cf:https://github.com/AllenNeuralDynamics/aind-behavior-blog/issues/1070