BioDataLab.csv

,Index,database,domain,task_type,Verifier,Task Name,Data Type,Task Instruction,Origin Paper Description
0,1,A3D-MODB,Protein Sequence,Open-World Data Retrieval,a3d_modb_retrieval,a3d_modb_retrieval,Sequence,"Query the UniProt database to find all reviewed (Swiss-Prot) proteins from Escherichia coli (strain K12) that are annotated with the Protein name ""transmembrane"". 

    Save the list of their UniProt accession numbers to `<r></r>/a3d_modb_retrieval.json`.

    Output example format:
    ```json
    [
""P0A8T7"",
""P0A8T8"",
""P0A8T9"",
...
    ]
    ```

    All intermediate results should be saved to `<t></t>/a3d_modb_retrieval`
","A3D-MODB incorporates prediction of transmembrane regions for membrane proteins annotated as trans- or intra- membrane in UniProt KB using the TOPCONS consensus algorithm 28 and ists predicted opologies rom OCTOPUS (43 , Philius (44 , Polyphobius (45 , SCAMPI (46 and SPOCTOPUS (47 bioinformatics tools.
"
1,2,ADCdb,Human Genes and Diseases,Structured Information Extraction,adcdb_extract_1,adcdb_extract_1,Text,"# Core Task
    Extract detailed pharmacological information about Antibody-Drug Conjugates (ADCs) from the literature, specifically identifying Key fields including ADC Name, Conjugate Type, Drug-Antibody Ratio (DAR), mAb, Linker, Payload, and Drug Status.
    Literature to analyze: `dataset/ADCdb/ref2.pdf`, `dataset/ADCdb/ref4.pdf`, `dataset/ADCdb/ref5.pdf`

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `Paper`: The reference ID or filename of the paper (e.g., ref1).
- `ADC Name`: The name of the Antibody-Drug Conjugate.
- `mAb`: The specific monoclonal antibody used.
- `Linker`: The type of linker used to connect the antibody and payload.
- `Payload`: The cytotoxic payload delivered by the ADC.
- `Conjugate Type`: The method or type of conjugation.
- `DAR`: The Drug-Antibody Ratio.
- `Drug Status`: The current clinical or development status of the ADC.
    - Save Path: Save the result to: `<r></r>/adcdb_extract_1.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/adcdb_extract/`

    # Output Example (For Reference Only)
    Paper,ADC Name,mAb,Linker,Payload,Conjugate Type,DAR,Drug Status
    ref2,Pseudo-ADC-X,Anti-Target-Y IgG1,Enzymatic linker,Cytotoxin Z,Site-specific,3.8,Phase 3
    ref4,NextGen-ADC,Bispecific mAb,pH-sensitive linker,Potent Payload A,Cysteine conjugation,N/A,Preclinical
","The development of novel ADC was frequently inspired by the valuable pharma-information of the pre-existed ADCs (16–18). Such information included: the aimed disease indications together with their corresponding clinical status, ADC’s structure describing antibody, linker and payload, diverse clinical response data (such as the clinical details of both tested ADC and enrolled patient, the reported ADMET properties of ADC, administration dosage, reported adverse drug reactions and clinical primary endpoints), drug-antibody ratio of each studied ADC (19–24)."
2,3,ADCdb,Human Genes and Diseases,Structured Information Extraction,adcdb_extract_2,adcdb_extract_2,Database,"# Core Task
    Conduct a comprehensive PubMed literature search to identify original English research articles investigating the biological activity of Zilovertamab vedotin. The search query should combine drug-specific keywords鈥擹ilovertamab vedotin (MK-2140/VLS-101)鈥攚ith biological activity terms (such as pharmacology, mechanism of action, efficacy, and ADC), restricted to the title and abstract fields.  For each retrieved article, analyze its abstract to determine if it investigates the biological activity of the drug.

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets the biological activity investigation criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/adcdb_extract_2.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/adcdb_extract/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""11111111"",
""Accepted"": false
},
{
""PMID"": ""22222222"",
""Accepted"": true
}
    ]
","Third, the biological activity of these ADCs was further retrieved using comprehensive literature review in diverse literature knowledgebases (such as PubMed) based on such keyword combinations as antibody–drug conjugate $^+$ biological activity’, ADC name $^+$ activity’"
3,4,AMDB,Structure,Open-World Data Retrieval,amdb_retrieval,amdb_retrieval,Database,"Search the NCBI GEO database for Series containing 16S rRNA amplicon sequencing data from animal gut or fecal samples. 

    The data should published between 2010.01.01 and 2024.01.01. 

    The search criteria must include the keywords ""<key>gut microbiota</key>"", ""<key>16S rRNA</key>"", ""<key>V4</key>"", and ""<key>Illumina</key>"". 

    The search should be limited to studies that are linked to a publication. 

    Save the list of resulting SRA Study accession numbers (SRR IDs) to a JSON file `<r></r>/amdb_retrieval.json`.

    Output format (for reference only):
    [
""SRR12345678"",
""SRR12345679"",
""SRR12345680"",
...
    ]

    All intermediate results should be saved in `<t></t>/amdb_retrieval` directory.
","We manually selected candidate data for AMDB from the NCBI Sequence Read Archive (SRA) based on the following criteria: (i) samples included fecal or intestinal contents from individual healthy animal hosts, (ii) the PCR primers had to target the V4 hypervariable region of the 16S rRNA gene, (iii) amplicons had to be sequenced on Illumina instruments, (iv) samples had to be linked to research articles. For longitudinal data, only one sample was selected as follows; only one adult sample was included when the samples were from multiple life stages, and one sample from an earlier time point was selected for a given life stage. Samples that were duplicates of those previously included in the  
"
4,5,AMDB,Structure,Structured Information Extraction,amdb_extract,amdb_extract,Database,"Review the published materials corresponding to the NCBI Biosample ERS1713185 and SRS4889584, collect host information metadata, and extract dietary information of host animals from databases such as MammalDIET, and so on.

    Saving the results to `<r></r>/amdb_extract.csv`.

    Output format (for reference only):
    Sample Accesion,Host,Diet
    ERS1713185,Capra hircus,Herbivore
    SRS4889584,N/A,N/A

    All intermediate results should be saved in `<t></t>/amdb_extract` directory.
","We checked the suitability of samples by reading the publication materials and methods, and we collected metadata, including the accession numbers and the host information. We extracted information on host diets from the MammalDIET (30) and the EltonTraits database (31). A total of 4633 samples were obtained from 51 projects (Figure 1).  
"
5,6,ASMdb,Nucleotide Sequence,Functional Feature Annotation,asmdb_annotate,asmdb_annotate,Tabular,"Identify the top 100 genes that most frequently exhibit Allele-Specific Methylation (ASM) across a population of samples. 

    Given detected ASM sites across multiple biological samples `benchmark/dataset/ASMdb/mock/mock_asm_sites.csv` and genomic coordinates for all genes `benchmark/dataset/ASMdb/mock/mock_genes.csv`.

    For each sample, quantified the number of ASM sites within each gene body and its $3~\mathrm{kb}$ upstream region. The frequency of genes covered by ASM was then aggregated across all samples. 

    Finally, identified the top 100 genes with the highest ASM frequency, and save the result into a csv file `<r></r>/asmdb_annotate.csv`

    Output example for reference:
    gene_id,frequency
    GENE_1,15
    GENE_2,14
    GENE_3,13
    GENE_4,12
    ...

    All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/asmdb_annotate`
","In each sample, we counted the number of ASM loci in each gene and the upstream $3~\mathrm{kb}$ range of the gene and then added up the frequency of the gene covering ASM in all samples. Finally, we identified the 100 ASM genes with the highest frequencies. Similarly, we calculated the ASEGs with the highest frequency in each sample, which were called high-frequency ASEGs.  
"
6,7,ASMdb,Nucleotide Sequence,Data Refinement and Integration,asmdb_refinement_1,asmdb_refinement_1,Other Structured File,"Identify candidate genomic sites for Allele-Specific Methylation (ASM) analysis.

    You are provided with a BED file containing methylation information for individual cytosine sites `benchmark/dataset/ASMdb/toy_bed/mini_test.bed`. Your task is to filter this dataset to retain only ""partially methylated"" sites which serve as valid candidates for haplotype-dependent ASM identification. 

    Retain only the sites with intermediate methylation levels (0.1 <= Methylation Level <= 0.9).

    Saving the filtered bed file in `<r></r>/asmdb_refinement_1.bed`

    All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/asmdb_refinement_1/`
","The DNA methylation level was calculated with BatMeth2 software (31). ASM was identified by MethHaplo (24) with the default parameters. In the ASM detection process, all the totally methylated (methylation level $>0.9$ ) and totally unmethylated (methylation level $<0.1$ ) sites were removed first, and only partially methylated cytosine sites, denoted as effective sites, were retained for haplotype region identification.  
"
7,8,ASMdb,Nucleotide Sequence,Data Refinement and Integration,asmdb_refinement_2,asmdb_refinement_2,Sequence,"
    Process quality control and genomic mapping of the the raw sequencing data located at `benchmark/dataset/ASMdb/rna_seq/SRR942022_1.fastq`.

    Using fastp with default parameters.

    The reference genome located at `benchmark/dataset/ASMdb/ref/TAIR10_chr_all.fas`.

    Converting the alignment output into BAM format and perform coordinate-based sorting.

    Saving the final bam file in `<r></r>/asmdb_refinement_2.bam`

    All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/asmdb_refinement_2/`
","The raw reads from the RNA-Seq data were trimmed as clean reads using Fastp with the default parameters. The clean reads were mapped to the corresponding reference genome using Hisat2 (37), and SAMtools was used to sort the BAM file. The SNP information used for ASEG detection was derived from the BS-Seq corresponding to the RNA-Seq data. The ASEGs were detected by ASEQ (38).  
"
8,9,ASMdb,Nucleotide Sequence,Open-World Data Retrieval,asmdb_retrieval,asmdb_retrieval,Database,"# Core Task
    Search all available whole-genome bisulfite sequencing data from the NCBI GEO database for Series entries published from October 1, 2019 to October 30, 2019.

    # Output Requirements & File Path
    - Output format: A JSON list of Series accession numbers (strings).
    - Include all Series accession numbers in the final output
    - Save Path: Save the complete list to: `<r></r>/asmdb_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/asmdb_retrieval/`

    # Example Output (For Reference Only)
    [
""GSE143728"",
""GSE224457""
    ]
","Therefore, we collected 5,998 Gene Expression Omnibus (GEO) (28) samples (including 4400 BS-Seq data and 1598 RNA-Seq data) from 47 species, including Homo sapiens and Mus musculus, and performed DNA methylation, ASM and allele-specific expressed gene (ASEG) analyses of the corresponding samples. "
9,10,ATLAS,Protein Sequence,Open-World Data Retrieval,atlas_retrieval_1,atlas_retrieval_1,Geometric,"# Core Task
    You are a specialized Bioinformatics Agent acting as an expert in Structural Proteomics and Protein Curation. Your core competency is filtering the PDB (Protein Data Bank) to generate a high-quality, non-redundant dataset of protein chains. You prioritize structural integrity, low resolution, and biological relevance by integrating ECOD domain classifications and monomeric state verification.

    # Workflow Instructions
    - `Initial Quality Filtering`: Filter for X-ray crystal structures from the July 2022 PDB release with: 1) Resolution below or equal to 2 angstrom; 2) Minimum chain length of 38 residues (based on MolProbity quality thresholds).
    - `Domain & Membrane Filtering`: 1) Retain only proteins with a valid ECOD ID (version v285); 2) Exclude membrane proteins based on consensus data from OPM, PDBTM, MemProtMD, and the mpstruc database (as of April 2023).
    - `Redundancy Reduction (ECOD X-group)`: Select the single most representative chain for each ECOD X-group using these hierarchical criteria:
1) Connectivity: Exclude any chain with more than 10 consecutive missing residues;
2) Structural Precision: Prioritize structures present in the Top8000 and Top2018 rotamer datasets;
3) Quaternary State: Prioritize monomeric structures in this order: 1) Experimental Monomer + PISA Predicted Monomer; 2) Experimental Monomer + PISA Predicted Multimer; 3) Multimeric structures.
    - `Tie-breaking`: If multiple structures remain, select based on: (i) minimum consecutive missing residues, (ii) maximum sequence length, (iii) lowest percentage of missing residues.
    - `Dataset Augmentation`: For a subset of these non-redundant proteins, include one additional high-quality protein structure sharing the same ECOD X-group to expand the data.

    # Output Requirements & File Path 
    Generate a JSON file containing a flat list of strings. Each string must follow the format [pdb_id]_[chain_id], where the PDB ID is in lowercase.
    Save Path: `<r></r>/atlas_retrieval_1.json`

    # Output Example (For Reference Only)
    [""4pjs_A"", ""1a62_B"", ""3b5y_D"",""5vms_T""]

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/atlas_retrieval_1`
","High-quality protein chains from the PDB (version of July 2022) were thoroughly filtered to ensure structural diversity by removing redundancy in terms of X-class ECOD (37  domains (indicating similar fold and possible homology). We have first selected all X-ray structures of protein chains of at least 38 residues long with resolution below or equal to $2\textup{\AA}$ , n accordance with MolProbity’s quality thresholds (38 . We have filtered out proteins without an ECOD ID (v285) and membrane proteins (consensus of OPM (39 , PDBTM (40 , MemProtMD 29 , and mpstruc from RCSB PDB April 2023) (41 . We then selected the best chain for each ECOD X-class domain. To do so, proteins with more than 10 consecutive missing residues were excluded. Among the remaining proteins, we prioritised hose present n he quality-filtered rotamer datasets Top8000 and Top2018 (38 42 . We also gave priority o proteins crystalised n monomeric state by selecting first structures crystalised as monomers and predicted by PISA (43  as such, then monomeric structures predicted as multimeric by PISA and finally multimeric structures."
10,11,ATLAS,Protein Sequence,Open-World Data Retrieval,atlas_retrieval_2,atlas_retrieval_2,Sequence,"# Core Task
    You are a specialized Bioinformatics Agent acting as an expert in Protein Conformation and Sequence-Structure Mapping. Your core competency is querying the ChSeq (Chameleon Sequence) database to identify sequences capable of undergoing conformational switching or existing in distinct structural states. Your goal is to extract specific 8-residue sequences that have been experimentally resolved in both alpha-helical and beta-sheet conformations.

    # Workflow Instructions
    - `Sequence Selection`: Search the ChSeq database for sequences with a fixed length of exactly 8 residues.
    - `Structural Mapping`: For each identified 8-mer (ChSeq), find two high-quality protein structures:
Helix State: One protein fragment where the 8-mer adopts a helical conformation.
Sheet State: One protein fragment where the 8-mer adopts a beta-sheet conformation.
    - `Data Extraction`: Retrieve the source information for each state, ensuring the metadata includes the PDB ID and the Chain ID.

    # Output Requirements & File Path
    Generate a JSON file structured as a dictionary.
    Key: The 8-residue ChSeq string.
    Value: A sub-dictionary with two secondary structure sequence computed by DSSP from the experiental structure, e.g. ""EEEEEEEE"" for alpha-helix and ""HCHHHHHH"" for beta-sheet.
    Value Format: Each secondary structure sequence must be a string formatted as [pdb_id].[Chain_ID] (lowercase PDB ID).
    Save Path: Save the result to: `<r></r>/atlas_retrieval_2.json`

    # Output Example (For Reference Only)
    {
'KSLLDYEV': {
'EEEEEEEE': '2r5r.A',
'HHHHHHHH': '2rca.B',
},
'AYLEEEVL': {
'EEEEEEEE': '2z14.A',
'HHHHHHHH': '3b0p.A',
},
    }

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/atlas_retrieval_2`
","For all chameleon sequences of the ChSeq database (36) longer than 7 amino acids we have manually chosen those containing high-quality structures. We have selected two protein structures per chameleon sequence: one for helical and one for $\upbeta$ -strand conformation.  
"
11,12,BioKA,Human Genes and Diseases,Structured Information Extraction,bioka_extract,bioka_extract,Text,"Based on the paper content in the file `benchmark/dataset/BioKA/paper_content.md`, retrieve the biomarker. 

    Extract the `biomarkerName` (abbreviation name), `biomarkerType` (protein, gene, or, others ...), `curated_disease_name`, and `biomarkerUsage` (must be one of diagnosis, prediction, therapeutic, prognosis, monitoring) for each biomarker and save them into a CSV file `<r></r>/bioka_extract.csv` with the headers `biomarkerName`, `biomarkerType`, `curated_disease_name`, and `biomarkerUsage`.


    Ouput example for reference:
    ```
    biomarkerName,biomarkerType,curated_disease_name,biomarkerUsage
    ```

    All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/bioka_extract/`
","Papers with a specific focus on a particular disease/ rait, concise conclusions, and detailed experimental information (e.g. in vitro, in vivo  clinical tests, behavioral tests) were selected for he curation of essential biomarker nformation name, associated disease/ rait, usage, conclusion), sample and experiment details (e.g. numbers, breed/strain, age, sex, technology, tissue, significance), as well as regulatory relation (e.g. circRNA–miRNA). Additionally, n order o enhance he comprehensiveness of BioKA, hree curated resources focusing on human biomarkers CBD (27 , DAAB (29  and EBD (30 , were integrated nto he BioKA according o he curation data standardization.  "
12,13,BioKA,Human Genes and Diseases,Open-World Data Retrieval,bioka_retrieval,bioka_retrieval,Database,"# Core Task
    Conduct a comprehensive literature search on PubMed for articles published between 2022/01/01 and 2022/01/05 related to biomarkers in cat species, using specific keywords (biomarker, marker, indicator, predictor). Exclude non-research articles (reviews, comments, letters, editorials) and non-English publications. Refine the abstract analysis to extract sentences containing both entity terms ('biomarker', 'marker', 'indicator', 'target') and clinical qualifiers ('diagnostic', 'prognostic', 'therapeutic', 'valuable') to filter the entire paper collection.. Finally, perform a thorough filtration and curation of the collection, selecting papers with a specific focus on a particular disease or trait, concise conclusions, and detailed experimental information (e.g., in vitro, in vivo, clinical tests, or behavioral tests).

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets all inclusion and filtering criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/bioka_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/bioka_retrieval/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""99999999"",
""Accepted"": false
},
{
""PMID"": ""88888888"",
""Accepted"": true
}
    ]
","Firstly, literature published before $10\ \mathrm{July}\ 2023$ , was searched using pre-defined keywords such as biomarker’, ‘marker’, ‘indicator’ and ‘predictor’ for each species. A total of $64~935$ literatures were obtained with other basic information (e.g. itle, ournal, paper type, abstract) through NCBI E-utilities and utilized as the original data. Secondly, reviews, comments, letters, editorials and non-English literature were excluded from the dataset. Abstracts were further analyzed using the Natural Language Toolkit (https://www.nltk.org/ to identify sentences containing both an entity term (e.g. ‘biomarker’, ‘marker’, ‘indicator’ or ‘target’) and a qualifying word e.g. diagnostic’, prognostic’, therapeutic’ or valuable’) using regular expression operations in Python (https: //www.python.org/ . This meticulous filtering process resulted in a refined collection of 19297 corresponding articles, serving as the data source for BioKA. "
13,14,CancerMIRNome,Metabolic and Signaling Pathways,Functional Feature Annotation,cancermirnome_annotate,cancermirnome_annotate,Tabular,"From the provided circulating miRNome abundance data (`benchmark/dataset/CancerMIRNome/circulating_miRNome_abundance.csv`), filter for the top 500 most abundant miRNAs based on the ""Abundance"" column. 

    Save the list of their `miRNA_ID`s to a text file named `<r></r>/cancermirnome_annotate.txt`.

    Output example for reference:
    ```
    xxx
    xxx
    ```

    All intermediate results should be saved to `<t></t>/cancermirnome_annotate`
","Highly expressed miRNAs. miRNAs with relatively high abundances may be more reliable, robust and practical to be used as diagnostic or prognostic biomarkers in clinical settings. The highly expressed miRNAs are identified if their counts per million (CPM) are $^{>1}$ in $550\%$ of the samples in a TCGA project or if their abundances are ranked among the top 500 miRNAs in a circulating miRNome dataset.  
"
14,15,CancerProteome,Human Genes and Diseases,Functional Feature Annotation,cancerproteome_annotate,cancerproteome_annotate,Tabular,"Calculate the Spearman correlation coefficient between protein abundance and transcript expression levels. 

    The input data is provided in two separate CSV files: `benchmark/dataset/CancerProteome/protein_abundance.csv` and `benchmark/dataset/CancerProteome/transcript_levels.csv`. 

    Both files contain a ''GeneID'' column for matching, a ''Protein_Abundance'' column, and a ''Transcript_Level'' column, respectively.

    Merge the two files on ''GeneID'', then compute the correlation. Save the final calculated correlation coefficient as a single numeric value in `<r></r>/cancerproteome_annotate.txt`.

    Output Example:
    ```
    0.85
    ```

    All intermidate results should be save in `<t></t>/cancerproteome_annotate` directory.
","We obtained the transcriptome data from the supplementary tables of associated publications. If the transcriptome data was not normalized, we normalized the expression across samples by absolute deviations from the median within each cancer type and dataset. A rich body of researches demonstrate that the mRNA and protein abundances frequently have a poorer correlation than expected, a consequence of both translational and post-translational regulation. Spearman’s correlation analysis was used to measure correlations between protein abundance and corresponding transcript expression of a given protein, or to measure correlations between protein abundance and drug $\mathrm{IC}_{50}$ values. Additionally, the $\cdot\mathrm{lm}^{\mathrm{,}}$ method in R was used to assess correlations between protein abundance and modification levels at sites.  
"
15,16,CancerProteome,Human Genes and Diseases,Functional Feature Annotation,cancerproteome_annotate_2,cancerproteome_annotate_2,Tabular,"Identify protein-protein associations by co-expression analysis with data from the STRING database.

    Protein expression data are given in `benchmark/dataset/CancerProteome/cancer_proteome.csv`.

    For all possible pairs of proteins in the file, calculate the pairwise Spearman correlation coefficient. 

    Identify ""significantly co-expressed pairs"" where the absolute value of the correlation coefficient is >= 0.7 and the p-value is < 0.05.

    For each significantly co-expressed pair (ProteinA, ProteinB), query the STRING database API to check for known interactions between them for Homo sapiens (taxid: 9606).

    Retain only the pairs that have a high-confidence interaction in STRING, defined as a combined score > 700.

    Save the final list of validated protein pairs to a file named `<r></r>/cancerproteome_annotate_2.txt`.

    Output Example:
    ProteinA,ProteinB
    ProteinC,ProteinD

    All intermediate results should be save in `<t></t>/cancerproteome_annotate_2` directory.
","Protein co-expression and PPIs were used o evaluate protein– protein associations in different cancers. First, significant coexpressed (micro-)protein pairs were identified in the specific cancer using Spearman’s correlation and aforementioned thresholds. We also assembled and integrated known PPIs from STRING (41 , MINT (42  and IntAct (43 , and only high-confidence interactions were retained in STRING.  "
16,17,CancerSCEM,Metabolic and Signaling Pathways,Functional Feature Annotation,cancerscem_annotate,cancerscem_annotate,Other Structured File,"Based on the CellMarker file `benchmark/dataset/CancerSCEM/CellMarker.csv`, annoting the cell type `benchmark/dataset/CancerSCEM/cell_1.txt`. 

    Types include CD4+ Tcm, CD4+ Tem,CD8+ effector T cells, CD8+ naive T cells, CD8+ Tcm, CD8+ Tem, Dendritic cells, Endothelial cells, Fibroblasts, Macrophages, Malignant cells, Monocytes, NK cells, Plasma cells, Tregs, Unassigned. 

    Saving the result tpye into `<r></r>/cancerscem_annotate.json`.

    Output format:
    {
""cell_1"": ""CD4+ Tcm""
    }

    All intermediate results should be save in `<t></t>/cancerscem_annotate` directory.
","Manual annotations were subsequently performed based on the expression of dozens of canonical markers for common cell types like T cells (e.g. CD3D, CD3E), B cells (e.g. MS4A1, BANK1), Macrophages/Monocytes (e.g. CD68, CD14), Mast cells (e.g. SLC18A2, ASIC4), Endothelial cells (e.g. VWF, PECAM1), Fibroblasts (e.g. FAP, NECTIN1), Oligodendrocytes (e.g. OLIG1, PLP1) and Astrocytes (e.g. SLC1A3, GFAP) and etc."
17,18,CancerSCEM,Metabolic and Signaling Pathways,Functional Feature Annotation,cancerscem_annotate_2,cancerscem_annotate_2,Other Structured File,"Calculate the Pearson correlation matrix for the genes in the provided expression data `benchmark/dataset/CancerSCEM/gene_expression.csv`. 

    Identify the most highly correlated gene pairs (with correlation coefficient > 0.9).

    Saving the result to `<r></r>/cancerscem_annotate_2.txt`.

    Output format requirements (for reference only):
    gene_1
    gene_2

    All intermediate results should be saved in `<t></t>/cancerscem_annotate_2/` directory.
",GE Correlation––pearson correlation calculation between any two genes in the specific sample ranked by $P-$ -values and (iv) GE Comparison––gene expression comparison across different scRNA-seq or TCGA bulk RNA-seq datasets (Figure 4A).
18,19,CDS-DB,Human Genes and Diseases,Open-World Data Retrieval,cds_db_retrieval,cds_db_retrieval,Database,"# Core Task
    Search the NCBI GEO database for Series published from 2019/10/01 to 2019/10/15 using the query: '(Cancer-related keywords) AND (Drug-related keywords) AND (Patient-related keywords) AND ""Homo sapiens""' to collect paired pre- and post-treatment transcriptome data. The datasets must satisfy the following inclusion/exclusion criteria: (i) paired (i.e. from the same patient) pre- and posttreatment patient samples were required; (ii) for microarray data, only datasets from Affymetrix, Agilent and Illumina platforms were collected; (iii) bulk transcriptomes were collected; (iv) datasets, in which drug therapy is combined with other physical herapies, were excluded to ensure hat he perturbation is induced only by drug therapy.


    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all Series obtained from the initial GEO search.
    - Each object must contain exactly two fields:
- `""Series""`: The GEO Series accession number (e.g., ""GSE123456"").
- `""Accepted""`: Boolean value (`true` if the Series meets all inclusion/exclusion criteria, `false` otherwise).
    - Save Path: Save the complete list to: `<r></r>/cds_db_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/cds_db_retrieval/`

    # Output Example (For Reference Only)
    [
{
""Series"": ""GSE111111"",
""Accepted"": true
},
{
""Series"": ""GSE222222"",
""Accepted"": false
}
    ]
","Paired pre- and post-treatment transcriptomes derived from patients with cancer were manually mined from GEO (18 and ArrayExpress (19  (until July 2022). First, we obtained potentially related datasets by retrieving GEO/ArrayExpress with ‘cancer-related keywords AND drug-related keywords AND patient-related keywords AND Homo sapiens  see Supplementary methods for the detailed keyword lists). Then, we manually picked hose hat met he nclusion criteria described below. For the eligible source datasets, we downloaded rawformatted microarray gene expression data directly from the GEO/ArrayExpress website and downloaded RNA-seq SRA files from NCBI using the ‘prefetch’ function in the sra-tools (version: 2.11.0) (https://github.com/ncbi/sra-tools , and hen extracted FASTQ files from the SRA files with the ‘fastqdump’ function. In CDS-DB, the raw SRA files cannot be obtained for only two RNA-seq source datasets (GSE136114 and GSE191127), and their count data were directly downloaded from GEO. Metadata were manually curated from the corresponding source references, including sample identifier (ID), patient ID, herapeutic regimen and administration dosage, the most fine-grained cancer subtype, drug response status and the response status definition given by the original authors (if available), sampling time and location, pediatric tumor or not, data production platform, source dataset and source publication. ntrinsic drug response status nformation was available for $45\%$ of patients in CDS-DB. Based on the drug response status, we further classified patients into response and non-response groups (see Supplementary methods for the detailed response grouping criteria).  

Source dataset inclusion/exclusion criteria include the following: (i) paired (i.e. from the same patient) pre- and posttreatment patient samples were required; (ii) for microarray data, only datasets from Affymetrix, Agilent and Illumina platforms were collected; (iii) bulk transcriptomes were collected; (iv) datasets, in which drug therapy is combined with other physical herapies, were excluded to ensure hat he perturbation is induced only by drug therapy).  "
19,20,CellCommuNet,Metabolic and Signaling Pathways,Data Refinement and Integration,cellcommunet_refinement,cellcommunet_refinement,Other Structured File,"Perform quality control on a single-cell RNA-seq dataset. 

    Load the raw PBMC3k dataset from `benchmark/dataset/CellCommuNet/pbmc3k_raw.h5ad`. 

    First, filter out cells that express fewer than 200 genes. 

    Second, filter out cells that express more than 2500 genes. 

    Third, calculate the percentage of counts from mitochondrial genes for each cell and remove any cell where this percentage exceeds 5%.

    Finally, save the filtered data as an AnnData object to `<r></r>/cellcommunet_refinement.h5ad`.

    All intermediate results should be stored in the `<t></t>/cellcommunet_refinement` directory.
","The single-cell RNA-seq datasets and associated metadata information (including dataset ID, species, disease type, organ/ issue, therapeutic regime, sequencing platform, etc.) in CellCommuNet were sourced from CancerSCEM, CancerSEA, SCEA and GEO. Datasets associated with metastatic cancers and from patients treated with drug therapy were excluded. After normalization of expression profile data and quality control to remove low-quality cells, a total of $4\ 327\ 804$ cells from 376 datasets across humans and mice were collected into CellCommuNet. These datasets encompass 35 tissue types and 82 disease types. Based on the metadata information of these datasets, we organized 118 comparison datasets between disease and normal samples from the same scRNA-seq study to enable the comparison of cell cell communication between these pairs.  
"
20,21,CellSTAR,Human and Other Vertebrate Geomes,Data Refinement and Integration,cellstar_integration,cellstar_integration,Tabular,"Given a csv file `benchmark/dataset/CellSTAR/cellstar_integration.csv` containing basic metadata of single-cell RNA-seq samples, including species, tissue, and cell type information.

    Standardizing species, tissues, and cell types with taxonomy, Uberon and the Cell Ontology, respectively.

    Saving the standardized metadata in `<r></r>/cellstar_integration.csv`.

    Output format (for reference):
    species,tissue,cell type
    axolotl,bladder,acinic cell
    African malaria mosquito,haemolymph,adipocyte
    ...

    All intermediate files should be saved in the `<t></t>/cellstar_integration` directory.
","Therefore, careful data cleaning and systematic standardization were conducted on he collected raw data. The standardization process relied on he atest versions of he Taxonomy 23 , Uberon 24 , Cell Ontology 25 , Entrez Gene (26  and WHO ICD-11 for accurate alignment of species, issues, cell types, cell markers and diseases respectively.  
"
21,22,circMine,Nucleotide Sequence,Structured Information Extraction,circmine_extract,circmine_extract,Text,"# Core Task
    Query the NCBI GEO database for Series `GSE101684` to retrieve all associated samples and extract specific **patient-level** clinical metadata (age, gender, disease state, stage, and country). It is critical to ensure that extracted attributes describe the individual patient; specifically, if multiple samples (such as controls or replicates) originate from the same individual, they must be consolidated so that each patient is counted only once in the final dataset.

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `series`: The GEO Series accession (e.g., ""GSE101684"").
- `age`: The patient's age as a numeric value.
- `gender`: The patient's sex (e.g., ""male"", ""female"").
- `disease state`: The full textual description of the disease status.
- `stage`: The clinical tumor stage description.
- `country`: The patient's country of origin.
    - Save Path: Save the result to: `<r></r>/circmine_extract.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/circmine_extract/`

    # Output Example (For Reference Only)
    series,age,gender,disease state,stage,country
    GSE114342,24,male,Early stage lung adenocarcinoma,T1bN0M0 IA,USA
    GSE114342,N/A,female,Non-small cell lung cancer,N/A,China
","For facilitating fast and accurate retrieval, each dataset has been manually annotated with specific pathological and physiological conditions. The pathological and physiological conditions include disease grade and stage, genotype, drug resistance, metastasis, immune, lifestyle, virus infection, age and gender. "
22,23,circMine,Nucleotide Sequence,Open-World Data Retrieval,circmine_retrieval,circmine_retrieval,Database,"# Core Task
    Search the NCBI GEO database for Series entries published from 2021/01/01 to 2021/01/15 that investigate **circular RNA (circRNA) in Homo sapiens**, and analyze each entry to determine if it provides accessible human circular RNA expression data derived from human biological sources.

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all Series obtained from the initial GEO search.
    - Each object must contain exactly two fields:
- `""Series""`: The GEO Series accession number (e.g., ""GSE164803"").
- `""Accepted""`: Boolean value (`true` if the Series provides accessible human circRNA expression data, `false` otherwise).
    - Save Path: Save the complete list to: `<r></r>/circmine_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/circmine_retrieval/`

    # Output Example (For Reference Only)
    [
{
""Series"": ""GSE999999"",
""Accepted"": true
},
{
""Series"": ""GSE888888"",
""Accepted"": false
}
    ]
","To collect the human circRNA transcriptome data, we retrieved the GEO resource (23,24) by searching keywords of ‘[(circRNA OR circular RNA) AND Homo sapiens]’. About 358 candidate datasets had been retrieved, which were made public before 10 April 2021. Moreover, all of these datasets were manually curated by at least two professional curators based on two criteria: (i) the datasets providing circRNA expression data of human samples (including tissue, plasma, exosome and cell line) were included regardless of the high-throughput platforms and (ii) the sample information and expression data of the datasets could be downloaded for further integration. "
23,24,ClinicalOmicsDB,Human Genes and Diseases,Functional Feature Annotation,clinicalomicsdb_annotate,clinicalomicsdb_annotate,Other Structured File,"Analyze the provided gene expression data in `benchmark/dataset/ClinicalOmicsDB/expression_data.csv` to compare expression levels between ""responder"" and ""non-responder"" groups. 

    For each gene, perform a Wilcoxon rank-sum test to get a p-value and calculate the Area Under the Receiver Operating Characteristic (AUROC) score. 

    Save the results as a CSV file to `<r></r>/clinicalomicsdb_annotate.csv`, with columns for ""gene"", ""p_value"", and ""auroc"".

    Output example format:
    ```
    gene,p_value,auroc
    GeneA,0.01,0.85
    GeneB,0.05,0.75
    ```

    All intermediate results should be saved in `<t></t>/clinicalomicsdb_annotate` directory.
","We used the Wilcoxon rank-sum test and the area under receiver operating characteristic (AUROC) analysis to quantify gene expression differences between nonresponders and responders within each dataset. The Wilcoxon rank-sum test implemented in the ‘stats’ R package was used for $P$ value computation, and a sign was assigned based on the direction of the medians according to a comparison of nonresponders versus responders. A higher median for the nonresponder group results in a positive sign $\left(+\right)$ , whereas a higher median for the responder group results in a negative sign $(-)$ . To address multiple hypothesis testing, $P$ values were additionally adjusted using the Benjamini–Hochberg method (23  to control the false discovery rate (FDR). The ‘pROC’ R package was used for AUROC analysis. The ‘direction’ parameter in the ‘roc’ function was used to specify that the nonresponder group was expected to have higher gene expression values.  
"
24,25,COLOCdb,Human and Other Vertebrate Geomes,Data Refinement and Integration,colocdb_refinement,colocdb_refinement,Tabular,"In genomic studies, datasets are often aligned to different reference genome builds (e.g., GRCh37/hg19 vs. GRCh38/hg38). 


    Given `benchmark/dataset/COLOCdb/GCST90000064_buildGRCh37.tsv` file, converting the genomic coordinates of the variants in the input file to the GRCh38 reference build. Skip the variants that cannot be successfully converted.

    Saving the converted results in a new file named `<r></r>/colocdb_refinement.tsv`.

    Output format (for reference):
    variant_id	MarkerName	effect_allele	other_allele	effect_allele_frequency	FreqSE	MinFreq	MaxFreq	beta	standard_error	p_value	Direction	HetISq	HetChiSq	HetDf	HetPVal	n	chromosome	base_pair_location
    rs75333668	1:762320	t	c	0.0031	0.0026	0.0009	0.006	NA	NA	0.624	+-	66	2.941	1	0.08636	644545	1	762320

    All intermediate steps should be saved in `<t></t>/colocdb_refinement` directory.
",All datasets were harmonized to the human genome version GRCh37 (hg19) using the liftover tool provided by the UCSC genome browser (28 
25,26,CompoDynamics,Genomics (non-vertebrate),Functional Feature Annotation,compodynamics_annotate,compodynamics_annotate,Sequence,"Calculate the genome-wide GC content for sequence from gene Accession Number NZ_WHYS01000010.1. 

    This should be done by computing a weighted average of the GC content of all individual CDS features. 

    The weight for each CDS is its length. Use the provided genome sequence `benchmark/dataset/CompoDynamics/GCF_009428885.1_ASM942888v1_cds_from_genomic.fna` and corresponding annotation file `benchmark/dataset/CompoDynamics/GCF_009428885.1_ASM942888v1_genomic.gff`. 

    Save the final calculated numeric value to `<r></r>/compodynamics_annotate.txt`.

    Output format example:
    0.5678

    All temp data should be save in `<t></t>/compodynamics_annotate`.
","For each genome, sequence compositions and features were estimated based on the results of all related CDSs, by adopting the following three strategies: (i) weighted averaging based on CDS length, such as GC content, (ii) direct averaging across all CDSs, such as the averaged CDC and ENC, (iii) calculation based on genome-scale values, such as RSCU. For instance, GC and ${\mathrm{GCl}}/2/3$ (positional GC content at three codon positions) contents were averaged across all CDSs by weights of CDS lengths (this is equal to calculation by concatenating all the CDSs in a genome) and their distributions were represented as boxplots. Mean values of CUB, viz., averaged CDC and ENC, were calculated and their distributions were represented as boxplots. RSCU values were calculated based on total frequencies of each codon within genome."
26,27,CompoDynamics,Genomics (non-vertebrate),Data Refinement and Integration,compodynamics_integration,compodynamics_integration,Sequence,"Process the provided CDS sequences in `benchmark/dataset/CompoDynamics/GCF_009428885.1_ASM942888v1_cds_from_genomic.fna`. 

    Filter out sequences that meet any of the following criteria: 1) length is not a multiple of 3; 2) does not start with an ""ATG"" codon; 3) does not end with a standard stop codon (""TAA"", ""TAG"", or ""TGA""); 4) contains an internal stop codon. 

    Save the remaining valid sequences to `<r></r>/compodynamics_integration.fna`.

    All temp data should be save in `<t></t>/compodynamics_integration`.
","Sequences whose lengths are not multiple of three were removed. Most of CDSs were annotated as canonical, whereas the remaining ones were labelled accordingly if they (i) lack start codon, (ii) lack stop codon or (iii) contain in-frame stop codon."
27,28,COVID-19,Structure,Structured Information Extraction,covid_19_extract,covid_19_extract,Text,"Extract metadata for each sample in GEO dataset GSE153428 and GSE154769. 
    Identify age, gender, clinical phenotypes, drug from the GEO series and sample information.

    Saving the finai results to ``.
    # Output Requirements & File Path
    - Output format: A single JSON list (array) of objects. Use ""N/A"" for missing values. Each object must contain the following keys:
- `""sample""`: The GEO sample accession (e.g., ""GSM101684"").
- `""age""`: The patient's age as a numeric value (Years or Months).
- `""gender""`: The patient's sex (e.g., ""male"", ""female"").
- `""clinical phenotypes""`: The observable characteristics.
- `""drug""`: The drug used to treat the patient.
    - Save Path: Save the complete list to: `<r></r>/covid_19_extract.json`

    # Output Example (For Reference Only)
    [
{
""sample"": ""GSM101684"",
""age"": ""24 Years"",
""gender"": ""male"",
""clinical phenotypes"": ""COVID-19 recovered"",
""drug"": ""IFNB""
},
{
""sample"": ""GSM101685"",
""age"": ""48 Months"",
""gender"": ""female"",
""staclinical phenotypesge"": ""COVID-19 recovered"",
""drug"": ""IFNB""
},
{
""sample"": ""GSM101685"",
""age"": ""48 Months"",
""gender"": ""female"",
""staclinical phenotypesge"": ""COVID-19 recovered"",
""drug"": ""N/A""
}
    ]

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/covid_19_extract`
","In addition, to enable accurate data retrieval on the database, we manually curated each dataset with rich clinical information, such as microbe, clinical phenotype, drug, age and gender.  "
28,29,COVID-19,Structure,Data Refinement and Integration,covid_19_integration,covid_19_integration,Other Structured File,"There are two NCBI GEO dataset GSE153428 and GSE154769. 

    The NCBI GEO attachment files are located at  `benchmark/dataset/COVID-19/GSE153428` and `benchmark/dataset/COVID-19/GSE154769`.

    From this two dataset, find the Affymetrix microarray data and extract the express matrix from corresponding series_matrix file.

    And then, standardize the series matrix by mapping the probe IDs to HGNC Gene Symbols.

    Save the final processed matrix (rows as HGNC Gene Symbols, columns as GEO Accessions) to `<r></r>/covid_19_integration.csv`.

    Ouput format requirements (for reference only):
    ,GSM4644121,GSM4644122,GSM4644123,GSM4644124,GSM4644125,....
    ""1-172,CRYAA"",3.97,4.05,4.17,3.92,3.86,4.17,4.03,3.38,4.07,4.09,4.31,4.05
    ""1-8D,IFITM2"",12.68,13.8,12.93,16.09,16.88,16.88,16.17,16.75,16.92,16.01,16.68,16.9
    1-O,7.33,7.675,7.789999999999999,6.99,7.28,6.94,6.835,7.36,7.449999999999999,6.695,6.92,7.385
    ...

    All temp data should be save in `<t></t>/covid_19_integration`.
","The 95 transcriptomic datasets generated from 15 different high-throughput platforms have great data heterogeneity that hinder them for further applications. To overcome the data heterogeneity, we standardized and normalized the gene identifiers (IDs), sample IDs, and expression values in the datasets by using the R package of org.Hs.eg.db (Supplementary Table S2) and the annotation files in GEO (13). The gene IDs and sample IDs were respectively standardized to the gene symbols in the HUGO Gene Nomenclature Committee (HGNC) (16) and the sample accessions in GEO respectively, while the expression values (such as read counts and fragments per kilobase of exon model per million mapped fragments (FPKM)) in the datasets were standardized to transcripts per million (TPM) and normalized by log2. Approximately $86.32\%$ (82/95), $71.58\%$ (68/95) and $43.16\%$ (41/95) of the datasets were standardized and normalized for the sample IDs, expression values, and gene IDs, respectively. In addition, to enable accurate data retrieval on the database, we manually curated each dataset with rich clinical information, such as microbe, clinical phenotype, drug, age and gender.  
"
29,30,CovPDB,RNA Sequence,Functional Feature Annotation,covpdb_annotate,covpdb_annotate,Text,"Given a pdf content file `benchmark/dataset/CovPDB/pdf_content.md`. 

    Determining the covalent bonding mechanisms based on the given paper. 

    The mechanisms must be one of: Aziridine Opening; Beta-Lactam Addition; Borylation; Composite Reaction; Cyclohemiaminoacetalization; Disulfide Formation; Epoxide Opening; Hemiaminalization; Hemi(thio)acetalization; Imidazolidinone Opening; Imine Condensation; Lactone Addition; Michael Addition; Nucleophilic Acyl Substitution; Nucleophilic Addition to a Double Bond; Nucleophilic Addition to a Triple Bond; Nucleophilic Aliphatic Subsititution; Nucleophilic Aromatic Substitution; Phosphorylation; Ring Opening; Sulfonylation

    Saving the mechanisms into `<r></r>/covpdb_annotate.txt`.  

    # Output Format (For Reference)
    Sulfonylation

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/covpdb_annotate`
","The resulting cP–L complexes were visually inspected to ascertain the presence of a covalent bond between interacting pairs. Finally, the chemical structures (SMILES) and warheads (SMARTS) of the pre-reactive electrophilic ligands as well as the covalent bonding mechanisms to their target proteins were expertly manually annotated. This was performed through extraction of structural information from primary citations associated with a given PDB entry. In a few cases with missing primary citations, the features of pre-reactive electrophilic ligands were nonetheless annotated directly from the PDB structures.  "
30,31,CovPDB,RNA Sequence,Data Refinement and Integration,covpdb_integration,covpdb_integration,Geometric,"For given covalent protein-ligand (cP-L) complexes, perform data enrichment and structural property calculations. 

    The pdb ids are given in the file `benchmark/dataset/CovPDB/cov_pdb_ids.json`, and corresponding structures are located in the directory `benchmark/dataset/CovPDB/complex_structures`.

    Retrieve experimental metadata from the PDB, specifically the structure determination method, resolution, and binding affinity (BindingDB). 

    Calculate the total Solvent Accessible Surface Area (SASA) for the complex structure. 

    Saving the annotated results as a csv file `<r></r>/covpdb_integration_complex.csv`.  

    # Output CSV Format for Complex (For Reference)
    index, PDB_ID, Method, Resolution, Affinity, SASA
    0, ABCD, X-RAY DIFFRACTION, 2.0,IC50 50 nM, 150.5
    0, EGFD, X-RAY DIFFRACTION, 2.0,N/A, 150.5
    ...

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/covpdb_integration`
","For the cP–L complexes of this refined dataset, 3D structure characterization methods, resolutions, and binding affinities were retrieved from PDB (13). Moreover, protein features including official name, symbol, class, synonyms, organism, and sequence were retrieved from PDB (13), UniProt (28), and NCBI Gene (29) records. The acid dissociation constant $(\mathsf{p}K_{\mathrm{a}})$ and solvent accessible surface area (SASA) values of the targetable residues were computationally predicted with PROPKA (30) and FreeSASA (31), respectively."
31,32,CovPDB,RNA Sequence,Open-World Data Retrieval,covpdb_retrieval,covpdb_retrieval,Geometric,"Query the Protein Data Bank (PDB) database. 

    Finding high-quality covalent protein-ligand complexes released between 2018-10-01T00:00:00Z and 2018-10-07T00:00:00Z. 

    Here, covelatent complexes are protein-ligand structures that binding with a covalent bond (distance threshold typically <= 2.0 脜 between protien and ligand). 

    You should only keep structures with a resolution 鈮?2.5 脜. 

    Remove entries where the ligand is a known artifact (refer to the BioLiP artifact file in the path `benchmark/dataset/CovPDB/ligand_list`). 

    Saving the filtered pdb id list to the file `<r></r>/cov_pdb_retrieval.json`.

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/covpdb_retrieval`
","This process was carried out in a semi-automated fashion as illustrated in Figure 1. As a starting point, all registered PDB entries were retrieved on 31 August 2020 and parsed with the help of a custom PyMOL (Schr¨odinger LLC, New York, https://www.schrodinger.com/) Python script for cocrystal covalent structures of target proteins in complex with electrophilic ligands. Because not all PDB structures are suitable for prospective modeling studies, especially those with ambiguous ligand electron densities in the protein binding site, we only kept PDB structures with resolution equal to or better than 2.5 A˚ . To further refine the collected dataset, the list of artifact PDB ligands (http://zhanglab.ccmb.med.umich.edu/ BioLiP/ligand list) compiled by Zhang et al. (27), was used to filter out all retrieved complexes wherein the ligand is biologically irrelevant. Typical artifact ligands include crystallization additives such as glycerol. Additionally, complexes containing covalent cofactors acting as prosthetic groups were discarded (e.g. retinal), and likewise ligands that crosslink two (dis)similar protein chains."
32,33,CROST,Human and Other Vertebrate Geomes,Open-World Data Retrieval,crost_retrieval,crost_retrieval,Database,"# Core Task
    Search PubMed for Homo sapiens spatial transcriptomics literature published between 2021/01/01 and 2021/02/01 to build a comprehensive knowledge base, specifically by executing a query that combines species identifiers ('""Homo sapiens""[Mesh] OR ""human""[Title/Abstract]') with topic-specific keywords ('""spatial transcriptomics""[Title/Abstract] OR ""spatially resolved transcriptomics""[Title/Abstract]'), while strictly excluding non-original research by appending 'NOT (""review""[PT] OR ""editorial""[PT] OR ""comment""[PT] OR ""letter""[PT])'. Filter the collected datasets based on the following criteria: (i) availability of raw sequencing files; (ii) presence of spatial location information or image files.

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets all inclusion and filtering criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/crost_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/crost_retrieval/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""99999999"",
""Accepted"": false
},
{
""PMID"": ""88888888"",
""Accepted"": true
}
    ]
","Spatial transcriptomics datasets were acquired and downloaded from databases such as the National Center for Biotechnology Information (NCBI) (29 , European Bioinformatics Institute (EBI) (30 , China National Center for Bioinformation (CNCB) (31 , Broad Institute Single Cell Portal (32 , $10\times$ Genomics (12  and DNA Data Bank of Japan (DDBJ) (33 . Keyword searches were conducted using terms such as ‘spatial transcriptomics’, spatial transcriptome’, spatial genome’, ‘spatial RNA-seq’ and ‘spatial sequencing.’ Furthermore, additional searches were conducted on PubMed and bioRxiv to enhance the datasets, focusing on relevant literature (the query was performed before June 2023). The collected datasets underwent screening based on the following criteria: (i) availability of original sequencing files and (ii) presence of spatial location information or image files. After a thorough screening, 182 spatial transcriptome datasets, encompassing 1033 samples derived rom eight different species, 35 tissues and 56 diseases, were included.  
"
33,34,CTR-DB,Metabolic and Signaling Pathways,Open-World Data Retrieval,ctr_db_retrieval,ctr_db_retrieval,Database,"# Core Task
    Search GEO for cancer-related datasets involving Docetaxel treatment in human samples ('Homo sapiens') by executing a query that combines the drug identifier (DRUG = 'Docetaxel'), the species limit (SPECIES = '""Homo sapiens""[Organism]'), and cancer-specific descriptors (CANCER = 'cancer OR tumor OR carcinoma OR neoplasm'). The search must be restricted to the Series data type ('""gse""[Filter]') and filtered by a specific publication window from 2021/01/01 to 2021/06/01 using the logic '(""{START_DATE}""[PDAT] : ""{END_DATE}""[PDAT])'. Then manually select qualified datasets that meet five criteria: (1) include baseline (pre-treatment) expression profiles; (2) have a sample size of at least 10; (3) provide raw data files (CEL for microarray, FASTQ for RNA-seq); (4) for microarray data, are restricted to platforms GPL96 [HG-U133A], GPL570 [HGU133 Plus 2], and GPL571 [HG-U133A 2].

# Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""Series""`: Series ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets all inclusion and filtering criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/ctr_db_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/ctr_db_retrieval/`

    # Output Example (For Reference Only)
    [
{
""Series"": ""GSE999999"",
""Accepted"": false
},
{
""Series"": ""GSE888888"",
""Accepted"": true
}
    ]
","For TCGA data, firstly we downloaded the TCGA patient clinical information with the help of ‘TCGAbiolinks’ R package (19). Then according to the inclusion criteria described below, we manually picked qualified patients together with the corresponding drug response information, mainly based on ‘days to drug therapy start’, ‘days to drug therapy end’, ‘days to sample procurement’, ‘drug name’ and ‘measure of response’ fields. We manually recorded metadata of each qualified patient, with data fields the same as those of GEO/ArrayExpress data above. We downloaded expression profiles (count data) of qualified patients from UCSC Xena browser (20). For a patient with multiple sample expression profiles, the expression profile with sample ID containing $\mathrm{\cdot_{01A}},$ was preferred.  
"
34,35,CyanoOmicsDB,Structure,Data Refinement and Integration,cyanoomicsdb_annotate_1,cyanoomicsdb_annotate_1,Tabular,"Extract basic information for each gene from the annotation GFF file `benchmark/dataset/CyanoOmicsDB/ncbi_dataset/data/GCF_000009725.1/demo_genomic.gff`, including locus tags, gene symbols, genomic coordinates, protein ID, and product. 

    Then, for each protein-coding gene, integrate the Pfam accessions from InterProScan. The protein sequence as saved in `benchmark/dataset/CyanoOmicsDB/ncbi_dataset/data/GCF_000009725.1/protein.faa`.

    Finally, aggregate the extracted gene information and functional annotations into a comprehensive master TSV file `<r></r>/cyanoomicsdb_annotate_1.tsv`.

    All temp data should be save in `<t></t>/cyanoomicsdb`.

    The reference format for the output file:
    Locus_Tag       Symbol  Old_Locus_Tag   Location        Protein_ID      Product      Pfam
    SGL_RS01970     -       -       NC_005232.1:77988-78551(+)      WP_223211257.1  hypothetical protein      PF01497
    SGL_RS01975     -       -       NC_005232.1:78616-78828(-)      WP_011153842.1  hypothetical protein      PF11741;PF07715;PF00593
    SGL_RS01980     -       -       NC_005232.1:78981-79160(+)      WP_011153843.1  hypothetical protein      PF12833

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/cyanoomicsdb_annotate_1`
","The metainformation of these genomes was extracted using the NCBI data format tool and formatted as a tab-separated values (TSV) file. The basic information for each gene, including locus tag, gene symbol, old locus tag, genomic location, protein id, and encoding product, was extracted from the general feature format (GFF) files. The Enzyme Commission (EC) (22), Gene Ontology (GO) (23,24), Protein Families Database (Pfam) (25), MetaCyc (26), and Kyoto Encyclopedia of Genes and Genomes (KEGG) (27,28) identifiers for each gene were extracted from the resulting InterProScan outputs. The data for each gene were aggregated as another TSV file (Figure 1 and Supplementary Dataset S1).  
"
35,36,CyanoOmicsDB,Structure,Functional Feature Annotation,cyanoomicsdb_annotate_2,cyanoomicsdb_annotate_2,Sequence,"Given a pair-ended RNA-seq data in the directory `benchmark/dataset/CyanoOmicsDB/toy_srr`.

    Align them to the Synechocystis sp. PCC 6803 reference genome GCF_000009725.1 (`benchmark/dataset/CyanoOmicsDB/ncbi_dataset/data/GCF_000009725.1`). 

    Quantify gene-level raw read counts.

    Saving result to `<r></r>/cyanoomicsdb_annotate_2.txt`

    Output example format (for reference):
    ```
    SGL_RS01258	51
    SGL_RS00455	4
    SGL_RS05634	4
    ...
    ```
","For RNAseq data, raw reads were downloaded from the SRA database, aligned to reference genomes (GCA 000009725.1 and GCF 000009725.1) of Synechoccystis sp. PCC 6803 using the Bowtie2 (32). Raw read counts of genes were computed using the HTSeq-count program (33)."
36,37,CyanoOmicsDB,Structure,Open-World Data Retrieval,cyanoomicsdb_retrieval_2,cyanoomicsdb_retrieval_2,Database,"# Core Task
    Search for the latest published literature related to cyanobacteria to capture the most recent research dynamics in the field. Identify the 5 most recent papers published before 2026-01-01 and extract their Title, Journal, DOI

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers).
- `Title`: The title of the paper.
- `Journal`: The journal name.
- `DOI`: The Digital Object Identifier.
    - Save Path: Save the result to: `<r></r>/cyanoomicsdb_retrieval_2.csv`

    # Output Example (For Reference Only)
    Title,Journal,DOI
    Recent Advances in Cyanobacteria,Journal of Phycology,10.1234/jp.2025.001
    Cyanobacteria Ecology,Nature Microbiology,10.5678/nm.2025.002

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/cyanoomicsdb_retrieval_2`
","Entrez package in Python language script with ‘cyanobacteria’ or ‘cyanobacterium’ as keywords. The recent 200 publications were integrated into a TSV file. For the reference page of each gene, the information was recovered using the same script with the keywords containing both the species name and the locus tag or gene symbol.  
"
37,38,DDA,Human Genes and Diseases,Structured Information Extraction,dda_extract,dda_extract,Database,"# Core Task
    Query the NCBI GEO database for Series `GSE116315`, retrieve all associated samples, and extract specific metadata fields including sample name, project ID, cell line/tissue, organism, treatment, genotype, and technique for each sample.

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `Sample Name`: The GSM accession and title. Connect them using a hyphen (-) in the middle, for example: GSM1234567-Sample1.
- `Source Project ID`: The GEO Series accession (e.g., GSE999999).
- `Cell Line/Tissue`: The specific cell type or tissue used (e.g., HeLa cells).
- `Organism`: The scientific name of the organism (e.g., Homo sapiens).
- `Treatment`: Specifies the experimental treatment or drug applied (e.g., Doxorubicin). For combination therapies involving multiple drugs, use a plus sign (+) as a strict separator without any spaces (e.g., Doxorubicin+Cyclophosphamide); ensure no whitespace exists around the plus sign, as formats like 'Doxorubicin + Cyclophosphamide' are incorrect.
- `Genotype`: The genetic background (e.g., CRISPR-KO).
- `Technique`: The experimental technique used (e.g., ChIP-seq).
    - Save Path: Save the result to: `<r></r>/dda_extract.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/dda_extract/`

    # Output Example (For Reference Only)
    Sample Name,Source Project ID,Cell Line/Tissue,Organism,Treatment,Genotype,Technique
    GSM1234567-Sample1,GSE999999,HeLa cells,Homo sapiens,Doxorubicin,CRISPR-KO,ChIP-seq
    GSM7654321-Sample2,GSE999999,Mouse Liver,Mus musculus,Vehicle control,WT,RNA-seq
","For each dataset, we manually curated meta information (such as cell type, organism, treatment details, genotype, sequencing instrument model, etc) based on the sample descriptions submitted by the authors and supplementary materials from original publication (Figure 1A, step 2).  "
38,39,DDA,Human Genes and Diseases,Data Refinement and Integration,dda_refinement,dda_refinement,Sequence,"Filter the provided FASTQ file (`benchmark/dataset/DDA/reads.fastq.gz`) to extract all reads containing at least four consecutive human telomeric repeats. 

    The G-rich motif to search for is ""TTAGGGTTAGGGTTAGGGTTAGGG"" and the C-rich motif is ""CCCTAACCCTAACCCTAACCCTAA"". 

    Save the filtered reads, in standard FASTQ format, to `<r></r>/dda_refinement.fastq.gz`.

    All intermediate files generated during the filtering process should be stored in `<t></t>/dda_refinement/`.
","Following the pipeline described in the Materials and Methods, C-motif reads and G-motif reads were extracted as potential telomeric reads."
39,40,DDInter,Metabolic and Signaling Pathways,Functional Feature Annotation,ddinter_annotate_1,ddinter_annotate_1,Text,"Reviewing the three drug-drug interactions description in the file `benchmark/dataset/DDInter/DDI.txt` and classified them into different risk levels (Major, Moderate, Minor, and Unknown). 

    Generally, Major interactions were highly clinically significant and the drug combinations should be strictly avoided; Moderate interactions may result in exacerbation of the disease of the patient and/or change in therapy; Minor interactions were minimally clinically significant and usually they do not require changes in therapy; Unknown interactions meant that the interaction description was unavailable or incomplete.

    Finially, saving the classification list into file <r></r>/ddinter_annotate_1.json.

    Output format  (for reference):
    [risk_level_1, risk_level_2, ...]

    All intermediate results should be save in `<t></t>/ddinter_annotate_1` directory.
",". Therefore, the registered pharmacist team from Xiangya hospital intensively reviewed these interactions and classified them into different risk levels (major, moderate, minor, and unknown) as suggested by DRUGDEX (20) and other similar resources."
40,41,DDInter,Metabolic and Signaling Pathways,Functional Feature Annotation,ddinter_annotate_2,ddinter_annotate_2,Text,"Reviewing the three drug-drug interactions description in the file `benchmark/dataset/DDInter/DDI.txt` and annotating their with interaction mechanisms, including Absorption, Distribution, Metabolism, Excretion, Synergy and Antagonism. 

    Saving the classification list into file `<r></r>/ddinter_annotate_2.json`.

    Output format  (for reference): 
    [mechanism_1, mechanism_2, ...] 

    All intermediate results should be save in `<t></t>/ddinter_annotate_2` directory.
","In addition to severity level, each of the interaction entries was also manually annotated with interaction mechanisms, including absorption, distribution, metabolism, excretion, synergy and antagonism. More details of the severity and mechanism annotations have been summarized in the website and Supplementary data.  
"
41,42,DDInter,Metabolic and Signaling Pathways,Data Refinement and Integration,ddinter_integration_1,ddinter_integration_1,Database,"Query DrugBank to obtain generic names and brand names for drug in `benchmark/dataset/DDInter/integration_1.txt`.

    Saving the results to `<r></r>/ddinter_integration_1.csv`

    The output result should following the format (for reference):
    query_name,generic_name,brand_name
    name_1,generic_name_1,brand_name_1
    name_2,generic_name_2,brand_name_2
    name_3,generic_name_3,brand_name_3

    All intermediate results should be save in `<t></t>/ddinter_integration_1` directory.
","We used the drug names to query for standard names (generic names) and synonymies (chemical names, brand names, etc.) in DrugBank and Drugs@FDA. The standardization of drug names and merging of interactions were conducted based on the lexicon of drug synonymies."
42,43,DDInter,Metabolic and Signaling Pathways,Data Refinement and Integration,ddinter_integration_2,ddinter_integration_2,Sequence,"For small-molecule drug entry in `benchmark/dataset/DDInter/integration_2.txt`, collecting the Anatomical Therapeutic Chemical (ATC) code.

    Additionally, annotating each drug with basic chemical information, including molecular formula, molecular weight, IUPAC name, InChI, InChIKey, and canonical SMILES. If none applicable values, using N/A.

    Saving the results to `<r></r>/ddinter_integration_2.csv`

    The output result should following the format (for reference):
    drug,ATC,formula,IUPAC,InChIKey,SMILES
    drug_1,N/A,formula_1,IUPAC_1,InChIKey_1,SMILES_1
    drug_2,ATC_2,formula_2,N/A,InChIKey_2,SMILES_2
    drug_3,ATC_3,N/A,IUPAC_3,InChIKey_3,SMILES_3

    All intermediate results should be save in `<t></t>/ddinter_integration_2` directory.
","The small-molecule drugs in DDInter were also annotated with the basic chemical information, including molecular formula, molecular weight, IUPAC name, InChI, InChI key and canonical SMILES.  "
43,44,DDInter,Metabolic and Signaling Pathways,Open-World Data Retrieval,ddinter_retrieval,ddinter_retrieval,Database,"# Core Task
    Systematically search the PubMed database for research published between 2021/01/01 and 2021/03/01 to identify studies reporting drug-drug interactions (DDIs) involving Ciprofloxacin, acting as either the precipitant or the object drug. The search should employ keywords such as ""Drug Interactions""[MeSH] or ""drug-drug interaction"", while strictly excluding drug-food and drug-gene interactions by filtering out terms like ""Food-Drug Interactions""[MeSH Terms], ""food""[Title], ""Pharmacogenetics""[MeSH Terms], or ""gene""[Title/Abstract]. Following the initial search, conduct a detailed abstract analysis to screen for and isolate literature that specifically focuses on clinical or pharmacological interactions between Ciprofloxacin and other pharmaceutical agents, ensuring all irrelevant topics are removed from the final collection.

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets all inclusion and filtering criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/ddinter_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/ddinter_retrieval/`

# Output Example (For Reference Only)
    [
{
""PMID"": ""99999999"",
""Accepted"": false
},
{
""PMID"": ""88888888"",
""Accepted"": true
}
    ]
","The PubMed database was systematically searched to identify the studies that reported the effects of one drug upon another (as of March 2021). Literature pertaining to drug-food or drug– gene interactions and other irrelevant references were ruled out. Then, all the medication guides that had been approved by the US FDA for avoiding serious adverse events were downloaded from Drugs@FDA, with duplicates excluded."
44,45,DIANA-miTED,Genomics (non-vertebrate),Data Refinement and Integration,diana_mited_refinement,diana_mited_refinement,Sequence,"Given a small RNA sequencing data `benchmark/dataset/DIANA-miTED/sample.100k.fastq.gz`.

    Please first perform a quality control and adapter trimming on the raw reads, then align the preprocessed reads to the human reference genome (GRCh38, chromosome 22) provided in `benchmark/dataset/DIANA-miTED/chr22.fa`

    Then quantify miRNA expression levels with miRNA annotations from miRBase v22 `benchmark/dataset/DIANA-miTED/miRBase_v22`.

    Finally, calculating the proportion of preprocessed reads that are successfully assigned to known miRNAs, and save the ratio in `<r></r>/diana_mited_refinement.txt`.

    Output format (for reference):
    50.5%

    All intermediate files should be saved in the `<t></t>/diana_mited_refinement` directory.
","Pre-processing utilized the external tools FastQC (v0.11.7) (25), DNApi (v1.1) (26), and Cutadapt (v1.16) (27). Raw datasets were quality checked allowing a minimum Phred score of 10, adapters were inferred when not provided and trimmed using an 18 bp minimum allowed read length after trimming. Reads were subsequently aligned to the GRCh38 human genome assembly using Bowtie (v1.1.1) (28), allowing up to five multi-maps per read. Finally, the aligned reads were quantified through miRDeep2 (v0.0.8) (12), using the miRBase v22 hairpin and mature human miRNA forms (29), and allowing one mismatch on the precursor during alignment."
45,46,DIANA-miTED,Genomics (non-vertebrate),Open-World Data Retrieval,diana_mited_retrieval,diana_mited_retrieval,Database,"Search the SRA database for ""Homo sapiens"" datasets with a ""miRNA-Seq"" library strategy between 2021.10.01 and 2021.10.05. 

    Retrieve the accession numbers and sort them alphabetically. 

    Save the accession numbers as a list to a json file named `<r></r>/diana_mited_retrieval.json`

    Output format (for reference):
    [
    'SRRXXXXXXX',
    'SRXXXXXXXX',
    ...
    ]

    All intermediate files should be saved in the `<t></t>/diana_mited_retrieval` directory.
","NCBISRA data were identified by selecting all entries with Library Strategy and organism fields as ‘miRNA-Seq’ and ‘Homo sapiens’, respectively, and retrieved locally. Only datasets comprising tissue or cell line information metadata were included in the analysis."
46,47,DISCO,Protein Sequence,Structured Information Extraction,disco_extract,disco_extract,Text,"# Core Task
    Extract comprehensive sample metadata from the GSE210066 Series, including identifiers (Sample ID, Project ID), biological context (Sample type, Tissue, Disease), and technical details (Platform, RNA Source). Furthermore, carefully analyze the data to determine the total number of high-quality cells successfully captured (#Cell) and the Median UMI count for each sample.

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `Sample ID`: The unique identifier for the sample (e.g., GSM number).
- `Project ID`: The project identifier (e.g., GSE number).
- `Sample type`: The type of sample (e.g., tumor tissue).
- `Tissue`: The tissue of origin (e.g., liver).
- `Disease`: The disease condition (e.g., intrahepatic cholangiocarcinoma).
- `Platform`: The sequencing platform used (e.g., 10x5').
- `RNA Source`: The source of RNA (e.g., cell).
- `#Cell`: The total number of high-quality cells.
- `Median UMI`: The median Unique Molecular Identifier count.
    - Save Path: Save the result to: `<r></r>/disco_extract.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/disco_extract/`

    # Output Example (For Reference Only)
    Sample ID,Project ID,Sample type,Tissue,Disease,Platform,RNA Source,#Cell,Median UMI
    GSM1234567,GSE987654,normal tissue,lung,adenocarcinoma,10x3',nucleus,5000,2500
    GSM7654321,GSE987654,tumor tissue,lung,adenocarcinoma,10x3',nucleus,7200,3100
","For each field, we set up a controlled vocabulary or applied Experimental Factor Ontology (EFO) on the entries therein (17)."
47,48,DISCO,Protein Sequence,Data Refinement and Integration,disco_refinement,disco_refinement,Other Structured File,"Given a sparse matrix representing single-cell data in MatrixMarket format. 

    The data is provided in three files: `benchmark/dataset/DISCO/refinement/matrix.mtx`, `benchmark/dataset/DISCO/refinement/features.tsv`, and `benchmark/dataset/DISCO/refinement/barcodes.tsv`. 

    Assemble them into a single sparse matrix object with appropriate row and column names, and then save this object into a single file in RDS format at `<r></r>/disco_refinement.rds`.

    All intermediate results should be save in `<t></t>/disco_refinement` directory.
"," For the 10x Genomics platform, cell debarcoding and unique molecular identifiers (UMIs) identification was performed using Cell Ranger (version 3.1) with the RNA read offset increased from 26 to 39 in the SC5P-PE mode to remove the TSO sequence. For the Drop-seq platform, demultiplexing was done using Drop-seq tools (version 2.4.0, https://github. com/broadinstitute/Drop-seq). Finally, we employed UMItools (13) for demultiplexing data acquired with other technologies like BD Rhapsody, CEL-seq2, and Seq-Well. After debarcoding, reads tagged with a cell barcode and UMI were mapped to the human reference genome assembly hg38 using STAR (version 020201) (14), and then assigned to the corresponding gene (Ensembl 93) with featureCounts (version 2.0.2) (15). We then performed cell level quality control (QC) by filtering on mitochondrial mRNA counts and unique feature counts"
48,49,dNTPpoolDB,Human Genes and Diseases,Structured Information Extraction,dntppooldb_extract,dntppooldb_extract,Text,"# Core Task
    Analyze the full text of the provided literature to locate and extract dNTP measurement data from tables and text descriptions, including values, dimensions, errors, and experimental details (extraction method, measurement type, organism, and sample source).

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `dNTP`: The specific dNTP measured (e.g., dATP, dTTP).
- `Value`: The measured quantitative value.
- `Dimension`: The unit of measurement (e.g., pmol/10^6 cell).
- `Error`: The error or standard deviation associated with the measurement.
- `Extraction method`: The method used to extract dNTPs (e.g., methanol).
- `Measurement type`: The technique used for measurement (e.g., dNTP incorporation assay).
- `Organism`: The scientific name of the organism.
- `Sample source`: The biological source of the sample (e.g., cell line).
    - Save Path: Save the result to: `<r></r>/dntppooldb_extract.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/dntppooldb_extract/`

    # Output Example (For Reference Only)
    dNTP,Value,Dimension,Error,Extraction method,Measurement type,Organism,Sample source
    dCTP,55,pmol/10^6 cell,2.5,PCA,HPLC,Mus musculus,liver tissue
    dGTP,12.5,pmol/10^6 cell,N/A,PCA,HPLC,Mus musculus,liver tissue
","During data collection, it was a challenge to extract the measured values and their errors from figure diagrams. As shown in Figure 1A, about $70\%$ of the data was graphically presented in the literature. To retrieve graphically presented data, we used WebPlotDigitizer 4.4, a free online tool developed by Ankit Rohatgi (https://automeris. io/WebPlotDigitizer). By converting several thousands of graphical dNTP quantitation points into numerical format, we made these valuable data accessible for subsequent bioinformatic analysis. Graphical to numerical representation conversion is indicated in the entry pages.  
"
49,50,dNTPpoolDB,Human Genes and Diseases,Data Refinement and Integration,dntppooldb_refinement,dntppooldb_refinement,Tabular,"Standardize the data in `benchmark/dataset/dNTPpoolDB/raw_measurements.csv`. 

    Create a new `proportion_percent` column by converting the `proportion` column from a fraction to a percentage. 

    Create a new `fold_change` column by converting the `relative_increase` column (e.g., 0.5 for a 50% increase) to fold change (1 + relative_increase). 

    Save the resulting table, including all original and new columns, to `<r></r>/dntppooldb_refinement.csv`.

    Output example format:
    ```
    sample_id,proportion,relative_increase,proportion_percent,fold_change
    sample_1,0.2,0.5,20.0,1.5
    sample_2,0.3,1.0,30.0,2.0
    ``` 
    All intermediate results should be saved in `<t></t>/dntppooldb_refinement` directory. 
","The Experimental details section indicates the major type of dNTP quantitation. We defined five main methods comprising the enzymatic dNTP incorporation assay, HPLC–UV, LC–MS, radioimmune assay, and thin layer chromatography. Here, we also indicate the method of dNTP extraction from the biological sample. Any alterations from the predefined methods are indicated in the Remarks to measurement type row. The most often used subtypes of dNTP measuring methods are also referred to in this section (e.g. the method of Sherman and Fyfe is commonly used for dNTP incorporation-based measurements (30).  

In the Treatment section, we classified treatments into drug and/or stress treatments and genetic manipulation. Some stress treatments are described in free words as standardization was not always an option. However, in case a chemical was applied on the source organism, we unambiguously specified it using its PubChem link in addition to indicating the drug name as used in the publication. In case the measurement was done following a genetic modification, we indicated the details of this in the Genes and Proteins section. Here, the type of genetic modification/variation (e.g. silencing, overexpression, mutation) and all reported affected genes and proteins are identified using Gene and UniProt IDs, respectively, in addition to indicating the gene and protein names given in the article. If mutated forms of the proteins were applied in the measurement, those are also described. The Treatment details/effects comment potentially also contains information on the effects of the applied mutations or drugs."
50,51,DRMref,Human Genes and Diseases,Functional Feature Annotation,drmref_annotate,drmref_annotate,Other Structured File,"Given a single-cell dataset from `benchmark/dataset/DRMef/pbmc3k_cd4_fixed.h5ad`.

    Read the dataset. It contains 'CD4 T cells' with a pre-annotated observation column named 'group', which has two labels: 'resistant' and 'sensitive'.

    Perform a differential expression analysis (Wilcoxon rank-sum test) to find genes that are upregulated in the 'resistant' group compared to the 'sensitive' group.

    Filter the results to identify significant genes meeting these criteria: Adjusted p-value (Benjamini-Hochberg) < 0.05; Log2 fold change > 1.0.

    Save the names of the identified genes into a CSV file `<r></r>/drmref_annotate.csv`.

    Output example for reference:
    gene 
    gene1
    gene2
    ...

    All intermediate results should be save in `<t></t>/drmref_annotate` directory.
","We performed differential gene expression analysis between the resistant and sensitive cells for each cell type using the FindMarkers function in the Seurat package. The pre- and post-treatment samples were analyzed separately. DEGs with p_val_adj $<0.05$ and $|\mathrm{avg\_log}_{2}\mathrm{FCl}>0.25$ between the resistant and sensitive cells were considered drug resistance-related genes and are displayed in the database.  
"
51,52,FusionNeoAntigen,Human Genes and Diseases,Functional Feature Annotation,fusionneoantigen_annotate,fusionneoantigen_annotate,Sequence,"Extract a fusion junction peptide sequence. 

    Given a full-length fusion protein sequence in `benchmark/dataset/FusionNeoAntigen/full_fusion_protein.fasta` and a breakpoint coordinate in `benchmark/dataset/FusionNeoAntigen/breakpoint.txt`, extract a 27-amino-acid peptide centered on the breakpoint. 

    The peptide should consist of 13 amino acids upstream of the breakpoint, the breakpoint amino acid itself, and 13 amino acids downstream. 

    Save the resulting peptide sequence in FASTA format to `<r></r>/fusionneoantigen_annotate.fasta`. 

    All intermediate files generated during the extraction process should be stored in `<t></t>/fusionneoantigen_annotate/`.
","For more reliable fusion genes, we checked the distance between the two breakpoints in case of intra-chromosomal rearrangements and created fusion sequences when those genes are separated by ${\tt>}100\$ kb. We also selected fusion genes when both breakpoints aligned at the exon junction."
52,53,FusionNeoAntigen,Human Genes and Diseases,Functional Feature Annotation,fusionneoantigen_annotate_2,fusionneoantigen_annotate_2,Geometric,"Given a list of smiles in `benchmark/dataset/FusionNeoAntigen/dock/smiles.txt`.

    Identify the smiles have highest vina affinity to protein `benchmark/dataset/FusionNeoAntigen/dock/1a1e_protein.pdb`.

    Docking pocket and size are listed in `benchmark/dataset/FusionNeoAntigen/dock/pocket_config.txt`.

    Save the final smiles `<r></r>/fusionneoantigen_annotate_2.txt`. 

    Output example for reference:
    ```
    cccccc
    ```

    All intermediate files generated during the extraction process should be stored in `<t></t>/fusionneoantigen_annotate_2/`.
"," Since the minimum length required for the accurate predictions of protein structure through RoseTTAFold, was 14AA, we made the fusion breakpoint 14AA peptide sequences of $\sim10\mathrm{K}$ fusion breakpoints (Supplementary Table S3) that have the potential fusion neoantigens from running the binding affinitybased prediction tools. We made the library of 3D structures of ${\sim}10\mathrm{K}$ predicted neoantigens or he neoantigen-HLA docking interaction study.  
"
53,54,FusionNeoAntigen,Human Genes and Diseases,Structured Information Extraction,fusionneoantigen_extract,fusionneoantigen_extract,Text,"# Core Task
    Search the PubMed database for downloadable literature published between 2010/01/01 and 2025/01/01 to identify studies focused on BCR-ABL1 fusion gene neoantigens. The search should be executed using a comprehensive query string: (""BCR-ABL1""[All Fields] OR ""BCR-ABL""[All Fields] OR ""BCR/ABL""[All Fields]) AND (""neoantigen""[All Fields] OR ""neo-antigen""[All Fields] OR ""neoepitope""[All Fields] OR ""tumor specific antigen""[All Fields]). Following the retrieval, carefully analyze the full text or abstracts of each article to verify that it provides an extensive discussion of neoantigens specifically derived from the BCR-ABL1 fusion event, filtering out studies where these terms appear only incidentally.

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PMC search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string.
- `""Accepted""`: Boolean value (`true` if the article confirms research on BCR-ABL1 fusion neoantigens, `false` otherwise).
    - Save Path: Save the complete list to: `<r></r>/fusionneoantigen_extract.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/fusionneoantigen_extract/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""40459423"",
""Accepted"": false
},
{
""PMID"": ""35456656"",
""Accepted"": true
}
    ]
","For this, we made a Python script to grab the HTML page information of the PMC searching results and parsed the number of studies and PMIDs. Then, we manually checked those papers on he usion neoantigens. While we searched he web using our script, we considered gene synonyms also. Ultimately, there were only 56 previous studies on fusion neoantigens for 33 fusion genes.  "
54,55,GPEdit,Metabolic and Signaling Pathways,Data Refinement and Integration,gpedit_refinement,gpedit_refinement,Tabular,"Identify expression-derived Quantitative Trait Loci (edQTLs) that overlap with Genome-Wide Association Study (GWAS) tag SNPs. 

    Using the provided GWAS SNP list in `benchmark/dataset/GPEdit/gwas_snps.tsv` and the edQTL list in `benchmark/dataset/GPEdit/edqtls.tsv`, filter for edQTLs whose genomic range (`start` to `end` on a specific `chr`) contains a GWAS SNP''s position (`pos`). 

    Save the complete rows of the identified overlapping edQTLs to a tab-separated file at `<r></r>/gpedit_refinement.tsv`.

    Output example format:
    ```
    chr	start	end	edqtl_id	gene
    chr1	100000	105000	edqtl_1	GeneA
    chr2	200000	205000	edqtl_2	GeneB
    ``` 

    All intermediate results should be saved in `<t></t>/gpedit_refinement` directory.
","To emphasize the cis-regulatory role of detected edQTLs, only SNPs within $200~\mathrm{kb}$ of editing sites were included in our analysis (14). "
55,56,iNClusive,Protein Sequence,Structured Information Extraction,inclusive_extract_1,inclusive_extract_1,Text,"# Core Task
    You are a specialized Bioinformatics NLP Agent acting as an expert in Synthetic Biology and Genetic Code Expansion (GCE). Your core competency is the high-fidelity extraction of experimental data from scientific literature regarding the site-specific incorporation of non-canonical amino acids (ncAAs). You will process specific PDF publications to curate a structured dataset of orthogonal translation systems (aaRS/tRNA pairs), their origins, and their applications in protein engineering.

    # Workflow Instructions
    - `Literature Processing`: Parse the provided PDF files located at the specified path.
    - `Data Extraction & Formatting`:
1) ncAA Identifiers: Extract abbreviations and full names. Use ""not available"" if missing.
2) aaRS ID Construction: Follow the syntax [Org Abbr]-[Natural Substrate]RS ([Mutations/Wild Type]). (e.g., Ec-MetRS (wild type) or Mb-PylRS (Y271C, N311Q, Y349F, V366C)).
3) tRNA ID Construction: Follow the syntax [Org Abbr]-tRNA [Natural AA] [Anticodon]. (e.g., Bs-tRNA Tyr CUA). Each tRNA has a name composed of three words: (a) abbreviation of the organism from which it was derived; (b) tRNA; (c) AA naturally transported by the tRNA; (d) anticodon. For example, 'Bs-tRNA Tyr CUA' indicates a tRNA naturally found in Bacillus subtilis, which transports tyrosine, whose anticodon is CUA. 
4) Organism Mapping: Identify full scientific names for both aaRS and tRNA origins.
5) Protein Application: Identify the target protein. Critical: If multiple proteins are tested in one paper, create a unique row for each protein.
6) Site Mapping: If multiple independent sites are tested, separate with ; and sort by residue number (ascending). If multiple sites are incorporated simultaneously (jointly), use +.
7) Host System: Identify the test organism or specify ""in vitro"".

    # Input File Path
    Input Path: `benchmark/dataset/iNClusive/` (Files: paper_data_17_21.pdf, paper_data_947.pdf, paper_data_966_1011.pdf, paper_data_1748.pdf, paper_data_2126_2127.pdf)

    # Output Requirements & File Path
    Generate a CSV file using a semicolon (;) as the delimiter for each input PDF file. 
    Output Path: <r></r>/inclusive_extract_1/
    Naming Convention: Output filenames must match the input PDF filename (e.g., paper_data_17_21.pdf results in paper_data_17_21.csv).
    The CSV must contain the following columns exactly as named:
""ncAA abbreviation(s) used in the publication""
""ncAA name, as mentioned in the publication""
""aaRS ID (abbr. organism, abbr. natural substrate, RS, mutations if any)""
""aaRS origin organism full name""
""tRNA ID (abbr. organism, tRNA, natural AA transported, anticodon)""
""tRNA organism""
""Tested in (protein)""
""Tested in (protein position)""
""Tested in (organism/in vitro)""

    # Output Example (For Reference Only)
    ncAA abbreviation(s) used in the publication;ncAA name, as mentioned in the publication;aaRS ID (abbr. organism, abbr. natural substrate, RS, mutations if any);aaRS origin organism full name;tRNA ID (abbr. organism, tRNA, natural AA transported, anticodon);tRNA organism;Tested in (protein);Tested in (protein position);Tested in (organism/in vitro)
    not available;p-acetyl-L-phenylalanine;Ec-TyrRS (mutant);Escherichia coli;Ec-tRNA Tyr CUA;Escherichia coli;human superoxide dismutase 1 (hSOD);33;Saccharomyces cerevisiae

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/inclusive_extract_1`
","From each publication, we retrieved the following information:  

1) abbreviation and name of ncAA   
2) organism of origin and natural substrate of the aaRS used to incorporate the ncAA   
3) mutations applied to the aaRS. Note that we decided to assign to each aaRS a unique name composed of the following nformation: abbreviation of he organism of origin; abbreviation of the natural substrate of the aaRS; the letters ‘RS’; and, if applicable, the mutations carried by the synthetase. For example, Ec-MetRS is the aaRS from E. coli that naturally loads methionine on the corresponding tRNA   
4) name of RNA used or he ncorporation. We decided o assign to each tRNA a name composed of three words: (a) abbreviation of the organism from which it was derived; (b) tRNA; (c) AA naturally transported by the tRNA. For example, ‘Bs-tRNA Tyr’ indicates a tRNA naturally found in Bacillus subtilis, which transports tyrosine   
5) codon recognized by the tRNA   
6) modifications made to the tRNA (if any)   
7) protein in which the ncAA was incorporated   
8) position of incorporation (if given)   
9) organism in which incorporation was tested, if any (could be in vitro   
0) application for the ncAA (if given)   
11) original publication in APA citation style   
2) DOI link to the publication  "
56,57,iNClusive,Protein Sequence,Functional Feature Annotation,inclusive_extract_2,inclusive_extract_2,Text,"# Core Task
    You are a specialized Bioinformatics NLP Agent acting as an expert in Chemical Biology and Genetic Code Expansion (GCE). Your core competency is the high-fidelity extraction of chemical and experimental data from scientific literature regarding non-canonical amino acids (ncAAs). You will process specific PDF publications to identify ncAAs, their nomenclature, and their corresponding chemical structures represented as SMILES notations.

    # Workflow Instructions
    - `Literature Processing`: Parse the provided PDF files located at `benchmark/dataset/iNClusive/`.
    - `Information Extraction & Chemical Identification`:
1) ncAA Abbreviation: Extract the short-hand abbreviation(s) for the ncAAs used in the publication. If not found, use ""not available"".
2) ncAA Name: Extract the full chemical or common name as mentioned in the text. If not found, use ""not available"".
3) SMILES Extraction: Identify and extract the SMILES (Simplified Molecular Input Line Entry System) string for each corresponding ncAA. This may require cross-referencing names/structures mentioned in the text with chemical databases if the string is not explicitly provided.

    # Input File Path
    Input Path: `benchmark/dataset/iNClusive/` (Files: paper_data_25.pdf, paper_data_73.pdf, paper_data_249_274.pdf, paper_data_342.pdf, paper_data_2228_2230.pdf)

    # Output Requirements & File Path
    For each input PDF, generate a separate CSV file using a semicolon (;) as the delimiter.
    Output Path: <r></r>/inclusive_extract_2/
    Naming Convention: Output filenames must match the input PDF filename (e.g., paper_data_25.pdf results in paper_data_25.csv).
    Three Columns: ""ncAA abbreviation(s) used in the publication"", ""ncAA name, as mentioned in the publication"", ""ncAA SMILES notation""

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/inclusive_extract_2`
","Also in this case, we had to read many publications and retrieve the sequence from other databases or repositories e.g. GenBank 34 or Addgene)   
3) molecular-input line-entry system (SMILES) and IUPAC name of the ncAA. To this aim, we uploaded images of the ncAAs into the ‘Optical Structure Recognition A (OSRA)’ software (32  and compared the generated SMILES with the entries of the PubChem database (33 . Some of the ncAAs could unfortunately not be found on PubChem, but we have nevertheless provided the (SMILES) and the chemical formulas for these   
4) AA closest in structure to the ncAA   
5) comments (if applicable). These include, for example, Addgene inks, nformation on flanking sequences on he tRNA, or specific conditions necessary for the incorporation of the ncAA  "
57,58,iNClusive,Protein Sequence,Data Refinement and Integration,inclusive_extract_3,inclusive_extract_3,Text,"# Core Task
    You are a specialized Bioinformatics Agent acting as an expert in Genetic Code Expansion (GCE) and Molecular Cloning. Your core competency is identifying genetic components (plasmids, genes, and non-coding RNAs) mentioned in scientific literature and retrieving their full biological sequences from public repositories such as Addgene and GenBank. You will process a specific PDF to locate plasmid nomenclature and cross-reference these clues to extract the complete protein sequences for aminoacyl-tRNA synthetases (aaRS) and the complete RNA sequences for tRNAs.

    # Workflow Instructions
    - `Literature Mining`: Analyze the provided PDF file to identify plasmid names, accession numbers, or specific synthetic biology constructs related to the orthogonal translation system.
    - `Database Retrieval`: Using the identified clues, query public databases (Addgene, NCBI GenBank, or UniProt) to retrieve:
1) The full-length protein sequence of the aaRS (including any specified mutations);
2) The full-length RNA sequence of the suppressor tRNA.
    - `Entity Identification & Formatting`: 
1) aaRS ID: Construct using the syntax [Org Abbr]-[Natural Substrate]RS ([Mutations/Wild Type]). (e.g., Ec-MetRS (wild type)).
2) tRNA ID: Construct using the syntax [Org Abbr]-tRNA [Natural AA] [Anticodon]. (e.g., Bs-tRNA Tyr CUA).
    - `Data Structuring`: Map the retrieved sequences to their respective IDs into two separate dictionaries within a single JSON object.

    # Input File Path
    Input Path: benchmark/dataset/iNClusive/science.1060077.pdf

    # Output Requirements & File Path
    Generate a JSON file containing A dictionary with two keys where one key is the aaRS ID and its value is the full protein sequence (string), and the other one key is tRNA ID and its value is the full RNA sequence (string).
    Save Path: <r></r>/inclusive_extract_3.json

    # Output Example (For Reference Only)
    {
""Ec-MetRS (L13G)"": ""KAKHGKGVLGPKPEAGKAKPAAKPA"",
""Ec-tRNA Met CAT"": ""CCGGCGGTA"",
    }

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/inclusive_extract_3`
","e additionally collected the following information:  

1) sequence of the aaRS. This was a very time-consuming and laborious task, because the sequences are typically not found in the publications themselves, but had to be retrieved elsewhere with nformation provided n he materials and methods section e.g. plasmid name and number as deposited in a public repository such as Addgene (https://www.addgene.org/ )  

![](images/ae12ef6fe17ffa2919b481dfe4d609ebb9db01354796fb9c3c840a4fedf3fc06.jpg)  
Figure 2. Scheme depicting he workflow used o generate he NClusive database. The hits were generated using he Publish or Perish’ software 31 upstream of Google Scholar. The publications were hen manually eviewed and nserted nto a preliminary database. The SMILES of he ncAAs were generated with he Optical Structure Recognition OSRA) software 32 and compared with hose given n he PubChem database 33 The etrieved information was confirmed manually by a different eam member and only hen added nto he NClusive database. A second eview process done by a third distinct person was additionally carried out on $10\%$ of andomly picked entries.   
Figure 2 shows the entire workflow used to create iNClusive.  

2) sequence of the tRNA. Also in this case, we had to read many publications and retrieve the sequence from other databases or repositories e.g. GenBank 34 or Addgene)   
3) molecular-input line-entry system (SMILES) and IUPAC name of the ncAA. To this aim, we uploaded images of the ncAAs into the ‘Optical Structure Recognition A (OSRA)’ software (32  and compared the generated SMILES with the entries of the PubChem database (33 . Some of the ncAAs could unfortunately not be found on PubChem, but we have nevertheless provided the (SMILES) and the chemical formulas for these   
4) AA closest in structure to the ncAA   
5) comments (if applicable). These include, for example, Addgene inks, nformation on flanking sequences on he tRNA, or specific conditions necessary for the incorporation of the ncAA  "
58,59,iNClusive,Protein Sequence,Open-World Data Retrieval,inclusive_retrieval,inclusive_retrieval,Database,"# Core Task
    You are a specialized Bioinformatics Literature Agent acting as an expert in Non-Canonical Amino Acid (ncAA) Proteomics. Your core competency is conducting systematic literature reviews and cross-referencing multi-disciplinary keywords to identify original research articles focused on the incorporation of non-standard amino acids into proteins, specifically those utilizing mass spectrometry for characterization.

    # Workflow Instructions
    - `Keyword Expansion & Search`: Execute a comprehensive search (e.g., via Google Scholar) using combinations of the following thematic clusters:
1) Detection Method: ""Mass spectrometry"" OR ""Electrospray ionization"" OR ""Electrospray ionisation"" OR ""MALDI"" OR ""MS/MS"" OR ""LC MS"" OR ""GC MS"" OR ""HPLC MS"".
2) Core Subject: ""Unnatural Amino Acid"" (UAA) OR ""non-canonical Amino Acid"" OR ""noncanonical Amino Acid"" OR ""non canonical Amino Acid"" (ncAA) OR ""non-standard Amino Acid"" OR ""nonstandard Amino Acid"" OR ""non standard Amino Acid"" (nsAA).
3) Biological Mechanism: ""tRNA Synthetase"".
    - `Temporal Filtering`: Retain only those articles published on or before September 22, 2023.
    - `Document Type Exclusion`: Rigorously screen the results to exclude: Duplicate entries; Review articles (Reviews); Patents; Theses or Dissertations (Master's or PhD); Non-peer-reviewed preprints.
    - Keep only peer-reviewed original research.
    - `Link Extraction`: Extract the permanent DOI URL for each qualifying publication.

    # Output Requirements & File Path
    Generate a JSON file containing a flat list of strings. Each string must be the full DOI link.
    Save Path: `<r></r>/inclusive_retrieval.json`

    # Output Example (For Reference Only)
    ['https://doi.org/10.1038/s41467-021-27025-w', 'https://doi.org/10.1038/s41929-019-0410-8', 'https://doi.org/10.1038/s41467-025-63733-3']

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/inclusive_retrieval`
","To retrieve these publications, we used the ‘Publish or Perish’ software (31  to search Google Scholar for terms typical of studies nvolving ncAAs ncorporated nto proteins and MS as validation method Figure 2 . Publish or Perish’ allows setting keywords and saving the bibliographic data of the papers as comma-separated value files, which can be later used for downstream analysis.  
As there is no standardized nomenclature in the field (e.g. some researchers speak of ncAAs, while other of non-standard AAs; unnatural AAs might not be included in the term ncAAs etc.) and considering that the same term could be written differently in different journals (for instance noncanonical vs non-canonical), we used several combinations of search terms to retrieve the publications from which to extract the data Table 1 . Before processing he hits or he database, duplicates, as well as reviews, patents and master or doctoral theses were removed, leaving 687 peer-reviewed publications to manually analyse.  
"
59,60,Kincore,Protein Sequence,Data Refinement and Integration,kincore_renumbering,kincore_renumbering,Geometric,"# Core Task
    You are a specialized Bioinformatics Agent acting as an expert in Structural Kinomics and Residue Numbering. Your core competency is the structural decomposition of PDB files and the application of universal numbering schemes to protein kinases. You will process PDB ID `9D8U`, isolate its kinase chains from the asymmetric unit, and renumber the residues according to a structure-based universal alignment scheme.

    # Workflow Instructions
    - `Structural Decomposition`: Retrieve and parse the PDB file for entry `9D8U`. Split the asymmetric unit into individual protein chains. Remove all non-protein components including ligands, ions, and water molecules.
    - `Mapping Configuration`: Utilize the local reference file `benchmark/dataset/Kincore/All-organisms-alignment-residue-corresspondence.tab` which contains the structure-based multiple sequence alignment (MSA) data.
1) Key Columns: Use UniNum (the 1-based UniProt residue index) and AlignNum (the universal residue numbering index).
2) Domain Scope: Ensure residues fall within the range defined by DomainBegin and DomainEnd.
    - `Renumbering Logic`:
1) For each isolated protein chain, identify the corresponding UniProt entry.
2) Map each residue's original PDB position to its UniNum, then reassign the residue index to its corresponding AlignNum.
3) Filtering: Retain only the residues that have a valid mapping to the universal numbering scheme (AlignNum) in the final output.
    - `Aggregation`: Process each chain independently. Combine all renumbered chains into a single multi-chain PDB file.

    # Output Requirements & File Path
    Generate a single PDB file containing all renumbered kinase chains.
    Residue ID: Must reflect the AlignNum from the mapping file.
    Format: Standard PDB format.
    Save Path: <r></r>/kincore_renumbering.pdb

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/kincore_renumbering`
","The structure files are split into individual kinase chains in the asymmetric unit and renumbered according to their UniProt sequences. The necessary alignments are obtained from the Structure Integration with Function, Taxonomy and Sequence (SIFTS) database (37). The SIFTS files were also used to extract mutation, phosphorylation and missing residue annotations.  
"
60,61,Kincore,Protein Sequence,Open-World Data Retrieval,kincore_retrieval,kincore_retrieval,Geometric,"# Core Task
    ou are a specialized Bioinformatics Agent acting as an expert in Kinase Structural Bioinformatics and Sequence Homology. Your core competency is performing sensitive sequence searches using Position-Specific Iterative BLAST (PSI-BLAST) to identify protein chains belonging to the same family as a reference kinase. You will utilize a PSSM-based query against a localized version of the PISCES pdbaa database to retrieve and format a comprehensive list of Aurora A kinase-related protein chains.

    # Workflow Instructions
    - `PSSM Construction`: Use the protein sequence of the human Aurora A kinase (residues 125-391) as the initial query. Run three iterations of NCBI PSI-BLAST with default cutoff values against the PDB database to build a Position-Specific Scoring Matrix (PSSM).
    - `Database Search`: Use the generated PSSM to perform a command-line PSI-BLAST search against the local pdbaa.txt file. 
Input Database: The pdbaa file (located at `benchmark/dataset/Kincore/pdbaa.txt`) contains FASTA sequences for every chain in the PDB asymmetric units, including metadata like resolution, R-factor, and SwissProt identifiers (e.g., AURKA_HUMAN).
    - `Hit Processing`: Export up to the top 10,000 hits in XML format and extract the PDB ID and Chain ID for every identified hit.
    - `Formatting`: Format each entry as a string combining the the Chain ID and lowercase PDB ID (e.g., [PDB_ID]_[Chain_ID]).

    # Output Requirements & File Path
    Generate a JSON file containing a flat list of strings representing the identified protein chains.
    Format example: [""4pjs_A"", ""1ol5_A""]
    Constraints: PDB IDs must be in lowercase.
    Save Path: <r></r>/kincore_retrieval.json

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/kincore_retrieval`
","The database contains protein kinase domains from Homo sapiens and nine model organisms. To identify structures from these organisms the sequence of human Aurora A kinase (residues 125–391) was used to construct a PSSM matrix from three iterations of NCBI PSI-BLAST on the PDB with default cutoff values (35). This PSSM matrix is used as query to run command line PSI-BLAST on the pdbaa file from the PISCES server (http://dunbrack.fccc.edu/pisces) (36). pdbaa contains the sequence of every chain in every asymmetric unit of the PDB in FASTA format with resolution, R-factors and SwissProt identifiers (e.g. AURKA HUMAN). We include only typical protein kinases and exclude ‘atypical kinases’ as defined by UniProt (https: //www.uniprot.org/docs/pkinfam).  
"
61,62,M2OR,Human Genes and Diseases,Functional Feature Annotation,m2or_annotate,m2or_annotate,Sequence,"Given a FASTA file `benchmark/dataset/M2OR/mutant_M2OR.fasta` containing a mutated sequence of the human Muscarinic Acetylcholine Receptor M2. 

    Identify the single point mutation by comparing it against the canonical wild-type sequence (UniProt accession: P08172). 

    Saving the result to `<r></r>/m2or_annotate.json` with `wild_type_sequence` key and `mutation`.

    Output example (for reference):
    {
""wild_type_sequence"": ""FAFGSFAFGSFAFGSFAFGSFAFGSFAFGSFAFGSFAFGSFAFGSFAFGSFAFGS"",
""A100G""
    }

    All intermediate results should be saved to `<t></t>/m2or_annotate`
","Protein sequence serves as a standardized OR identifier in M2OR. If it is not available in the original publication, it is retrieved from UniProt (70  using the name or other identifier provided by the authors. For mutated receptors, he wild-type identifier and sequence is retrieved, and the mutation is separately indicated using XpositionY format where amino acid $X$ from the wild-type sequence is mutated to amino acid $Y$ at he given position Additions are indicated by setting $X$ o $\cdot_{+},$ and deletion with $\cdot_{-}\cdot$ as Y  The taxon name of the species from which the OR originates is also included.  
"
62,63,M2OR,Human Genes and Diseases,Functional Feature Annotation,m2or_refinement,m2or_refinement,Geometric,"Convert the molecular structure provided in the MOL file `benchmark/dataset/M2OR/1a1e_ligand.mol2` into its standard InChIKey.
    Save the resulting InChIKey string into the output file `<r></r>/m2or_refinement.txt`.
    All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/m2or_refinement/`","The International Chemistry Identifier Key (InChIKey) (68) serves as a standardized molecular identifier in the M2OR database. It is retrieved from PubChem (69  using the provided molecular identifiers found in the articles. Identifiers used by the authors were preferentially kept as stated in the orignial publication, and other identifiers, such as CID, were inferred from PubChem using InChIKey. Name and CAS are preferably retained as they are in the source, although there might be some inconsistencies due to CAS being a proprietary identifier and name not standardized. If only the molecular structure s available, a SMILES representation s nferred and further used to search for the corresponding InChIKey. n one study, he authors used a newly-synthesised molecule which is only identified by its SMILES string.  
"
63,64,MACC,Metabolic and Signaling Pathways,Structured Information Extraction,macc_extract,macc_extract,Text,"# Core Task
Retrieve the abstracts for the articles with PMIDs 19033652, 19949098, and 10483897, and extract all instances of Metabolite-associated cell communications (MACC pairs). Each extracted MACC pair must be formatted as a csv file containing exactly the following fields: ""PMID"", ""Cell"", ""Interaction"", ""Metabolite"", and ""Disease"".

    # Output Requirements & File Path
- Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `PMID`: The PubMed ID of the article.
- `Cell`: The specific cell type described (e.g., ""Macrophage"", ""Neuron"").
- `Interaction`: The nature of the interaction (e.g., ""Secrete"", ""Inhibit"").
- `Metabolite`: The name of the metabolite involved (e.g., ""Glucose"", ""ATP"").
- `Disease`: The disease context (e.g., ""Diabetes"", ""Cancer"").
- Save Path: Save the complete list to: `<r></r>/macc_extract.csv`
- Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/macc_extract/`

    # Example Output (For Reference Only)
    PMID,Cell,Interaction,Metabolite,Disease
    22222222,Hepatocyte,Synthesize,Urea,Liver Failure
    33333333,Adipocyte,Store,Lipid,Obesity
","At last, wo independent curators carefully evaluated ${>}30$ thousand abstracts, resulting into highquality MACC pairs.  "
64,65,mBodyMap,Structure,Data Refinement and Integration,mbodymap_integration,mbodymap_integration,Database,"Given a list NCBI GEO Sample accession number ``, identifying and filtering for paired-end 16S rRNA SRA raw sequencing data.

    Saving the accession number of identifier 16S rRNA SRA raw sequencing runs to `<r></r>/mbodymap_integration.json`

    Output example for reference:
    ```
    [
""SRR1"",
""SRR2""
    ]
    ```

    All temp data should be save in `<t></t>/mbodymap_integration`.
","Next, we downloaded the raw sequencing data from EBI ENA (41) and NCBI SRA (42) (Sequence Read Archive, https://www.ncbi.nlm.nih.gov/sra) using enaBrowserTools (https://github.com/enasequence/enaBrowserTools) and SRA-Tools (https://github.com/ncbi/sra-tools) facilitated by Aspera (a high-speed data transfer tool)."
65,66,MetazExp,Genomics (non-vertebrate),Functional Feature Annotation,metazexp_annotate,metazexp_annotate,Other Structured File,"
    Perform a differential expression analysis for 6 DRX experiments in `benchmark/dataset/MetazExp/count`. 

    The basic information of the 6 DRX experiments is provided in `benchmark/dataset/MetazExp/sample_info.tsv`, which includes sample annotations specifying biological conditions (e.g., tissue, developmental stage) and study or batch identifiers.

    You must explicitly correct for study-specific batch effects by including the batch variable in the design formula (e.g., design = ~ batch + condition).

    Identify genes that are significantly differentially expressed in the target condition compared to all other conditions combined.

    Output the top 50 most significantly differentially expressed genes (ranked by adjusted p-value) to `<r></r>/metazexp_annotate.tsv`.

    Output Format (Only for Reference):
    gene_id	baseMean	log2FoldChange	lfcSE	stat	pvalue	padj
    FBgnxxxx	1254.66665476386	-2.87772911936465	0.87905045247991	-27.2422049305457	2.05567383473253e-163	3.25310384346423e-159
    FBgnyyyy	52136.516872705251	-4.730648430188	0.676023255336235	-24.3843527676287	2.50655533940541e-131	1.98331191230453e-127

    All temp data should be save in `<t></t>/metazexp_annotate`.
"," Two commonly used differential expression methods, DESeq2 (21) and edgeR (22), were used to compare overall gene expression while statistical models implemented in rMATS were used to compare alternative splicing. When possible (study not completely confounded with treatment), batch effects due to studies were adjusted using DESeq2 or edgeR. In addition to comparing expression and alternative splicing across two groups of samples, condition specific (e.g. tissue specific) expression or alternative splicing was identified if a gene’s expression or PSI was higher/lower than all other conditions.  "
66,67,MetazExp,Genomics (non-vertebrate),Data Refinement and Integration,metazexp_refinement,metazexp_refinement,Sequence,"Identify high-quality RNA-seq Runs from three Runs in `benchmark/dataset/MetazExp/toy_srr1`, `benchmark/dataset/MetazExp/toy_srr2`, and `benchmark/dataset/MetazExp/toy_srr3`.

    High-quality RNA-seq Runs should meet all of the following criteria: (i) Paired-end sequencing; (ii) Read length 鈮?100 bp; (iii) Uniquely mapped read rate 鈮?50% for the reference genome `benchmark/dataset/MetazExp/ncbi_dataset/data/GCF_001039355.2/GCF_001039355.2_LinAna2.0_genomic.fna`. 

    Output the directory nname of high-qualit Runs to `<r></r>/metazexp_refinement.json`.

    Output Format (Only for Reference):
    [
""toy_srr2""
    ]

    All temp data should be save in `<t></t>/etazexp_refinemen`.
","The genome sequences and reference annotations of 69 and 3 species were obtained from the Ensembl (10) and RefSeq (11) databases, respectively (Table 1). Except for a few well annotated genomes, the annotations of most species were largely incomplete. For example, in 33 species, no alternatively spliced transcripts were annotated in multi-exon genes (Supplementary Table S2). We adopted a previously described procedure (8) to improve genome annotation and obtain a uniform annotation for each species against which mapping will be performed. Briefly, high coverage RNAseq data curated manually were mapped to the reference genome using HISAT2 (12). We defined high coverage, high quality data as those that were paired end, at least 100 bp in read length, at least $50\%$ unique mapping rate and at least a certain sequencing depth. The requirement for sequencing depth varied depending on data availability in each species. In Drosophila melanogaster, we required that at least 4G bases were sequenced. The alignments produced by HISAT2 were assembled into reference guided gene models in GTF format using StringTie2 (13). The resulting GTFs were compared iteratively with the merged GTF using cuffcompare (1). In each iteration, novel multi-exonic transcripts that were at least 200 bp long with at least $2\mathrm{x}$ coverage per transcript and $1\times$ per exon for all exons were merged to the GTF. Finally, all unannotated transcripts must occur in at least three experiments and account for at least $50\%$ experiments of any tissue or at least one-third of all experiments.  
"
67,68,MVIP,Structure,Functional Feature Annotation,mvip_annotate,mvip_annotate,Text,"There are 5 summary infromation about NCBI GEO datasets in the file `benchmark/dataset/MVIP/gds_test_data.txt`. 

    Based on these information, fetching out the high throughput sequencing dataset certainly related to virus infection. 

    Saving the keeped dataset idx list [index_1, index_2, ...] in <r></r>/mvip_annotate.tsv. 

    Output format example:
    [
""1"",
""2"",
""3"",
""4"",
""5""
    ]

    All temp data should be save in `<t></t>/mvip_annotate`.
","We firstly searched NCBI GEO DataSets database with keywords ‘virus’ and ‘seq’ up to September 2019 using Entrez E-utility. We further filtered these accessions by requiring the presence of a keyword ‘virus’, ‘viruses’ and ‘viral’ appear in its GEO summary ignoring cases (Figure 1A). "
68,69,NPCDR,Metabolic and Signaling Pathways,Structured Information Extraction,npcdr_extract_1,npcdr_extract_1,Text,"# Core Task
    Analyze the literature to classify the primary therapeutic effect of the natural product-based drug combination into one of three categories: (1) enhancing drug efficacy by augmenting sensitivity and achieving synergy, (2) decreasing adverse drug reactions, or (3) reversing drug resistance.
    Literature to analyze: `dataset\NPCDR\27980427.pdf`, `dataset\NPCDR\25653233.pdf`, `dataset\NPCDR\23429291.pdf`, `dataset\NPCDR\21088503.pdf`, `dataset\NPCDR\32872376.pdf`, `dataset\NPCDR\29626307.pdf`

    # Output Requirements & File Path
    - Output format: A JSON list of dictionaries with ""PMID"" and ""TypeID"". 
    - ""PMID"" is the filename, and ""TypeID"" is the classification category mentioned above (int).
    - Save Path: Save the complete list to: `<r></r>/npcdr_extract_1.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/npcdr_extract_1/`

    # Example Output (For Reference Only)
    [
{
""PMID"": ""15380427"",
""TypeID"": 2
},
{
""PMID"": ""25653532"",
""TypeID"": 1
}
    ]
","irst, a number of clinically important drugs were collected from DrugBank (42) and TTD (43), and a systematic literature review on the NPs that were reported to combine with these drugs was conducted. As a result, the collected NP-based drug combinations (as shown in Figure 1) were found to (a) enhance drug efficacy by augmenting drug sensitivity (44,45) and achieving therapeutic synergy (46,47), (b) decrease the adverse drug reaction (48,49) and (c) reverse drug resistance (50,51). Second, the molecules or pathways regulated by these collected combinations were manually identified by additional literature review, and their regulation profiles (expression up/down-regulation, increased/decreased phosphorylation, etc.) were explicitly provided (shown in Figure 1)."
69,70,NPCDR,Metabolic and Signaling Pathways,Structured Information Extraction,npcdr_extract_2,npcdr_extract_2,Text,"# Core Task
Systematically extract all molecular regulation events described in the provided research paper and organize the output into a structured format containing the Regulation Direction, the Mechanism Type, and the Standardized Symbol for each molecule.
Literature to analyze: `dataset\NPCDR\paper.pdf`

    # Output Requirements & File Path
-  Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `Direction`: Standardize all values to either ""Up-regulation"" or ""Down-regulation"" based on the experimental results.
- `Type`: Specify the precise biological mechanism described (e.g., ""Expression"", ""Phosphorylation"", ""Activation"", ""Cleavage"", or ""Translocation"").
- `Symbol`: Convert the common names or aliases used in the text (e.g., ""p53"", ""EGFR"", ""TNF-alpha"") into their corresponding official HGNC approved gene/protein symbols (e.g., ""TP53"", ""EGFR"", ""TNF"") using all-uppercase letters to ensure database compatibility and uniqueness.
- Save Path: Save the complete list to: `<r></r>/npcdr_extract_2.csv`
- Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/npcdr_extract_2/`

    # Example Output (For Reference Only)
    Symbol,Direction,Type
    TP53,Up-regulation,Expression
    EGFR,Down-regulation,Phosphorylation
    TNF,Down-regulation,Expression
","Moreover, a variety of experimental techniques that were applied to identify the molecular and biological regulations were also recorded, which included shRNA, siRNA, western-blot, qPCR, etc., and the analytical results of various experiments were recorded to give comprehensive information for each combination, and the extended descriptions on each regulated molecule can be accessed by clicking the ‘Molecule Info’ buttons given in Figure 3. Additionally, the pathways altered by the particular drug combination were also identified by the literature review."
70,71,NPCDR,Metabolic and Signaling Pathways,Open-World Data Retrieval,npcdr_retrieval,npcdr_retrieval,Database,"# Core Task
    Search PubMed for Review articles published between November 1, 2025, and November 5, 2025, that investigate drug combinations or synergistic effects involving the natural product curcumin. The search should be executed using the following specific query: (""Curcumin""[Mesh] OR Curcumin[Title/Abstract]) AND (""Drug Therapy, Combination""[Mesh] OR ""Drug Synergism""[Mesh] OR combin*[Title/Abstract] OR synerg*[Title/Abstract]). Following the search, it is essential to perform a detailed analysis of the retrieved abstracts to manually verify and retain only those articles that explicitly mention drug combinations involving curcumin. 

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets all inclusion and filtering criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/npcdr_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/npcdr_retrieval/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""99999999"",
""Accepted"": false
},
{
""PMID"": ""88888888"",
""Accepted"": true
}
    ]
","Third, NP-based drug combinations were collected by the literature review in PubMed (55) using such keyword combinations: ‘[NP name] $^+$ drug combination’, ‘[NP name] $^+$ combination’, ‘[NP name] $^+$ synergistic effects’, ‘[NP name] $^+$ synergy’, ‘natural product $^+$ [drug name]’, and so on"
71,72,OncoDB,Metabolic and Signaling Pathways,Functional Feature Annotation,oncodb_annotate,oncodb_annotate,Tabular,"Perform a Cox Proportional Hazards regression analysis on the provided `benchmark/dataset/OncoDB/rossi_dataset.csv`. 

    The model should use `week` as the duration column and `arrest` as the event column. 

    All other columns should be used as covariates. 

    Save the resulting regression summary table to a CSV file named `<r></r>/oncodb_annotate.csv`.

    Output example:
    ```
    covariate,coef,exp(coef),se(coef),coef lower 95%,coef upper 95%,exp(coef) lower 95%,exp(coef) upper 95%,cmp to,z,p,-log2(p)
    age,0.02,1.02,0.01,0.00,0.04,1.00,1.04,age,2.00,0.045,-4.47
    ``` 

    All intermidate results should be saved to `<t></t>/oncodb_annotate` directory.
","$P$ -value from the log-rank test and hazard ratio from Cox proportional regression analysis. An example of the survival plot is shown in Figure 2E. In case the selected gene is significantly correlated with one or more clinical parameters, a table will also be presented to summarize the statistical analysis results.  
"
72,73,PCMDB,Human and Other Vertebrate Geomes,Structured Information Extraction,pcmdb_extract,pcmdb_extract,Text,"# Core Task
    Analyze the provided literature to extract comprehensive plant cell marker information, ensuring that data is gathered from the main text, tables, and figure legends. For each entry, you must identify the Species (scientific name), Tissue Type, Cell Marker Name, Symbol/ID (standardized to 'AT' format), and Cell Type. Strict Data Filtering: Do not retain any records that are incomplete, ambiguous, or contain missing values (N/A); specifically, discard any markers labeled as ""novel,"" ""newly identified,"" ""unnamed,"" or those lacking a confirmed 'AT' format ID. Only extract entries where all five fields are explicitly defined and valid.
    Literature to analyze: `dataset/PCMDB/30923229.pdf`

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `Species`: The scientific name of the organism (e.g., Arabidopsis thaliana).
- `Tissue Type`: The tissue origin of the sample (e.g., Root).
- `Cell Marker Name`: The name of the cell marker gene or protein. 
- `Symbol/ID`: The gene symbol or ID (standardized to 'AT' format).
- `Cell Type`: The specific cell type associated with the marker.
    - Save Path: Save the result to: `<r></r>/pcmdb_extract.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/pcmdb_extract/`

    # Output Example (For Reference Only)
    Species,Tissue Type,Cell Marker Name,Symbol/ID,Cell Type
    Plantus fakeus,Leaf,FAKEMARKER1,AT9G99999,Guard cells
    Plantus fakeus,Stem,MYB-FAKE,AT8G88888,Vascular cells
"," We then further carefully checked the entire text and confirmed the annotation/function in the official plant websites, and finally extracted relevant information, including the cell marker name, cell marker ID, and cell/tissue type for each plant (Supplementary Table S2, round 3).  "
73,74,PCMDB,Human and Other Vertebrate Geomes,Structured Information Extraction,pcmdb_extract_2,pcmdb_extract_2,Text,"There are three paper contents `example_1.md, example_2.md, example_3.md` in the directory `benchmark/dataset/PCMDB`. 

    Filter out the paper research contain cell marker inferred through biological experiments, including GFP reporter systems, GUS gene reporter systems and western blotting. 

    Saving the keeped paper name list in the `<r></r>/pcmdb_extract_2.json`. 

    Output format: A JSON file with the following format:
    [name_1, name_2, ...] 

    Paper name must not contain suffix.

    All intermediate results should be save in `<t></t>pcmdb_extract_2` directory.
","Then, we generally checked the entire text and selected the studies containing cell markers or genes inferred through biological experiments, including GFP reporter systems, GUS gene reporter systems and western blotting."
74,75,PGS-Depot,Human and Other Vertebrate Geomes,Data Refinement and Integration,pgs_depot_refinement,pgs_depot_refinement,Tabular,"Filter the provided SNP data in `benchmark/dataset/PGS-Depot/sample.vcf`. 

    The filtering criteria are: remove multi-allelic SNPs, remove SNPs with a minor allele frequency (MAF) less than 0.05, and remove SNPs with a genotype missing rate greater than 0.2. 

    Save the resulting filtered data as a VCF file named `<r></r>/pgs_depot_refinement.vcf`.

    All intermediate steps should be saved in `<t></t>/pgs_depot_refinement` directory.
","For SNP QC, we retained SNPs with a genotype calling confidence larger than 0.9. We filtered out SNPs i) with a MAF $<0.01$ , ii) with a Hardy–Weinberg equilibrium (HWE) test $P$ value $<10^{-7}$ (iii) with an imputation information score $<0.8$ , (iv) with a proportion of missingness $(P_{\mathrm{m}})>0.05$ or (v) that are a duplicated SNP. "
75,76,PharmGWAS,Human and Other Vertebrate Geomes,Structured Information Extraction,pharmgwas_extract,pharmgwas_extract,Other Structured File,"# Core Task
    Search the GWAS Catalog to identify and integrate GWAS datasets and corresponding articles with keywords ""Atrial Fibrillation"" that are published in 2025, roughly extracting some dataset details.

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers). Use ""N/A"" for missing values.
- `PMID`: The PubMed ID of the associated publication.
- `Accession`: The unique identifier of the dataset (e.g., GCST number).
    - Save Path: Save the result to: `<r></r>/pharmgwas_extract.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/pharmgwas_extract/`

    # Output Example (For Reference Only)
    PMID,Accession
    99999999,GCST88888888
    88888888,GCST77777777
","We also collected metadata such as sample size, ancestry and release year of the datasets from GWAS Catalog or manually reviewed the original publications. The summary of GWAS datasets is shown in Table 1  "
76,77,PLAbDab,Protein Sequence,Functional Feature Annotation,plabdab_annotate_1,plabdab_annotate_1,Sequence,"# Core Task
    You are a specialized Bioinformatics Agent acting as an expert in antibody sequence annotation and structural numbering. Your core competency is utilizing the ANARCI (Antibody Numbering and Antigen Receptor ClassIfication) tool to parse sequence data, apply standardized numbering schemes, and extract structural loop metrics. Process a FASTA file of antibody sequences using ANARCI to identify chain types, apply IMGT numbering, and calculate CDR lengths.

    # Workflow Instructions
    - `Sequence Parsing & Numbering`: The input is a FASTA file containing antibody sequences with Accession ID in the headers. Please use the `IMGT` numbering scheme.
    - `Chain Classification`: Classify each sequence based on the ANARCI output. Map results to a binary category: H (Heavy Chain) or L (Light Chain).
    - `CDR Length Calculation`: Identify the residue positions for CDR1, CDR2, and CDR3 according to the `IMGT` definition. Calculate the length for each region by counting residues only (exclude gaps).

    # Output Requirements & File Path
    Generate a CSV file with the following columns:
    - ""Accession"": The Accession ID from the FASTA header.
    - ""Numbered_Sequence"": The sequence string with `IMGT` numbering gaps.
    - ""CDR_Lengths"": The formatted length string (e.g., 8_8_12, which means the length of CDR1 is 8, CDR2 is 8, and CDR3 is 12).
    - ""Chain_Type"": The identified chain (either H or L).
    Input Path: Read the input file from: `benchmark/dataset/PLAbDab/plabdab_annotate_1_input.fasta`
    Save Path: Save the result to: `<r></r>/plabdab_annotate_1.csv`

    # Output Example (For Reference Only)
    Accession,Raw_Sequence,Numbered_Sequence,CDR_Lengths,Chain_Type
    QPY89049.1,EIVLTQSPGTLSLSPGERATLSCRASQSVGSSYLAWYQQKPGQAPRLLIYGAFSRATGIPDRFSGSGSGTDFTLTISRLEPEDFAVYYCQQYGSSPWTFGQGTKVEIK,EIVLTQSPGTLSLSPGERATLSCRASQSVG-----SSYLAWYQQKPGQAPRLLIYGA-------FSRATGIP-DRFSGSG--SGTDFTLTISRLEPEDFAVYYCQQYGS----SPWTFGQGTKVEIK,7_3_9,L
    QEP20941.1,DIQMTQSPSSLSASVGDRVTITCRASQGISNSLAWYQQKPGKAPNLLLYAASRLESGVPSRFSGSGSGTDYTLTISSLQPEDFATYYCQQYYRTPLTFGGGSKVEIK,DIQMTQSPSSLSASVGDRVTITCRASQGI------SNSLAWYQQKPGKAPNLLLYAA-------SRLESGVP-SRFSGSG--SGTDYTLTISSLQPEDFATYYCQQYYR----TPLTFGGGSKVEIK,6_3_9,L


    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/plabdab_annotate_1`
","Sequences for each of these 2.5 million entries were then searched for antibody variable domain sequences using ANARCI (22 . Sequences missing residues in the CDR regions were removed at this stage.
"
77,78,PLAbDab,Protein Sequence,Structured Information Extraction,plabdab_annotate_2,plabdab_annotate_2,Text,"# Core Task
    You are a specialized Bioinformatics NLP Agent expert in Antibody-Antigen entity recognition. Your core competency is parsing biomedical literature titles to identify the specific target antigen of an antibody. You must distinguish between the subject (the antibody) and the object (the antigen) and return the standardized abbreviation or common name of the targeted molecule.

    # Workflow Instructions
    - `Data Loading`: Read the input CSV containing ""ID"" and ""Title"" columns.
    - `Antigen Entity Recognition`: For each ""Title"", identify the specific antigen being targeted.
    - `Target Mention Labeling`: Extract the standardized antigen abbreviation or the single word (e.g., use ""VEGF"" for ""Vascular Endothelial Growth Factor"") where possible. If a Title describes a method or polypeptide improvement without a specific antigen target mentioned, return ""None"".
    - `Table Construction`: Map the extracted ""target_mention"" back to its original ID.

    # Output Requirements & File Path
    Generate a CSV file with exactly two columns:
    - `ID`: The original unique identifier from the input file.
    - `target_mention`: The identified antigen abbreviation, single word or ""None"".

    Input Path: Read the input file from: `benchmark/dataset/PLAbDab/plabdab_annotate_2_input.csv`
    Save Path: Save the result to: `<r></r>/plabdab_annotate_2.csv`

    # Output Example (For Reference Only)
    [""DEFB1"", ""PD-1"", ""CHL1"", ""None""]

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/plabdab_annotate_2`
","The target of the majority of antibodies in PLAbDab can be inferred from a manual review of their source literature. Although possible, this process can be time-consuming. To reduce the number of entries to review, we provide a ‘targets_mentioned’ column that lists common antigens mentioned in the source material.  

For patent entries, potential antigens were identified by searching the title, abstract, and claims sections for mentions of commonly recognised antigen names. When a term in the source iterature was prefixed with anti-’, t was also ncluded in he ist of possible antigens. For non-patent entries, only he title was searched for potential targets.  

The use of he targets_mentioned’ abel exhibits high recall: when an antibody’s antigen is known, t is often mentioned in the analysed text. However, its accuracy is moderate. Accurate labelling of each entry with antigen information requires manual inspection. Nonetheless, this approach offers a fast and straightforward method or generating an nitial reference set.  "
78,79,PLAbDab,Protein Sequence,Open-World Data Retrieval,plabdab_retrieval,plabdab_retrieval,Database,"# Core Task
    You are a specialized Bioinformatics expert in Python, the BioPython library, and the NCBI Entrez API. Your goal is to execute precise data mining and sequence processing tasks with high rigor and error handling. You must adhere strictly to the user's filtering criteria and output specifications to ensure data integrity for downstream biological analysis. Extract and filter human antibody-related protein sequences from the NCBI Protein database and export them into a structured FASTA file.

    # Workflow Instructions
    - `Database Query`: Use the Bio.Entrez module to search the protein database. Construct a query using the following keywords: ""antibody"", ""antibodies"", ""immunoglobulin"", ""scfv"", or ""bcr"". Apply these keywords across all fields.
    - `Exclusion Filtering (Keyword-based)`: Remove any entries containing the terms ""nanobody"" or ""nanobodies"" to prevent the inclusion of humanized nanobodies in the heavy chain dataset.
    - `Length Filtering`: Filter sequences longer than 1000 amino acids or shorter than 70 are also removed.
    - `Sampling & Ranking:`: Identify the most relevant hits based on the search relevance score. Cap the final dataset at a maximum of 10,000 unique hit sequences.

    # Output Requirements & File Path
    - Output format: Generate a final output in FASTA format. Each entry must include: Header: ""Hit ID"" and ""Description""; Body: The full protein sequence. 
    - Save Path: Save the result to: `<r></r>/antibody_seq_retrieval.fasta`

    # Output Example (For Reference Only)
    >XP_057929505.2 sema domain, immunoglobulin domain (Ig), short basic domain, secreted, (semaphorin) 3Fa isoform X3 [Doryrhamphus excisus]
    MLRDSLCCLLALLASASGALPPANEPLSAPRIFLSFKELKSTGTAHHFSFLLNSTDYRIL
    >sp|Q8IWA5.3|CTL2_HUMAN RecName: Full=Choline transporter-like protein 2; AltName: Full=Solute carrier family 44 member 2
    MGDERPHYYGKHGTPQKYDPTFKGPIYNRGCTDIICCVFLLLAIVGYVAVGIIAWTHGDP

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/plabdab_retrieval`
","The majority of data n PLAbDab s extracted rom he Protein database of NCBI (20 . The BioPython Entrez module (21  is used to query the database for entries containing the words ‘antibody’ , ‘antibodies’, ‘immunoglobulin’, ‘scfv’ or ‘bcr’ in any of their fields. Due to the lack of an automated method to accurately differentiate humanised nanobodies from human antibody heavy chains, entries containing the words nanobody or nanobodies are removed at this stage. Entries with sequences longer than 1000 amino acids or shorter than 70 are also removed. Around 2.5 million entries were returned from this search.  
"
79,80,PlantPAD,Plant,Functional Feature Annotation,plantpad_annotate,plantpad_annotate,Image,"Annotate the disease feature in the images in the directory `benchmark/dataset/PlantPAD`.

    Three features should be identified, including Color Of Diseases, Texture Of Diseases, and Leaf Morphology.

    Optional values for each features:
Color Of Diseases: Black, Green, Yellow, Brown, Gray, Red Brown, White
Texture Of Diseases: Spotted, Striped, Ring Spot, Netted Spot, Random Spot
Leaf Morphology: Atrophy, Wilt, Rot, Burn, Perforation

    Saving the results to `<r></r>/plantpad_annotate.csv`.

    The output result should following the format (for reference):
    image,color,texture,morphology
    image_1.jpg,Black,Spotted,Atrophy
    image_2.jpg,Green,Striped,Wilt
    image_3.jpg,Yellow,Ring Spot,Rot

    All intermediate results should be save in `<t></t>/plantpad_annotate` directory.
","Within PlantPAD three distinct categories of descriptive information pertaining to plant disease images exist. Firstly, we have the image metadata, encompassing fundamental attributes shared by all images. This comprises data on the image’s source, format, dimensions, resolution and sharpness. These metadata are vital for providing a foundational understanding of each image. Secondly, textual annotations represent another critical facet. These annotations are meticulously structured extual data, ailored o align with he common eatures and search criteria used in plant disease identification. They serve as a valuable resource for users, facilitating swift access to plant diseases of interest or those requiring diagnosis. Lastly, an innovative data type is introduced, involving textual descriptions of features exhibited within the images. These descriptions encompass various aspects such as disease morphology, coloration and spatial location."
80,81,PncsHub,Structure,Functional Feature Annotation,pncshub_annotate,pncshub_annotate,Text,"Based on the paper content in `benchmark/dataset/PncsHub/paper_conent.md`. 

    Determine determine the Pathway Type of katG. There are eight types: FPE銆丗EA銆丠olins銆丮Vs銆丼ecA2銆乀7SS, ABC transptor, Unknown. 

    Saving the result to <r></r>/pncshub_annotate.txt.

    Output format example:
    FPE

    All temp data should be save in `<t></t>/pncshub_annotate`.
","Finally, the other known sub-types are separated into the remaining six sub-types (right): ABC Transporter, flagella export apparatus (FEA), fimbrillin-protein exporter (FPE), Holin, SecA2 (the alternate Sec pathway), and type VII secretion system (T7SS) pathways, and in addition to the 27 overlapping proteins that are also secreted by MV, there is one protein that can be secreted by both T7SS and SecA2.  "
81,82,ProNAB,Human Genes and Diseases,Structured Information Extraction,pronab_extract,pronab_extract,Text,"# Core Task
    Extract protein-nucleic acid binding data from the provided literature, specifically identifying Key fields including UniProt ID, protein mutation(s), nucleic acid name, nucleic acid type, PDB complex ID, experimental method, and thermodynamic parameters (dG and Kd).
    Literature to analyze: `dataset/ProNAB/10383437.pdf`

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers):
- `UniProt Id`: The UniProt ID of the protein.
- `Mutation Protein`: The mutation(s) in the protein (e.g., 'R355A' or 'wild').
- `Nucleic Acid Name`: The name of the nucleic acid.
- `Type Nuc`: The type of nucleic acid (e.g., 'DNA').
- `PBD Compelx`: The PDB ID of the complex.
- `Method`: The experimental method used.
- `dG wild(Kcal/mol)`: The dG value for the wild type.
- `Kd wild(M)`: The Kd value for the wild type.
    - Save Path: Save the result to: `<r></r>/pronab_extract.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/pronab_extract/`

    # Output Example (For Reference Only)
    UniProt Id,Mutation Protein,Nucleic Acid Name,Type Nuc,PBD Compelx,Method,dG wild(Kcal/mol),Kd wild(M)
    P99999,wild,""Generic Promoter Region"",DNA,9xyz,""Fluorescence Anisotropy"",-9.5,1.5e-08
    P99999,H100A,""Generic Promoter Region"",DNA,9xyz,""Fluorescence Anisotropy"",-10.2,2.1e-09
","From each article, we manually curated the information about the name of the protein, nucleic acid, complex, experimental conditions, measurement, method, thermodynamic data, literature information and location of the data in the research article.  "
82,83,RAVAR,Human and Other Vertebrate Geomes,Structured Information Extraction,ravar_extract,ravar_extract,Text,"# Core Task
    Perform a detailed analysis of the provided literature (`dataset/RAVAR/32436959.pdf`) to identify rare variant-trait associations. Extract the standardized 'EFO Trait Label' for each identified association. Evaluate the statistical significance of each association against the following thresholds: gene-level tests (P < 1E-4) and SNP-level tests (P < 1E-6). Associations meeting these significance criteria should be marked as accepted.

    # Output Requirements & File Path
    - Output format: A JSON list of objects.
    - Each object must contain exactly two fields:
- `""EFO Trait Label""`: The standardized Experimental Factor Ontology trait label as a string.
- `""Accepted""`: Boolean value (`true` if the association meets the specified significance threshold, `false` otherwise).
    - Save Path: Save the result to: `<r></r>/ravar_extract.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/ravar_extract/`

    # Output Example (For Reference Only)
    [
{
""EFO Trait Label"": ""systolic blood pressure"",
""Accepted"": true
},
{
""EFO Trait Label"": ""total cholesterol"",
""Accepted"": false
}
    ]
","For each qualified publication, we extract association nformation ncluding he reported raits, association esting method/software, and P-value indicating statistical significance. For variant-level associations, we additionally record the minor allele frequency (MAF), beta values indicating variant effect sizes, $95\%$ confidence interval, and mapped genes for each variant. Overall, only results with $P$ value less than 1E-4 for gene-level associations or 1E-6 for SNP-level associations are included. Comprehensive information for all qualified publications included in the RAVAR database is listed in Supplementary Table S2.  
"
83,84,RAVAR,Human and Other Vertebrate Geomes,Data Refinement and Integration,ravar_refinement_1,ravar_refinement_1,Other Structured File,"Given a list of collected traits in the `benchmark/dataset/RAVAR/reported_trails.json`.

    Standardizing the collected traits by deleting publication specific symbols and ambiguous language. So that we can unify the terminology, definitions, and classification of traits. 

    You should return a mapping dict save in the `<r></r>/ravar_refinement_1.json`. 

    The mapping dict should have the following format:
    {
""original_trait_1"": ""standardized_trait_1"",
""original_trait_2"": ""standardized_trait_2"",
...
    }

    All intermediate steps should be saved in `<t></t>/ravar_refinement_1` directory.
","To unify the terminology, definitions, and classification of traits, the reported trait/disease information from each publication is standardized by manually mapping terms to the Experimental Factor Ontology (EFO) (28  and establishing well-defined classification criteria. First, we organize and standardize the collected traits by manually deleting publicationspecific symbols and ambiguous language. Additionally, we convert any abbreviations to their complete forms to improve the accuracy of mapped terms. It is worth noting that some UKBB phenotypes have been known to cause confusion among researchers due o ambiguities n heir naming conventions. We have made extensive efforts to address these issues and ensure accurate trait labeling throughout the database. For example, we describe the trait ‘gamma glutamyltransferase’ in UKBB with the more descriptive term ‘gamma glutamyltransferase measurement’ to improve clarity. In detail, we define originally reported raits’ as raits extracted directly from relevant literature. ‘Reported traits’ refers to traits after trait name preprocessing, and trait abel’ and EFO rait abel’ are the labels mapped to the EFO. We also reclassify reported traits that are excessively confusing or difficult to interpret under the label ‘Other traits’. Ontology mapping and classification are conducted manually on the corrected labels using the EFO hierarchy ree. To optimize searchability and nteroperability, we annotate each rait with crucial nformation rom the original publication ncluding he originally-reported rait, mapped EFO rait abel, related publications, ontology D, description, trait synonyms, and mapped terms. According to the ontological nformation described above, raits are categorized into the five distinct subcategories of material property, information entity, material entity, process and other traits. These categories enable users to more easily locate and identify raits of nterest, hereby acilitating he exploration of he dataset.  
"
84,85,RAVAR,Human and Other Vertebrate Geomes,Data Refinement and Integration,ravar_refinement_2,ravar_refinement_2,Tabular,"According to the ontological information provided in the file `benchmark/dataset/RAVAR/trails_data.csv`,  categorizing the traits into the 3 distinct subcategories of material property, information entity, and material entity. 

    Saving the classification result as a json list [subcategory_1, subcategory_2, ...] to `<r></r>/ravar_refinement_2.json`.

    Ouput format example:
    [""material property"", ""information entity"", ""material entity""]

    All intermediate results should be saved in `<t></t>/ravar_refinement_2` directory.
","To unify the terminology, definitions, and classification of traits, the reported trait/disease information from each publication is standardized by manually mapping terms to the Experimental Factor Ontology (EFO) (28  and establishing well-defined classification criteria. First, we organize and standardize the collected traits by manually deleting publicationspecific symbols and ambiguous language. Additionally, we convert any abbreviations to their complete forms to improve the accuracy of mapped terms. It is worth noting that some UKBB phenotypes have been known to cause confusion among researchers due o ambiguities n heir naming conventions. We have made extensive efforts to address these issues and ensure accurate trait labeling throughout the database. For example, we describe the trait ‘gamma glutamyltransferase’ in UKBB with the more descriptive term ‘gamma glutamyltransferase measurement’ to improve clarity. In detail, we define originally reported raits’ as raits extracted directly from relevant literature. ‘Reported traits’ refers to traits after trait name preprocessing, and trait abel’ and EFO rait abel’ are the labels mapped to the EFO. We also reclassify reported traits that are excessively confusing or difficult to interpret under the label ‘Other traits’. Ontology mapping and classification are conducted manually on the corrected labels using the EFO hierarchy ree. To optimize searchability and nteroperability, we annotate each rait with crucial nformation rom the original publication ncluding he originally-reported rait, mapped EFO rait abel, related publications, ontology D, description, trait synonyms, and mapped terms. According to the ontological nformation described above, raits are categorized into the five distinct subcategories of material property, information entity, material entity, process and other traits. These categories enable users to more easily locate and identify raits of nterest, hereby acilitating he exploration of he dataset.  
"
85,86,SCAN,Human and Other Vertebrate Geomes,Open-World Data Retrieval,scan_retrieval,scan_retrieval,Database,"# Core Task
    Search PubMed to identify studies providing publicly available Homo sapiens single-cell or single-nucleus RNA sequencing datasets for brain, spinal cord, retina, and embryo samples. The search should be executed by constructing a multi-dimensional query that combines species identifiers (SPECIES_QUERY = '""Homo sapiens""[Mesh] OR ""human""[Title/Abstract]'), sequencing methodologies (METHOD_QUERY = '""single-cell RNA sequencing""[Title/Abstract] OR ""scRNA-seq""[Title/Abstract] OR ""single-nucleus RNA sequencing""[Title/Abstract] OR ""snRNA-seq""[Title/Abstract]'), and target tissues (TISSUE_QUERY = '""brain"" OR ""spinal cord"" OR ""retina"" OR ""embryo""[Title/Abstract]'). To ensure the identification of actual data resources, include availability descriptors (DATA_AVAILABILITY_QUERY = '""dataset"" OR ""datasets""[Title/Abstract] OR ""GSE"" OR ""accession number""[All Fields]'). The search must be strictly restricted to the publication window from 2021/01/01 to 2021/04/01 using the logic '(""{START_DATE}""[PDAT] : ""{END_DATE}""[PDAT])'.Select studies that meet the following criteria: (i) availability of detailed sample records; (ii) availability of sequencing library construction methods; (iii) availability of cell type annotation files or supporting information such as expected cell types or markers.

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets all inclusion and filtering criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/scan_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/scan_retrieval/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""99999999"",
""Accepted"": false
},
{
""PMID"": ""88888888"",
""Accepted"": true
}
    ]
","We collected single-cell sequencing datasets by manually searching PubMed with the following keywords (‘brain scRNAseq’, ‘spinal cord scRNAseq’, ‘retina scRNAseq’, ‘embryo scRNAseq’, ‘brain snRNAseq’, ‘spinal cord snRNAseq’, ‘retina snRNAseq’ and ‘embryo snRNAseq’) for publicly available and integrated datasets from multiple databases, ncluding NCBI/GEO 47 , EMBL-EBI https://www.ebi.ac.uk/ (48 , descartes (49 50 , UCSC Cell Browser (23 , FishSCT (51 , Single Cell Portal (https://singlecell.broadinstitute.org/ single_cell/  and $10\times$ Genomics (https://www.10xgenomics. com/). The following criteria were considered for single-cell sequencing dataset selection: i) availability of sample records, (ii) availability of sequencing library construction methods, (iii) availability of cell type annotation files or supporting information (e.g. expected cell types, cell markers) for cell type annotation."
86,87,scAPAatlas,Metabolic and Signaling Pathways,Functional Feature Annotation,scapaatlas_annotate,scapaatlas_annotate,Other Structured File,"Find the intersections between the provided RBP binding sites (`benchmark/dataset/scAPAatlas/rbp_binding_sites.bed`) and a set of alternative polyadenylation (APA) regions (`benchmark/dataset/scAPAatlas/apa_regions.bed`). 

    Save the resulting intersected regions to `<r></r>/scapaatlas_annotate.bed`.

    Output example format:
    ```
    chr1  100	150	Peak_4557	841	.	8141.98552	454.85123	5120.84359	14278
    ```

    All intermediate results should be saved to `<t></t>/scapaatlas_annotate`
",We then identified RBP-binding sites potentially altered by cell-type-specific APA events through intersecting the RBP-binding sites with previously defined alternative regions regulated by the cell-type-specific APA events.
87,88,SCoV2-MD,Structure,Functional Feature Annotation,scov2_md_annotate,scov2_md_annotate,Geometric,"Analyze the Molecular Dynamics (MD) simulation trajectory of the NSP1 protein. 

    The trajectory file is located at `benchmark/dataset/SCoV2-MD/474_trj_245.xtc` and the corresponding topology file is `benchmark/dataset/SCoV2-MD/473_dyn_245.pdb`.

    Calculate the Root Mean Square Fluctuation (RMSF) for each residue. 

    Strictly restrict the calculation to Alpha-Carbon (CA) atoms only. 

    Save the result as a JSON file <r></r>/rmsf_ca.json, where the key is the residue index (0-based) and the value is the RMSF value in nanometers.

    Output Format (For Reference)
    {
""0"": 0.12,
""1"": 0.08,
...
    }

    # Intermediate Results
    All intermediate files generated should be saved in the directly `<t></t>/cov2_md_annotat`
","Time-dependent descriptors are extracted from atomistic simulations and include RMSF and a large variety of non-covalent contact types (such as Van der Waals, hydrogen bonding, etc.) among others. They are computed on the basis of the MDtraj (33) and GetContacts libraries (34). The list of descriptors provides the user with a rich repertoire to score and interrogate variant substitutions on-the-fly.  "
88,89,SCovid,Structure,Data Refinement and Integration,scovid_refinement,scovid_refinement,Sequence,"Load the NCBI GEO single-cell RNA-seq dataset GSE182298 located at `benchmark/dataset/SCovid/GSE182298`.'

    Perform Quality Control (QC) filtering with the following strict criteria:

    Filter out cells that have fewer than 200 detected genes.

    Filter out genes that are detected in fewer than 3 cells.

    Calculate the percentage of mitochondrial genes (genes starting with prefix ""MT-"") for each cell.

    Apply the tissue-specific mitochondrial filtering rule: If the input tissue is Liver, remove cells with a mitochondrial gene proportion > 50%. (Note: For non-liver tissues, the threshold would be > 20%).

    Save the list of remaining valid cell barcodes to a CSV file named <r></r>/qc_passed_barcodes.csv (single column, no header).

    All temp data should be save in `<t></t>/scovid_refinement`.

    Output Format (For Reference)
    ```
    <GSMID1>_barcode1
    <GSMID2>_barcode2
    ...
","Considering the technical noise of assay, we removed low quality cells and lowly expressed genes of each COVID-19 related scRNA-seq datasets for further analysis, using the following strategy: (i) cells that had fewer than 200 genes, as well as genes expressed in fewer than three cells; (ii) liver cells that contained greater than $50\%$ of mitochondrial genes, as well as other tissue cells that contained $520\%$ of mitochondrial genes."
89,90,scQTLbase,Human and Other Vertebrate Geomes,Data Refinement and Integration,scqtlbase_refinement,scqtlbase_refinement,Database,"Standardize the heterogeneous cell type names from `benchmark/dataset/scQTLbase/test_data.json`. 

    Map these names to their canonical names based on the cellmarker name reference `benchmark/dataset/scQTLbase/CellMarker.json`. 

    Saving the converted names [converted_name_1, converted_name_2, ...] list into a json file `<r></r>/scqtlbase_refinement.json`.

    Output format example:
    [""T cell"", ""Mast cell"", ""Goblet cell""]

    All intermediate results should be saved in `<t></t>/scqtlbase_refinement` directory.
","To address the challenge of inconsistent cell type names across different studies and datasets, we obtained expertannotated cell type reference datasets from Azimuth (18 . Subsequently, we conducted a manual review of each cell type name, accurately mapping them to the appropriate reference label. This process enabled us to align cell types with multiple names in different studies to their corresponding references."
90,91,scQTLbase,Human and Other Vertebrate Geomes,Open-World Data Retrieval,scqtlbase_retrieval,scqtlbase_retrieval,Text,"# Core Task
    Select the provided literature that meet the following criteria: (1) Studies have reported significant single-cell expression quantitative trait loci (sc-eQTLs) associated with SNP rs10095121. To further validate or conduct in-depth analysis, it is necessary to locate the corresponding raw genotype measurement data from the relevant literature. If the experimental data in a study includes this variant (rs10095121), it is considered acceptable even if the SNP is not explicitly mentioned in the text; (2) studies use actual biological samples (excluding meta-analyses or secondary analyses) and contain at least 40 samples or 5,000 cells; (3) data is generated using reliable single-cell sequencing technologies (e.g., 10x, Smart-seq/Smart-seq2, CITEseq); (4) datasets include genome-wide or local region eQTL mapping.
    Literature to analyze: `dataset/scQTLbase/35672358.pdf`, `dataset/scQTLbase/37558883.pdf`, `dataset/scQTLbase/35915177.pdf`, `dataset/scQTLbase/40562914.pdf`

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets all inclusion and filtering criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/scqtlbase_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/scqtlbase_retrieval/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""99999999"",
""Accepted"": false
},
{
""PMID"": ""88888888"",
""Accepted"": true
}
    ]
","We conducted a manual collection of single-cell eQTL datasets from published iterature, adhering o a predefined set of rules: (i) the selected studies utilized actual samples from a diverse range of biological contexts, including normal, reated or disease conditions. We excluded datasets from meta-analysis or secondary analyses. To ensure sufficient power for sc-eQTLs, we included only studies with a minimum of 40 samples or $5~000$ cells. (ii) We included only single-cell transcriptome data generated using well-established and reliable sequencing technologies such as $10\times$ , Smart-seq/Smart-seq2 and CITEseq. (iii) We considered both eQTL datasets derived from genome-wide primary eQTL mapping and local-region eQTL mapping."
91,92,StemDriver,Human and Other Vertebrate Geomes,Open-World Data Retrieval,stemdriver_retrieval,stemdriver_retrieval,Database,"# Core Task
    Search PubMed for previous HSC-focused studies using single-cell RNA sequencing data, with the following query and date restriction:`query = ('(""Hematopoietic Stem Cells""[Title/Abstract] OR ""HSC""[Title/Abstract])'AND (""single-cell RNA sequencing""[Title/Abstract] OR ""scRNA-seq""[Title/Abstract])'NOT (""Hepatic Stellate Cells""[Title/Abstract])') AND (""2023/02/10""[Date - Publication] : ""2023/03/10""[Date - Publication])`. Select studies that meet the following criteria: (i) datasets are publicly accessible; (ii) data is generated using single-cell RNA sequencing (scRNA-seq) technology; (iii) included cell types are restricted to early-formed HSCs, HSCs, and their derived cell types.

# Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search.
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if the article meets all inclusion and filtering criteria, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/stemdriver_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/stemdriver_retrieval/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""99999999"",
""Accepted"": false
},
{
""PMID"": ""88888888"",
""Accepted"": true
}
    ]
","We conducted a search of previous studies centered on HSCs using scRNA-seq data through the PubMed website, employing keywords such as ‘hematopoietic stem cell’, ‘single cell’, and ‘RNA-seq’. Subsequently, a total of 245 relevant papers were identified. From these datasets, we applied the following criteria or dataset selection: i) The datasets had o be publicly accessible. (ii) Only datasets generated using single-cell RNA sequencing echniques were considered. iii) The study ocused on organisms within the Homo sapiens and Mus musculus species. (iv) Included cell types were limited to early formed HSCs, HSCs, and their resultant cell types. (v) The datasets had to meet the quality control standards outlined in the data preprocessing section (Figure 1 ."
92,93,TF-Marker,Nucleotide Sequence,Functional Feature Annotation,tf_marker_annotate,tf_marker_annotate,Text,"Classify gene entries in `benchmark/dataset/TF-Marker/tf_marker_entries.json` into five categories based on their functional description. 

    Read the paper conent file `benchmark/dataset/TF-Marker/paper_content.md`. 

    Classify the gene based on the paper content and following rules: (i) TF: TFs, which regulate the expression of markers. For example, FOXA1 (HNF3A) is a TF involved in embryonic development which plays an important role in cancer. Studies showed that FOXA1 could regulate the expression of cell marker PLOD2 by binding to its promoters, thereby affecting the occurrence and development of lung cancer; (ii) T Marker: markers, which are regulated by TFs. For example, TF POU5F1 controls the expression of marker genes (e.g. YES1, FGF4, UTF1 and ZFP206) involved in embryonic development. These genes are defined as Tmarker; (iii) I Marker: markers, which influence the activity of TFs. For example, Marker gene CD27 could activate PRDM1 and XBP1 by binding to TF CD70 on B cells. CD27 plays a role as I Marker; (iv) TFMarker: TFs, which play roles as markers. TFMarkers are usually cell/tissue-specific TFs and used as cell markers. For example, TF GATA3 is a definitive cell marker of breast cancer; and (v) TF Pmarker: TFs, which play roles as potential markers. For example, Becker et al. (12) found that TF LGR5 was a potential marker of intestinal stem cells in human .  

    Save the resulting categories list to `<r></r>/tf_marker_annotate.json`.

    Output format (for reference):
    [category_1, category_2, ...] 

    All intermediate results should be save in `<t></t>/tf_marker_annotate` directory.
","Currently, through reviewing 2,091 published literature, we have manually classified TFs and related markers into five types according to their functions: (i) TF: TFs, which regulate the expression of markers. For example, FOXA1 (HNF3A) is a TF involved in embryonic development which plays an important role in cancer. Studies showed that FOXA1 could regulate the expression of cell marker PLOD2 by binding to its promoters, thereby affecting the occurrence and development of lung cancer (Figure 1 A 1); (ii) T Marker: markers, which are regulated by TFs. For example, TF POU5F1 controls the expression of marker genes (e.g. YES1, FGF4, UTF1 and ZFP206) involved in embryonic development. These genes are defined as Tmarker (Figure 1 A 2); (iii) I Marker: markers, which influence the activity of TFs. For example, Marker gene CD27 could activate PRDM1 and XBP1 by binding to TF CD70 on B cells. CD27 plays a role as I Marker (Figure 1A3); (iv) TFMarker: TFs, which play roles as markers. TFMarkers are usually cell/tissue-specific TFs and used as cell markers. For example, TF GATA3 is a definitive cell marker of breast cancer (Figure 1A4); and (v) TF Pmarker: TFs, which play roles as potential markers."
93,94,TF-Marker,Nucleotide Sequence,Open-World Data Retrieval,tf_marker_retrieval,tf_marker_retrieval,Database,"# Core Task
    Search PubMed for Meta-Analysis articles with abstracts published between 2025/01/01 and 2025/01/05 using the search query (transcription factor[Title/Abstract] OR transcription factors[Title/Abstract]), that are related to transcription factors, and then analyze each retrieved abstract to determine if it discusses a functional or regulatory relationship between Transcription Factors (TFs) and any type of molecular marker (e.g., biomarker, genetic marker, protein marker, epigenetic marker).

    # Output Requirements & File Path
    - Output format: JSON list of objects. The list must include all articles obtained from the initial PubMed search (the result of the first step).
    - Each object must contain exactly two fields:
- `""PMID""`: PubMed ID as a string
- `""Accepted""`: Boolean value (`true` if relationship is mentioned, `false` otherwise)
    - Save Path: Save the complete list to: `<r></r>/tf_marker_retrieval.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/tf_marker_retrieval/`

    # Output Example (For Reference Only)
    [
{
""PMID"": ""40459423"",
""Accepted"": false
},
{
""PMID"": ""35456656"",
""Accepted"": true
}
    ]
","The literature with ‘transcription factor(s)’ was initially retrieved from the PubMed database. We found $>70\ 00\dot{0}$ literature with TFs and related markers mentioned in the titles or abstracts. Then, we carefully read these abstracts and retained $>10\ 000$ literature that included the relationships between TFs and markers. We applied the following workflow to obtain specific information about the TFs and related markers. First, we screened the literature based on two standards: (i) experimental evidence that could confirm the relationships between TFs and markers was mentioned in each article (e.g. qRT-PCR, western blot); and (ii) names of the cells or tissues were mentioned in the corresponding experiments."
94,95,TheMarker,Human Genes and Diseases,Functional Feature Annotation,themarker_annotate,themarker_annotate,Text,"Given txt file containing biomarkers and their functions in the file `benchmark/dataset/TheMarker/biomarker.txt`. 

    According to the roles of these biomarkers played, labelled the biomarkers as pharmacodynamic biomarker (PDY), safety biomarker (SAF), monitoring biomarker (MOI), predictive biomarker (PRD) and surrogate endpoint (SUR). 

    Saving the classification list [label_1, label_2, ...] in the same sort as input file to json file `<r></r>/themarker_annotate.json`. 

    The output format example is as follows:
    [""PDY"", ""SAF"", ""MOI"", ""PRD"", ""SUR""]

    All intermediate results should be saved in `<t></t>/themarker_annotate` directory.
"," Moreover, one of the most widely applied strategies for categorizing biomarkers, also known as the ‘BEST’ category officially provided by U.S. Food & Drug Administration (3 , was adopted by the study for classifying all collected ThMARs nto five ypes: pharmacodynamic biomarker (PDY), safety biomarker (SAF), monitoring biomarker (MOI), predictive biomarker (PRD) and surrogate endpoint (SUR). The definition and importance of these ThMAR types were explicitly described in Table 1  and a detailed discussion on five ThMAR types was provided below.  "
95,96,TheMarker,Human Genes and Diseases,Structured Information Extraction,themarker_extract_1,themarker_extract_1,Text,"# Core Task
    Analyze the provided literature to classify the therapeutic biomarker (ThMAR) CRP (also known as PTX1) into one of the following five types:
    1. Pharmacodynamic biomarker (PDY)
    2. Safety biomarker (SAF)
    3. Monitoring biomarker (MOI)
    4. Predictive biomarker (PRD)
    5. Surrogate endpoint (SUR)

    The definition, importance, and typical examples of each ThMAR type are explicitly described in the image `dataset\TheMarker\ThMARs_TYPE.png`.
    Literature to analyze: `dataset\TheMarker\34903842.pdf`, `dataset\TheMarker\25135077.pdf`, `dataset\TheMarker\30289572.pdf`, `dataset\TheMarker\19435794.pdf`,
    `dataset\TheMarker\FDA.pdf`

    # Output Requirements & File Path
    - Output format: A JSON list of dictionaries with ""PMID"" and ""TypeID"".
    - ""PMID"" is the filename(For FDA.pdf, ""PMID"" is ""FDA""), and ""TypeID"" is the classification category mentioned above (int).
    - Save Path: Save the complete list to: `<r></r>/themarker_extract_1.json`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/themarker_extract_1/`

    # Example Output (For Reference Only)
    [
{
""PMID"": ""22222222"",
""TypeID"": 2
},
{
""PMID"": ""33333333"",
""TypeID"": 4
}
    ]
","Retrieved literatures were then carefully reviewed, and those reported ThMARs together with their corresponding therapies were recorded."
96,97,TheMarker,Human Genes and Diseases,Structured Information Extraction,themarker_extract_2,themarker_extract_2,Other Structured File,"# Core Task
    Retrieve the drug label for NDC 42681-0025-1 from DailyMed, parse the text to extract the active ingredient, potential toxicity mechanisms, and associated phenotypes, and return the results as a CSV file

    # Output Requirements & File Path
    - Output format: A CSV file with the following columns (exactly matching the ground truth CSV headers):
- `active_ingredient`: The extracted active ingredient as a string
- `toxicity_mechanism`: The extracted toxicity mechanism as a string
- `phenotype`: The associated phenotype as a string
    - Save Path: Save the result to: `<r></r>/themarker_extract_2.csv`
    - Intermediate Storage Path: All intermediate code generated during execution, downloaded files, and temporary results must be stored in `<t></t>/themarker_extract_2/`

    # Output Example (For Reference Only)
    active_ingredient,toxicity_mechanism,phenotype
    PseudoDrugA,Hypothetical metabolic inhibition,Hypothetical skin rash
","FDA-approved drug labels were extracted, which were scattered throughout the different sections of these labels such as indications and usage, adverse reactions, and use in specific populations  According to the roles of these biomarkers played, the types of these biomarkers were manually labelled."
97,98,VarEPS,Structure,Functional Feature Annotation,vareps_annotation,vareps_annotation,Geometric,"There are a list of possible single amino acid mutations for protein in `benchmark/dataset/VarEPS/4KMN.pdb`. The mutations are as listed in file `benchmark/dataset/VarEPS/mutation_list.txt`.

    Identify the most stable mutation among the list of possible mutations by saambe-3d. 

    Save the most stable mutation and its corresponding 螖螖G value to `<r></r>/most_stable_mutation.txt`.

    All temp data should be save in `<t></t>/vareps_annotation`.

    Output format (For Reference):
    A 415 Q T
","Changes in free energy of binding with neutralizing antibodies caused by single amino acid mutation: Saambe-3D (12) was used to predict changes in free energy of binding caused by single amino acid mutation and disruption of protein– protein interaction (PPI). Mutation types included destabilizing mutation ( $\Delta\Delta G>0$ ), stable mutation $(-1.5~<$ $\Delta\Delta G<0$ ), highly destabilizing mutation ( $\Delta\Delta G>1.5)$ and highly stable mutation ( $\Delta\Delta G<-1.5)$ ."
98,99,ViMIC,Structure,Structured Information Extraction,vimic_extract,vimic_extract,Text,"Extracting the mutation sites of Hepatitis B virus (HBV) from the provided pdf file `benchmark/dataset/ViMIC/12985_2021_Article_1493.pdf`.

    Saving the results into a json file `<r></r>/vimic_extract.json`.

    Result format reference:
    [
""G130E"",
""L109R""
    ]

    All intermediate results should be saved in `<t></t>/vimic_extract` directory.
","Full-text articles from PMC and abstracts from PubMed (BioC-XML format) with annotations of biomedical concepts in the Bioconepts2pubtatorcentral file were downloaded from PubTatorCentral (31), and this file was then used to extract biomedical entities such as genes, mutations, diseases and chemicals. A Python script was developed to automate the entity recognition task for those articles without PubTator annotation. The procedure included literature downloading, conversion of PDF documents to plain text files (PDFMiner library), and noise filtering and information extraction for sentences containing mutations. Mutation entries were automatically extracted from the PubMed and PubTator annotation files (32), followed by a manual curation to ensure the data quality of our database.  "
99,100,ZOVER,Structure,Structured Information Extraction,zover_extract,zover_extract,Text,"Extracting information for Biosamples (filter out those without SRA data) in BioProject PRJEB63303 and PRJNA1025946, including host, sample name, collection date and country.

    Saving the results into a csv file `<r></r>/zover_extract.csv`.

    Output format (for reference only):
    SampleName,Host,CollectionDate,Country,BioProject,BioSample
    Bat_throat,Eidolon helvum,2009/03,Ghana,ERP001979,SAMEA1573860

    All intermediate results should be saved in `<t></t>/zover_extract` directory.
","All remaining records were then downloaded to the local system and converted by in-house BioPerl (17) scripts into relational tables of meta-information, including virus name, sampling time and location, associated vector, detection methods, cell culture method (if available), accession, sequence submitter(s) and affiliation, and related publication (if exists). To ensure the accuracy of the meta-data, we further performed an intensive double-check for published records by proofreading any inconsistencies and incompleteness based on the information described in related literature."