Event Recognition in three domains Acquisition,Vendor-Supplier and Job events. For this purpose data is crawled from news webistes and tagged using semi-supervised techniques using navie bayes through EM and active learning approaches. Finally the classification was performed by majority voting of three classifiers RandomForest,GradientBoosting Tree and ada boost.
Libraries in python sckit-learn,pandas etc.
Clinical LLM Validator Prompt Functions – Documentation This documentation describes the structure, inputs, and outputs for a suite of LLM validator prompt functions for clinical entity extraction, classification, importance, and severity ranking. Each function is designed for rigorous audit of automated ICD-10-CM coding and subsequent category assignment pipelines. --- ## Input Parameters | Parameter | Type | Description | |--------------------------|-----------|-----------------------------------------------------------------------------------------------------| | cohort_name | str | The disease cohort category (e.g., Asthma, IBD, COVID-19). | | clinical_note | str | The full clinical note or EHR extract to be analyzed. | | rag_store | dict/list | Top ICD-10 code-description pairs retrieved from a RAG (OpenSearch) store, for use as retrieval context.| | diagnosis_codes | list | List of ICD-10 codes, extracted or to be classified. | | diagnosis_rationales | list | List of rationales for each code, as generated or manually annotated. | | llmx_json_answer | dict | The generator model's (LLMx) JSON response, with codes and rationales. | | LLMx_prompt | str | The original generation prompt used by LLMx to produce its output. | | llm1_validator_codes | list | Final "validated" code list produced by the LLM1 validator—used for stepwise category audits. | --- ## Function Overviews and Output ### 1. Diagnosis Extraction Validator Function:
def format_validator_prompt_q1_diagnosis_extraction( cohort_name, clinical_note, rag_store, llm1_json_answer, LLM1_prompt )
text
Purpose: Validate that all codes proposed by LLM1 are strictly supported by the clinical note and RAG store. Includes support for correction and audit-trail output. Output JSON: - "question1": The main question for this review step. - "agreement_with_llm1": bool – Is LLM1’s output fully correct? - "corrected_answer1": list or null – Null if all correct; otherwise the corrected code list. - "corrected_rationale1": list – Rationales for codes in corrected_answer1 (if corrections made). - "copy_from_LLM1": dict – The original diagnosis codes and rationales from LLM1 for traceability. - "relevance", "faithfulness", "confidence": int (1-5) – Audit scoring. Examples: Agreement:
{ "question1": "...", "agreement_with_llm1": true, "corrected_answer1": null, "copy_from_LLM1": { "answer1": ["J45.0", "U07.1"], "rationale1": ["Asthma mentioned as active issue.", "COVID-19 discussed in assessment."] }, "relevance": 5, "faithfulness": 5, "confidence": 5 }
text Disagreement:
{ "question1": "...", "agreement_with_llm1": false, "corrected_answer1": ["J45.0", "U07.1"], "corrected_rationale1": [ "J45.0: Directly supported by clinical note & RAG.", "U07.1: Explicitly mentioned and confirmed by RAG retrieval." ], "copy_from_LLM1": { "answer1": ["J45.0"], "rationale1": ["Asthma only mentioned."] }, "relevance": 5, "faithfulness": 5, "confidence": 5 }
text --- ### 2. Categorization Validator Function:
def format_validator_prompt_q2_classify_codes( cohort_name, clinical_note, diagnosis_codes, diagnosis_rationales, llm2_json_answer, LLM2_prompt1, llm1_validator_codes )
text
Purpose: Validate that each code has been assigned the correct clinical category ("Reason for Visit," "Other Active," etc.), including new codes added by the validator chain. Output JSON: - "question2": Review question. - "agreement_with_llm2": bool. - "corrected_classification2": list of {code, category, rationale} or null. - "copy_from_LLM2": dict (codes, answer2—categories, rationale2—justification). - "relevance", "faithfulness", "confidence": int (1-5). --- ### 3. Importance Level Validator Function:
def format_validator_prompt_q3_importance_level( cohort_name, clinical_note, diagnosis_codes, diagnosis_rationales, llm3_json_answer, LLM3_prompt, llm1_validator_codes )
text
Purpose: Validate that each ICD code’s patient-encounter importance is accurate (high, medium, low), including new/extra codes found by chained validation. Output JSON: - "question3": Review question. - "agreement_with_llm3": bool. - "corrected_importance3": list or null. - "copy_from_LLM3": (codes, answer3, rationale3 as annotated by LLM3). - "relevance", "faithfulness", "confidence": int (1-5). --- ### 4. Severity Level Validator Function:
def format_validator_prompt_q4_severity_level( cohort_name, clinical_note, diagnosis_codes, diagnosis_rationales, llm4_json_answer, LLM4_prompt, llm1_validator_codes )
text
Purpose: Validate each code’s severity-of-impact on the patient (high, medium, low, N/A), auditing LLM4’s output for coverage and correctness. Output JSON: - "question4": Review question. - "agreement_with_llm4": bool. - "corrected_severity4": list or null. - "copy_from_LLM4": (codes, answer4—severity levels, rationale4). - "relevance", "faithfulness", "confidence": int (1-5). --- ## General JSON Output Structure - All outputs must be a single strict JSON object—see examples above. - All codes are included (original and new if added by prior validator). - Copy of the generator’s original output ("copy_from_LLMX") is always included for post-hoc review/audit. - When corrections are necessary, rationales must be explicit and reference note text or retrieval context. - All scores must be integers between 1 and 5. - All fields outlined above are mandatory. --- ## Workflow Notes - The validator LLM cross-checks codes, categories, importance, and severity against the source clinical note, RAG results, and generator instructions. - Agreement (“agreement_with_llmx” = true) is permitted only if every code and rationale is supported; otherwise explicit corrections are made. - Chain-of-validation: Codes from the diagnosis extraction validator (Q1) become the ground truth set for subsequent steps (Q2–Q4). --- ## References - Code validity referencing coding guidelines: [web:128] [web:130] [web:81]. ---