From 7e6a24896980b144dfacc5628592914fc9a11bb9 Mon Sep 17 00:00:00 2001 From: Claude Code Date: Sun, 5 Apr 2026 08:35:59 +0000 Subject: [PATCH 1/2] fix: separate legal_status from judgment and ensure correct abstract source in screening The screening skill was storing search snippets as abstract_text instead of the official abstract from fetch_patent. This was caused by: 1. No explicit instruction to use fetch_patent.abstract_text (not snippet) 2. No legal_status column in screened_patents table, causing judgment to conflate relevance assessment with legal status 3. Tests only verified fetch_patent was invoked, not that results were used Changes: - Add legal_status column to screened_patents table - Change judgment CHECK constraint to only allow relevant/irrelevant - Update screening SKILL.md to explicitly distinguish abstract_text from snippet and batch fetch patents (up to 10 in parallel) - Add CRITICAL warnings against using snippet as abstract - Update record-screening.md with legal_status parameter - Add test checks: all_patents_screened, legal_status_recorded, patent_fetch_invoked for both screening and evaluating - Update all test fixtures to include legal_status in INSERT statements Co-Authored-By: Claude Opus 4.6 --- .../references/schema.md | 40 +++++++------- .../references/sql/initialize-database.sql | 5 +- .../skills/investigation-recording/SKILL.md | 2 +- .../instructions/record-screening.md | 55 +++++++++++-------- plugin/skills/screening/SKILL.md | 32 ++++++----- .../functional-absent-feature.toml | 2 +- tests/claim-analyzing/functional.toml | 2 +- tests/evaluating/functional.toml | 8 ++- .../functional-overall-progress.toml | 4 +- .../functional-pending-phases.toml | 2 +- ...tional-specific-patent-with-prior-art.toml | 2 +- .../functional-specific-patent.toml | 2 +- tests/prior-art-researching/functional.toml | 2 +- tests/screening/functional-with-data.toml | 30 ++++++++++ 14 files changed, 120 insertions(+), 68 deletions(-) diff --git a/plugin/skills/investigation-preparing/references/schema.md b/plugin/skills/investigation-preparing/references/schema.md index b87af0c..6506221 100644 --- a/plugin/skills/investigation-preparing/references/schema.md +++ b/plugin/skills/investigation-preparing/references/schema.md @@ -30,19 +30,21 @@ Stores patent master data imported from CSV files. Stores latest screening results only (no history tracking). -| Column | Type | Description | -| ------------- | ------------- | ------------------------------------------------ | -| patent_id | TEXT PK | Patent number (FK to target_patents.patent_id) | -| judgment | TEXT NOT NULL | Judgment: `relevant`, `irrelevant`, or `expired` | -| reason | TEXT NOT NULL | Screening rationale | -| abstract_text | TEXT NOT NULL | Abstract content (fetched during screening) | -| screened_at | TEXT | Screening timestamp | -| updated_at | TEXT | Last update timestamp | +| Column | Type | Description | +| ------------- | ------------- | -------------------------------------------------------------------------- | +| patent_id | TEXT PK | Patent number (FK to target_patents.patent_id) | +| judgment | TEXT NOT NULL | Relevance: `relevant` or `irrelevant` | +| legal_status | TEXT | Legal status from `fetch_patent` (e.g., `Pending`, `Expired`, `Withdrawn`) | +| reason | TEXT NOT NULL | Screening rationale | +| abstract_text | TEXT NOT NULL | Abstract from `fetch_patent.abstract_text` | +| screened_at | TEXT | Screening timestamp | +| updated_at | TEXT | Last update timestamp | **Constraints**: - `patent_id` is a FOREIGN KEY referencing `target_patents(patent_id)` with `ON DELETE CASCADE` -- `judgment` only allows: `relevant`, `irrelevant`, `expired` +- `judgment` only allows: `relevant`, `irrelevant` +- `legal_status` reflects the patent's legal status from `fetch_patent` - `reason` and `abstract_text` must NOT be NULL ### claims @@ -176,13 +178,13 @@ Stores element-level mappings between patent elements and prior art references. Aggregates screening statistics. -| Column | Type | Description | -| -------------- | ------- | ------------------------------------------- | -| total_targets | INTEGER | Count of all patents in target_patents | -| total_screened | INTEGER | Count of all patents in screened_patents | -| relevant | INTEGER | Count of patents with judgment='relevant' | -| irrelevant | INTEGER | Count of patents with judgment='irrelevant' | -| expired | INTEGER | Count of patents with judgment='expired' | +| Column | Type | Description | +| -------------- | ------- | ----------------------------------------------------------- | +| total_targets | INTEGER | Count of all patents in target_patents | +| total_screened | INTEGER | Count of all patents in screened_patents | +| relevant | INTEGER | Count of patents with judgment='relevant' | +| irrelevant | INTEGER | Count of patents with judgment='irrelevant' | +| expired | INTEGER | Count of patents with legal_status='Expired' or 'Withdrawn' | ## Triggers @@ -225,9 +227,9 @@ target_patents (1) -----> (1) screened_patents (1) -----> (*) claims (1) -----> | | | | | |-- patent_id (PK) |-- patent_id (PK, FK) |-- patent_id (FK) |-- patent_id (PK, FK) |-- patent_id (PK, FK) |-- title |-- judgment |-- claim_number (FK) |-- claim_number (PK, FK) |-- claim_number (PK, FK) - |-- country |-- reason |-- claim_type |-- element_label (PK) |-- element_label (PK, FK) - |-- assignee |-- abstract_text |-- claim_text |-- element_description |-- similarity_level - |-- extra_fields |-- screened_at |-- created_at |-- created_at |-- analysis_notes + |-- country |-- legal_status |-- claim_type |-- element_label (PK) |-- element_label (PK, FK) + |-- assignee |-- reason |-- claim_text |-- element_description |-- similarity_level + |-- extra_fields |-- abstract_text |-- created_at |-- created_at |-- analysis_notes |-- publication_date |-- updated_at |-- updated_at |-- updated_at |-- analyzed_at |-- filing_date | | | |-- updated_at |-- grant_date | | | diff --git a/plugin/skills/investigation-preparing/references/sql/initialize-database.sql b/plugin/skills/investigation-preparing/references/sql/initialize-database.sql index 8d3f558..2b5d58f 100644 --- a/plugin/skills/investigation-preparing/references/sql/initialize-database.sql +++ b/plugin/skills/investigation-preparing/references/sql/initialize-database.sql @@ -37,7 +37,8 @@ CREATE TABLE IF NOT EXISTS target_patents ( -- Create screened_patents table (with CHECK constraint for judgment) CREATE TABLE IF NOT EXISTS screened_patents ( patent_id TEXT PRIMARY KEY NOT NULL, - judgment TEXT NOT NULL CHECK(judgment IN ('relevant', 'irrelevant', 'expired')), + judgment TEXT NOT NULL CHECK(judgment IN ('relevant', 'irrelevant')), + legal_status TEXT, reason TEXT NOT NULL, abstract_text TEXT NOT NULL, screened_at TEXT DEFAULT (datetime('now')), @@ -52,7 +53,7 @@ SELECT (SELECT COUNT(*) FROM screened_patents) as total_screened, (SELECT COUNT(*) FROM screened_patents WHERE judgment = 'relevant') as relevant, (SELECT COUNT(*) FROM screened_patents WHERE judgment = 'irrelevant') as irrelevant, - (SELECT COUNT(*) FROM screened_patents WHERE judgment = 'expired') as expired; + (SELECT COUNT(*) FROM screened_patents WHERE legal_status IN ('Expired', 'Withdrawn')) as expired; -- Create timestamp triggers CREATE TRIGGER IF NOT EXISTS update_target_patents_timestamp diff --git a/plugin/skills/investigation-recording/SKILL.md b/plugin/skills/investigation-recording/SKILL.md index bec9535..9554139 100644 --- a/plugin/skills/investigation-recording/SKILL.md +++ b/plugin/skills/investigation-recording/SKILL.md @@ -41,7 +41,7 @@ internal reference files for this skill's internal use only. **Example requests**: -- "Record screening result: id=US1234567A1, judgment=relevant, reason=..." +- "Record screening result: id=US1234567A1, judgment=relevant, legal_status=Pending, reason=..." - "Record claims for patent US1234567A1: claim_1=..., claim_2=..." - "Record elements for patent US1234567A1: element_a=..., element_b=..." - "Record similarities for patent US1234567A1: element_a=Significant, element_b=Moderate..." diff --git a/plugin/skills/investigation-recording/references/instructions/record-screening.md b/plugin/skills/investigation-recording/references/instructions/record-screening.md index a495e8d..b00e160 100644 --- a/plugin/skills/investigation-recording/references/instructions/record-screening.md +++ b/plugin/skills/investigation-recording/references/instructions/record-screening.md @@ -9,10 +9,11 @@ Save or update a screening judgment in the `screened_patents` table. ### Main Query (UPSERT) ```sql -INSERT OR REPLACE INTO screened_patents (patent_id, judgment, reason, abstract_text, updated_at) +INSERT OR REPLACE INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text, updated_at) VALUES ( '', '', + '', '', '', datetime('now') @@ -23,6 +24,9 @@ VALUES ( - `INSERT OR REPLACE` provides UPSERT semantics - `patent_id` is a FOREIGN KEY referencing `target_patents(patent_id)` +- `judgment` must be `relevant` or `irrelevant` +- `legal_status` is the value from `fetch_patent` (e.g., `Pending`, `Expired`, `Withdrawn`) +- `abstract_text` must be from `fetch_patent.abstract_text` (NOT from `search_patents.snippet`) - `reason` and `abstract_text` are required (NOT NULL) - `updated_at` automatically set to current timestamp @@ -32,8 +36,8 @@ VALUES ( ```bash # Record screening result -sqlite3 patents.db "INSERT OR REPLACE INTO screened_patents (patent_id, judgment, reason, abstract_text, updated_at) -VALUES ('US1234567A', 'relevant', 'Core technology for LLM systems', 'Abstract content fetched during screening', datetime('now'));" +sqlite3 patents.db "INSERT OR REPLACE INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text, updated_at) +VALUES ('US1234567A', 'relevant', 'Pending', 'Core technology for LLM systems', 'Abstract content fetched during screening', datetime('now'));" ``` ### Using Variables @@ -41,21 +45,23 @@ VALUES ('US1234567A', 'relevant', 'Core technology for LLM systems', 'Abstract c ```bash PATENT_ID="US1234567A" JUDGMENT="relevant" +LEGAL_STATUS="Pending" REASON="Core technology for LLM systems" ABSTRACT_TEXT="Abstract content here" -sqlite3 patents.db "INSERT OR REPLACE INTO screened_patents (patent_id, judgment, reason, abstract_text, updated_at) -VALUES ('$PATENT_ID', '$JUDGMENT', '$REASON', '$ABSTRACT_TEXT', datetime('now'));" +sqlite3 patents.db "INSERT OR REPLACE INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text, updated_at) +VALUES ('$PATENT_ID', '$JUDGMENT', '$LEGAL_STATUS', '$REASON', '$ABSTRACT_TEXT', datetime('now'));" ``` ### Multi-Line SQL ```bash sqlite3 patents.db <: " + For each patent, invoke `Skill: investigation-recording` with request "Record screening result for patent : judgment=, legal_status=, reason=, abstract_text=" + - **CRITICAL**: The `abstract_text` passed to recording MUST be the `abstract_text` from `fetch_patent`, NOT the `snippet` from `search_patents`. -3. **Verify Results**: Confirm all patents have corresponding `screened_patents` entries +5. **Verify Results**: Confirm all patents have corresponding `screened_patents` entries ## State Management diff --git a/tests/claim-analyzing/functional-absent-feature.toml b/tests/claim-analyzing/functional-absent-feature.toml index 64d30fe..d0e07b7 100644 --- a/tests/claim-analyzing/functional-absent-feature.toml +++ b/tests/claim-analyzing/functional-absent-feature.toml @@ -14,7 +14,7 @@ Before asking me any questions about missing features, please use the question-r command = """ sqlite3 patents.db < /workspaces/patent-kit/plugin/skills/investigation-preparing/references/sql/initialize-database.sql sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US12231380B1', 'Trigger-based transfer of conversations from a chatbot to a human agent', 'US', '2023-10-11');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Pending', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" sqlite3 patents.db "INSERT INTO claims (patent_id, claim_number, claim_type, claim_text) VALUES ('US12231380B1', 1, 'independent', '1. A computer-implemented method for managing conversations in a chatbot system, comprising: detecting a trigger condition in a conversation context; and transferring the conversation from the chatbot to a human agent based on the trigger condition.');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'A', 'Detecting a trigger condition in a conversation context');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'B', 'Transferring the conversation from the chatbot to a human agent based on the trigger condition');" diff --git a/tests/claim-analyzing/functional.toml b/tests/claim-analyzing/functional.toml index 03b3776..e86d28c 100644 --- a/tests/claim-analyzing/functional.toml +++ b/tests/claim-analyzing/functional.toml @@ -14,7 +14,7 @@ Before asking me any questions about missing features, please use the question-r command = """ sqlite3 patents.db < /workspaces/patent-kit/plugin/skills/investigation-preparing/references/sql/initialize-database.sql sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US12231380B1', 'Trigger-based transfer of conversations from a chatbot to a human agent', 'US', '2023-10-11');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Pending', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" sqlite3 patents.db "INSERT INTO claims (patent_id, claim_number, claim_type, claim_text) VALUES ('US12231380B1', 1, 'independent', '1. A computer-implemented method for managing conversations in a chatbot system, comprising: detecting a trigger condition in a conversation context; and transferring the conversation from the chatbot to a human agent based on the trigger condition.');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'A', 'Detecting a trigger condition in a conversation context');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'B', 'Transferring the conversation from the chatbot to a human agent based on the trigger condition');" diff --git a/tests/evaluating/functional.toml b/tests/evaluating/functional.toml index ae9fa50..b8c65d3 100644 --- a/tests/evaluating/functional.toml +++ b/tests/evaluating/functional.toml @@ -13,8 +13,8 @@ command = """ sqlite3 patents.db < /workspaces/patent-kit/plugin/skills/investigation-preparing/references/sql/initialize-database.sql sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US12231380B1', 'Trigger-based transfer of conversations from a chatbot to a human agent', 'US', '2023-10-11');" sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US20230245651A1', 'Enabling user-centered and contextually relevant interaction', 'US', '2023-04-03');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US20230245651A1', 'relevant', 'Related to user interaction and context management', 'A method for enabling user-centered and contextually relevant interaction in conversational systems.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Pending', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US20230245651A1', 'relevant', 'Pending', 'Related to user interaction and context management', 'A method for enabling user-centered and contextually relevant interaction in conversational systems.');" """ [[setup]] @@ -60,3 +60,7 @@ command = { command = "skill-loaded", skill = "investigation-fetching" } [[checks]] name = "recording_loaded" command = { command = "skill-loaded", skill = "investigation-recording" } + +[[checks]] +name = "patent_fetch_invoked" +command = { command = "mcp-tool-invoked", tool = "fetch_patent" } diff --git a/tests/investigation-reporting/functional-overall-progress.toml b/tests/investigation-reporting/functional-overall-progress.toml index 29ba241..f59802f 100644 --- a/tests/investigation-reporting/functional-overall-progress.toml +++ b/tests/investigation-reporting/functional-overall-progress.toml @@ -13,8 +13,8 @@ command = """ sqlite3 patents.db < /workspaces/patent-kit/plugin/skills/investigation-preparing/references/sql/initialize-database.sql sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US12231380B1', 'Trigger-based transfer of conversations from a chatbot to a human agent', 'US', '2023-10-11');" sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US20230245651A1', 'Enabling user-centered and contextually relevant interaction', 'US', '2023-04-03');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US20230245651A1', 'irrelevant', 'Not related to core product features', 'A method for enabling user-centered and contextually relevant interaction in conversational systems.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Pending', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US20230245651A1', 'irrelevant', 'Pending', 'Not related to core product features', 'A method for enabling user-centered and contextually relevant interaction in conversational systems.');" """ [[setup]] diff --git a/tests/investigation-reporting/functional-pending-phases.toml b/tests/investigation-reporting/functional-pending-phases.toml index 4db327a..7fe0b33 100644 --- a/tests/investigation-reporting/functional-pending-phases.toml +++ b/tests/investigation-reporting/functional-pending-phases.toml @@ -12,7 +12,7 @@ I have a patent database with screening results ready for patent US12231380B1, b command = """ sqlite3 patents.db < /workspaces/patent-kit/plugin/skills/investigation-preparing/references/sql/initialize-database.sql sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US12231380B1', 'Trigger-based transfer of conversations from a chatbot to a human agent', 'US', '2023-10-11');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Pending', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" """ [[setup]] diff --git a/tests/investigation-reporting/functional-specific-patent-with-prior-art.toml b/tests/investigation-reporting/functional-specific-patent-with-prior-art.toml index 626878f..a844b60 100644 --- a/tests/investigation-reporting/functional-specific-patent-with-prior-art.toml +++ b/tests/investigation-reporting/functional-specific-patent-with-prior-art.toml @@ -12,7 +12,7 @@ I have investigation data for patent US12231380B1 including prior art research. command = """ sqlite3 patents.db < /workspaces/patent-kit/plugin/skills/investigation-preparing/references/sql/initialize-database.sql sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US12231380B1', 'Trigger-based transfer of conversations from a chatbot to a human agent', 'US', '2023-10-11');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Pending', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" sqlite3 patents.db "INSERT INTO claims (patent_id, claim_number, claim_type, claim_text) VALUES ('US12231380B1', 1, 'independent', '1. A computer-implemented method for managing conversations in a chatbot system, comprising: detecting a trigger condition in a conversation context; and transferring the conversation from the chatbot to a human agent based on the trigger condition.');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'A', 'Detecting a trigger condition in a conversation context');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'B', 'Transferring the conversation from the chatbot to a human agent based on the trigger condition');" diff --git a/tests/investigation-reporting/functional-specific-patent.toml b/tests/investigation-reporting/functional-specific-patent.toml index 6b48bf3..3cb946f 100644 --- a/tests/investigation-reporting/functional-specific-patent.toml +++ b/tests/investigation-reporting/functional-specific-patent.toml @@ -12,7 +12,7 @@ I have investigation data for patent US12231380B1. Please generate a specific pa command = """ sqlite3 patents.db < /workspaces/patent-kit/plugin/skills/investigation-preparing/references/sql/initialize-database.sql sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US12231380B1', 'Trigger-based transfer of conversations from a chatbot to a human agent', 'US', '2023-10-11');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Pending', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" sqlite3 patents.db "INSERT INTO claims (patent_id, claim_number, claim_type, claim_text) VALUES ('US12231380B1', 1, 'independent', '1. A computer-implemented method for managing conversations in a chatbot system, comprising: detecting a trigger condition in a conversation context; and transferring the conversation from the chatbot to a human agent based on the trigger condition.');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'A', 'Detecting a trigger condition in a conversation context');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'B', 'Transferring the conversation from the chatbot to a human agent based on the trigger condition');" diff --git a/tests/prior-art-researching/functional.toml b/tests/prior-art-researching/functional.toml index d1950a8..7c751a9 100644 --- a/tests/prior-art-researching/functional.toml +++ b/tests/prior-art-researching/functional.toml @@ -12,7 +12,7 @@ I have a patent database with Moderate and Significant similarity levels identif command = """ sqlite3 patents.db < /workspaces/patent-kit/plugin/skills/investigation-preparing/references/sql/initialize-database.sql sqlite3 patents.db "INSERT INTO target_patents (patent_id, title, country, publication_date) VALUES ('US12231380B1', 'Trigger-based transfer of conversations from a chatbot to a human agent', 'US', '2023-10-11');" -sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" +sqlite3 patents.db "INSERT INTO screened_patents (patent_id, judgment, legal_status, reason, abstract_text) VALUES ('US12231380B1', 'relevant', 'Pending', 'Related to chatbot-to-human transfer mechanism', 'A system for triggering transfer of conversations from a chatbot to a human agent based on conversation context.');" sqlite3 patents.db "INSERT INTO claims (patent_id, claim_number, claim_type, claim_text) VALUES ('US12231380B1', 1, 'independent', 'A system comprising a chatbot engine, a trigger detection module, and a human agent transfer interface.');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'A', 'Chatbot engine for multi-turn dialogue management');" sqlite3 patents.db "INSERT INTO elements (patent_id, claim_number, element_label, element_description) VALUES ('US12231380B1', 1, 'B', 'Trigger detection module for conversation context analysis');" diff --git a/tests/screening/functional-with-data.toml b/tests/screening/functional-with-data.toml index 6e8bff8..3dc0e75 100644 --- a/tests/screening/functional-with-data.toml +++ b/tests/screening/functional-with-data.toml @@ -56,6 +56,18 @@ command = { command = "skill-invoked", skill = "screening" } name = "database_exists" command = { command = "workspace-file", path = "patents.db" } +[[checks]] +name = "fetching_loaded" +command = { command = "skill-loaded", skill = "investigation-fetching" } + +[[checks]] +name = "recording_loaded" +command = { command = "skill-loaded", skill = "investigation-recording" } + +[[checks]] +name = "patent_fetch_invoked" +command = { command = "mcp-tool-invoked", tool = "fetch_patent" } + [[checks]] name = "target_patents_populated" command = { @@ -64,3 +76,21 @@ command = { query = "SELECT COUNT(*) FROM target_patents;", expected = "3", } + +[[checks]] +name = "all_patents_screened" +command = { + command = "db-query", + db = "", + query = "SELECT COUNT(*) FROM screened_patents;", + expected = "3", +} + +[[checks]] +name = "legal_status_recorded" +command = { + command = "db-query", + db = "", + query = "SELECT COUNT(*) FROM screened_patents WHERE legal_status IS NOT NULL;", + expected = "3", +} From c00a7cefb34a16b4281732b039d7f669d993a2a7 Mon Sep 17 00:00:00 2001 From: Claude Code Date: Sun, 5 Apr 2026 13:49:14 +0000 Subject: [PATCH 2/2] fix: use correct Cypher query pattern for claims retrieval in evaluating - Change from relationship pattern MATCH (p:Patent)-[:claims]->(c:claims) to direct node match MATCH (c:claims) to avoid c.text returning null - Remove ORDER BY toInteger(c.number) which also causes c.text null bug - Add batch parallel fetch (up to 10 patents) for performance - Add CRITICAL warnings documenting the Cypher parser bugs Co-Authored-By: Claude Opus 4.6 --- plugin/skills/evaluating/SKILL.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/plugin/skills/evaluating/SKILL.md b/plugin/skills/evaluating/SKILL.md index a42f6da..f8df809 100644 --- a/plugin/skills/evaluating/SKILL.md +++ b/plugin/skills/evaluating/SKILL.md @@ -50,23 +50,24 @@ Analyze screened patents by decomposing claims into elements and storing analysi 1. **Get Patents to Analyze**: - Invoke `Skill: investigation-fetching` with request "Get list of relevant patents without evaluation" -2. **For each patent**, execute Steps 2a–2d in order: - - **2a. Fetch Patent Data**: - - Invoke `Skill: google-patent-cli:patent-fetch` with patent ID - - Extract: title, abstract, all claims - - **2b. Analyze Claims**: - - Extract ALL claims from the patent (both independent and dependent) +2. **Batch Fetch Patent Data** (up to 10 patents in parallel): + - Split patents into batches of 10 + - For each batch, invoke `Skill: google-patent-cli:patent-fetch` for all patents **in parallel** + - After `fetch_patent` returns each dataset, use `execute_cypher` to get claims. + **You MUST use this EXACT query — do NOT modify the node label or property names:** + ```cypher + MATCH (c:claims) RETURN c.number, c.text + ``` + - **CRITICAL**: Do NOT add `ORDER BY toInteger(c.number)` — it causes `c.text` to return `expression: null` due to a Cypher parser bug. + Also do NOT use `MATCH (p:Patent)-[:claims]->(c:claims)` (relationship pattern), `[:HAS_CHILD]->(c:claim)`, `[:claim]->(c:claim)`, `p.claims`, or `[:claims]->(c:claim)`. + +3. **Analyze and Record** (for each patent): + - Extract ALL claims (both independent and dependent) - For EACH claim, decompose into constituent elements (A, B, C...) - - **2c. Record Claims**: - Invoke `Skill: investigation-recording` with request "Record claims for patent : " - - **2d. Record Elements**: - Invoke `Skill: investigation-recording` with request "Record elements for patent : " -3. **Verify Results**: Query database to confirm all claims and elements recorded +4. **Verify Results**: Confirm all claims and elements are recorded in the database ## State Management