Feat: kg evaluator

[CHERRY-ui8]

This pull request introduces a robust and multi-faceted knowledge graph (KG) evaluation system. It aims to provide developers with tools to automatically assess the quality of generated KGs by examining their accuracy in extracting information, consistency in data representation, and the overall structural integrity. This feature enhances the reliability and trustworthiness of KGs produced by the system.

[gemini-code-assist]

Summary of Changes

Hello @CHERRY-ui8, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a robust and multi-faceted knowledge graph (KG) evaluation system. It aims to provide developers with tools to automatically assess the quality of generated KGs by examining their accuracy in extracting information, consistency in data representation, and the overall structural integrity. This feature enhances the reliability and trustworthiness of KGs produced by the system.

Highlights

• Comprehensive KG Evaluation: Introduced a new KGQualityEvaluator module that provides a holistic assessment of knowledge graphs across accuracy, consistency, and structural robustness.
• LLM-as-Judge Accuracy: Implemented accuracy evaluation for entity recognition, relation extraction, and triple validation using an LLM as a judge, with support for concurrent requests.
• Consistency and Structure Metrics: Added evaluators for detecting attribute value conflicts (consistency) and measuring structural robustness (noise ratio, connected components, average degree, power law distribution).
• Dedicated CLI and Examples: Provided a new command-line interface (evaluate_kg.py) and a shell script example (evaluate_kg.sh) for easy execution and configuration of the KG evaluation.
• Refactoring of Base Evaluator: The base_evaluator.py module was refactored and moved to graphgen/bases/base_evaluator.py to improve module organization.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive Knowledge Graph (KG) Quality Evaluation module, encompassing accuracy, consistency, and structural robustness assessments. New files accuracy_evaluator.py, consistency_evaluator.py, and structure_evaluator.py implement these evaluations, with KGQualityEvaluator.py orchestrating them and utils.py providing helper functions like convert_to_networkx and text retrieval. An entry point evaluate_kg.py is added with command-line arguments for configuration and output, along with a shell script example and a detailed README.md. Review comments highlight several areas for improvement: the summary printing logic in evaluate_kg.py should use the pass flags from the StructureEvaluator results instead of re-implementing thresholds; the evaluation of relations and triples in accuracy_evaluator.py is redundant and should be consolidated; the metric calculation logic (precision, recall, F1) in accuracy_evaluator.py is duplicated and needs extraction into a helper method; structural metric thresholds and magic numbers for text truncation and conflict limits should be defined as class-level constants; the approximation of recall as precision in accuracy_evaluator.py is noted as potentially misleading; the get_relevant_text utility function's fallback behavior is questioned for potential memory issues; and minor code hygiene issues like unused imports and import placement are identified.