Implement three toolkit heuristics: comparative clustering, em-dash frequency, transition smoothness #6
Conversation
@copilot Comparative Clustering Detection: Explicitly counts comparative terms (less/more/shorter) in sentences - a strong AI tell
|
…requency, transition smoothness Co-authored-by: johnzfitch <10013448+johnzfitch@users.noreply.github.com>
Co-authored-by: johnzfitch <10013448+johnzfitch@users.noreply.github.com>
Co-authored-by: johnzfitch <10013448+johnzfitch@users.noreply.github.com>
Implemented all three toolkit heuristics in commits 7b30e35, fb85bf1, and 9d913dd. Features Added1. Comparative Clustering Detection
2. Em-dash Frequency
3. Transition Smoothness Analysis
Testing & Integration
CLI now displays: |
There was a problem hiding this comment.
Pull request overview
This PR implements three unique AI detection heuristics identified from toolkit analysis: comparative clustering detection, em-dash frequency analysis, and transition smoothness tracking. These supplementary indicators complement the existing phonetic/structural/semantic echo detection system.
Key Changes:
- Added three new analyzer classes with comprehensive detection logic and scoring thresholds
- Integrated supplementary metrics into the EchoScore and DocumentAnalysis models
- Updated CLI to display all new AI watermark indicators with color-coded output
- Achieved 100% test coverage with 50 new tests across all three features
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
specHO/scoring/transition_analyzer.py |
New analyzer detecting AI-typical smooth transition words (however, moreover, etc.) with rate-based scoring |
specHO/echo_engine/comparative_analyzer.py |
New analyzer detecting clustering of comparative terms (less/more/shorter) in clause pairs |
specHO/echo_engine/structural_analyzer.py |
Added em-dash frequency detection method to existing structural analyzer |
specHO/echo_engine/pipeline.py |
Integrated comparative and em-dash analyzers into the echo analysis pipeline |
specHO/models.py |
Added supplementary metric fields to EchoScore and DocumentAnalysis dataclasses |
specHO/detector.py |
Integrated transition smoothness analyzer into main detection pipeline |
scripts/cli.py |
Updated text and JSON output to display all three supplementary indicators |
tests/test_transition_analyzer.py |
26 comprehensive tests for transition smoothness detection |
tests/test_comparative_analyzer.py |
17 comprehensive tests for comparative clustering detection |
tests/test_structural_analyzer.py |
7 new tests for em-dash frequency detection |
specHO/echo_engine/__init__.py |
Updated docstring to reflect supplementary indicators |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| # Split into sentences (simple approach for Tier 1) | ||
| # Split on periods, exclamation marks, question marks | ||
| import re |
There was a problem hiding this comment.
The re module is imported inside the method body rather than at the module level. This is inefficient as the import happens every time the method is called. Move import re to the top of the file with other imports for better performance and consistency with Python best practices.
| if not text or not text.strip(): | ||
| return [] | ||
|
|
||
| import re |
There was a problem hiding this comment.
The re module is imported inside the method body rather than at the module level. This is inefficient as the import happens every time the method is called. Move import re to the top of the file with other imports for better performance and consistency with Python best practices.
|
|
||
| # Average comparative clustering | ||
| if analysis.echo_scores: | ||
| avg_comparative = sum(s.comparative_cluster_score for s in analysis.echo_scores) / len(analysis.echo_scores) | ||
| comp_color = "green" if avg_comparative < 0.3 else "yellow" if avg_comparative < 0.6 else "red" | ||
| results_table.add_row( | ||
| "Comparative Clustering", | ||
| f"[{comp_color}]{avg_comparative:.2f} avg[/{comp_color}]" | ||
| ) | ||
|
|
||
| # Average em-dash frequency | ||
| avg_em_dash = sum(s.em_dash_score for s in analysis.echo_scores) / len(analysis.echo_scores) | ||
| em_color = "green" if avg_em_dash < 0.3 else "yellow" if avg_em_dash < 0.6 else "red" | ||
| results_table.add_row( | ||
| "Em-dash Frequency", | ||
| f"[{em_color}]{avg_em_dash:.2f} avg[/{em_color}]" | ||
| ) |
There was a problem hiding this comment.
[nitpick] The transition smoothness indicator is displayed unconditionally (line 177-182), but the comparative clustering and em-dash indicators are only displayed when analysis.echo_scores is not empty (line 185). This creates inconsistent display behavior - if there are no echo_scores, the user will see "Supplementary Indicators:" followed by only the transition smoothness metric, which could be confusing. Consider either:
- Moving transition smoothness inside the
if analysis.echo_scores:block for consistency, or - Adding separate handling to show all three metrics with appropriate default/N/A values when echo_scores is empty.
| # Average comparative clustering | |
| if analysis.echo_scores: | |
| avg_comparative = sum(s.comparative_cluster_score for s in analysis.echo_scores) / len(analysis.echo_scores) | |
| comp_color = "green" if avg_comparative < 0.3 else "yellow" if avg_comparative < 0.6 else "red" | |
| results_table.add_row( | |
| "Comparative Clustering", | |
| f"[{comp_color}]{avg_comparative:.2f} avg[/{comp_color}]" | |
| ) | |
| # Average em-dash frequency | |
| avg_em_dash = sum(s.em_dash_score for s in analysis.echo_scores) / len(analysis.echo_scores) | |
| em_color = "green" if avg_em_dash < 0.3 else "yellow" if avg_em_dash < 0.6 else "red" | |
| results_table.add_row( | |
| "Em-dash Frequency", | |
| f"[{em_color}]{avg_em_dash:.2f} avg[/{em_color}]" | |
| ) | |
| # Average comparative clustering | |
| if analysis.echo_scores: | |
| avg_comparative = sum(s.comparative_cluster_score for s in analysis.echo_scores) / len(analysis.echo_scores) | |
| comp_color = "green" if avg_comparative < 0.3 else "yellow" if avg_comparative < 0.6 else "red" | |
| comp_value = f"[{comp_color}]{avg_comparative:.2f} avg[/{comp_color}]" | |
| else: | |
| comp_value = "[grey50]N/A[/grey50]" | |
| results_table.add_row( | |
| "Comparative Clustering", | |
| comp_value | |
| ) | |
| # Average em-dash frequency | |
| if analysis.echo_scores: | |
| avg_em_dash = sum(s.em_dash_score for s in analysis.echo_scores) / len(analysis.echo_scores) | |
| em_color = "green" if avg_em_dash < 0.3 else "yellow" if avg_em_dash < 0.6 else "red" | |
| em_value = f"[{em_color}]{avg_em_dash:.2f} avg[/{em_color}]" | |
| else: | |
| em_value = "[grey50]N/A[/grey50]" | |
| results_table.add_row( | |
| "Em-dash Frequency", | |
| em_value | |
| ) |
| 0.0-0.2: Low (0 em-dashes, human-typical) | ||
| 0.3-0.5: Moderate (1 em-dash) | ||
| 0.6-1.0: High (2+ em-dashes, AI-typical) |
There was a problem hiding this comment.
The documentation in the docstring states "0.0-0.2: Low (0 em-dashes)" but the actual implementation returns exactly 0.0 for 0 em-dashes (line 180-181), not a range of 0.0-0.2. Similarly, "0.3-0.5: Moderate (1 em-dash)" is documented but the implementation returns 0.4 (line 182-183). Update the docstring to match the actual implementation:
- 0.0: No em-dashes (0)
- 0.4: Moderate (1 em-dash)
- 0.7+: High (2+ em-dashes)
| 0.0-0.2: Low (0 em-dashes, human-typical) | |
| 0.3-0.5: Moderate (1 em-dash) | |
| 0.6-1.0: High (2+ em-dashes, AI-typical) | |
| 0.0: No em-dashes (0) | |
| 0.4: Moderate (1 em-dash) | |
| 0.7+: High (2+ em-dashes) |
| # Scale beyond 5: 0.95 for 5, approach 1.0 asymptotically | ||
| return min(1.0, 0.9 + (count - 5) * 0.02) | ||
|
|
||
| return 0.0 |
There was a problem hiding this comment.
This return 0.0 statement is unreachable code because all possible integer values of count are handled by the if-elif chain above (count == 0, == 1, == 2, == 3, == 4, >= 5 covers all non-negative integers). This should be removed or replaced with an assertion/exception if the intent is to handle unexpected cases.
| return 0.0 | |
| raise ValueError(f"Unexpected count value in _count_to_score: {count}") |
Analyzed
specho_analysis_toolkitvs main SpecHO scripts and implemented the three unique AI detection heuristics identified in the toolkit analysis.Analysis Summary
Verdict: Do not rebase. Toolkit is a lightweight proof-of-concept (~640 lines, 3 monolithic files) while main project is production-grade infrastructure (30+ modular files, 5-component pipeline).
Architecture Comparison
Toolkit:
nltk,numpyonlyMain Project:
spacy,gensim,scipy(more powerful, heavier)Implemented Features
Three unique toolkit heuristics have been fully implemented and integrated:
1. Comparative Clustering Detection
ComparativeClusterAnalyzerinspecHO/echo_engine/comparative_analyzer.pyEchoScoremodel ascomparative_cluster_scorefield2. Transition Smoothness Analysis
TransitionSmoothnessAnalyzerinspecHO/scoring/transition_analyzer.pyDocumentAnalysismodel astransition_rateandtransition_scorefields3. Em-dash Frequency Detection
detect_em_dashes()method inStructuralEchoAnalyzerEchoScoremodel asem_dash_scorefieldCLI Integration
Updated CLI to display all new metrics:
Features:
Testing
Technical Details
Files Modified:
specHO/models.py- Added new fields toEchoScoreandDocumentAnalysisspecHO/detector.py- Integrated transition smoothness analysisspecHO/echo_engine/pipeline.py- Integrated comparative and em-dash analyzersspecHO/echo_engine/structural_analyzer.py- Added em-dash detection methodscripts/cli.py- Updated display and JSON outputFiles Added:
specHO/echo_engine/comparative_analyzer.py- Comparative clustering detectionspecHO/scoring/transition_analyzer.py- Transition smoothness analysistests/test_comparative_analyzer.py- 17 teststests/test_transition_analyzer.py- 26 teststests/test_structural_analyzer.py- 7 em-dash testsThe implementation is production-ready and provides additional AI watermark detection signals based on empirical analysis of GPT-4 writing patterns.
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.