[Enhancement]Create a “Plain Language Score”

[bashandbone]

Category

new license or license feature/enhancement

Feature

Develop a plain language score to reliably measure writing “plainness”

Use Case

We could use it for educating and highlighting differences between licenses, rewarding plainness “in the wild” and to hold ourselves accountable with CI/CD scanning

Benefit

Make the world a little bit easier to understand.

Alternatives

There are many measures of readability, such as Flesch-Kincaid, Gunning-Fog, SMOG. I don’t feel these wholly capture the important parts of clear and accessible writing.

Impact

Drive us to write more plainly, highlight efforts (or non-efforts) at plainness, and maybe even create a standard measure for measuring plain writing.

Resources

There’s a decent body of research on readability.

Contact Details

adam@plainlicense.org

Additional Information

Enhancement Proposal: Plain Language Score

We need a way to measure how easy licenses are to read. Let's call it the "Plain Language Score". Here's what I'm thinking:

What's the idea?

We want to create a fair way to measure the readability of all licenses, including ours. This score would help us show how much clearer our licenses are compared to the originals. It could also show the benefits of plain writing and help us reward and recognize when people do it well.

What might the score look like?

Here are some ideas for what the score could measure:

1. Use of legal jargon
2. Sentence complexity
3. Organization (Use of headers, lists/bullets, and paragraph density)
4. Ratio of abstract to concrete words
5. A traditional readability score (like Flesch-Kincaid)

These are just ideas! We can change any part of this - what we measure, how much each part counts, or how we measure it. The main goal is to have a clear and fair way to score license readability. I also think there is an opportunity to modernize readability measures with NLP techniques.

How would we build it?

We'd make this a separate project from Plain License, but affiliated with it. It would have:

• Its own repository
• Version numbers for both the code and the scoring system
• Clear documentation on how it works

How would we manage changes?

Once we're out of the testing phase, we should think about how we approve changes to the score. We want it to be reliable, so we can't just change it whenever we want. Maybe we could have a group of experts review big changes, or a proposal review process?

What's next?

If you like this idea, we could:

1. Start a new repository for this project
2. Begin working on a basic version of the score
3. Test it on different licenses to see how it works
4. Ask for feedback from our community and legal experts

Example: How it might work

Let's look at a quick example of how this score might work. We'll use a 1-20 scale, where lower is better (like grade levels).

Here's a snippet from a made-up license:

Original: "The Licensee shall not utilize the Software in any manner that may impair the integrity or performance of the system."

Our version: "You may not use the work in a way that could damage or slow down the service."

Let's score them:

1. Legal Jargon (30%):

   • Original: Lots of jargon (15/20)
   • Ours: No jargon (2/20)

2. Sentence Variety (25%):

   • Original: One complex sentence (14/20)
   • Ours: One simple sentence (5/20)

3. Document Organization (25%):

   • Both: N/A for this short example

4. Concrete Language (10%):

   • Original: Mostly abstract (16/20)
   • Ours: Mostly concrete (4/20)

5. Flesch-Kincaid (10%):

   • Original: Grade 16
   • Ours: Grade 6

Putting it all together (with some rounding):

Original: (15 * 0.3) + (14 * 0.25) + (16 * 0.1) + (16 * 0.1) = 10.9
Our version: (2 * 0.3) + (5 * 0.25) + (4 * 0.1) + (6 * 0.1) = 2.85

Final scores:

• Original: 11 (rounded up, about 11th grade)
• Our version: 3 (rounded up, about 3rd grade)
  [my scoring system here clearly needs work]

Again, this is just an example - we can adjust how we calculate this as we develop the system. Some other things we could potentially include in our measure:

• ratio of active vs passive voice
• Frequency of conditional statements (e.g. if-then). I found these were the most difficult to translate to plain language.
• visual accessibility (as a component of organization). This could be a lot of things, to name a few: frequency of headers, length of median text blocks, frequency of bullets.
• Consistent word choice. How well it avoids using multiple synonyms in the same text.
• Clarity of definitions. If it includes a definitions section, I think there should be a higher penalty if the definitions suffer from the same inaccessible writing...

Let's discuss and refine this idea together!

We could also use it in our CI/CD process to enforce a target score, or identify parts of the site that we need to improve for clarity.

[bashandbone]

Additional research to support the planning:

Beyond readability: measuring what makes text actually understandable

Traditional readability formulas capture only 23–34% of what makes text easy or hard to understand. Three decades of cognitive science, computational linguistics, and NLP research have established that "understandability" is a multi-dimensional construct far richer than word length and sentence length — the only features classic formulas measure. The field now offers validated theoretical frameworks, mature NLP tools like Coh-Metrix, and emerging transformer-based approaches that can quantify cohesion, syntactic complexity, abstractness, and discourse structure. Yet significant gaps remain: no existing tool measures background knowledge requirements, visual layout effects, or the reader-text interaction that cognitive science identifies as central to comprehension. For a tool like readscore, the path forward is clear: combine multiple computational dimensions into a composite score, using freely available Python libraries (TextDescriptives, LFTK, textstat, sentence-transformers) as a foundation, while acknowledging what remains beyond automated measurement.

---

Understandability and readability are distinct constructs

Researchers consistently distinguish between readability — a property of the text's surface features — and comprehensibility/understandability — a property of the reader-text interaction. Oksuz & Keskin (2022) demonstrated this empirically: in a study with 121 fourth-graders, the "readable" versions (simplified per readability formulas) were actually less comprehensible than the originals, because simplification had stripped away the connective tissue that helped readers build meaning. Acar & İşisağ (2018) found statistically significant divergence between readability and comprehensibility scores on the same texts.

The theoretical foundation for this distinction comes from Kintsch's Construction-Integration model (1988, 1998), which identifies three levels of mental representation during reading. The surface structure captures verbatim wording — what readability formulas approximate. The propositional textbase encodes explicit meaning as interconnected propositions. The situation model integrates text content with the reader's prior knowledge into a coherent mental model of what the text describes. Deep understanding requires constructing a situation model, and the same text produces different situation models for different readers depending on their knowledge — which is precisely why readability (a text-only metric) diverges from understandability (a text × reader outcome).

Zwaan and Radvansky's (1998) Event-Indexing Model extended this by identifying five dimensions along which readers track comprehension: time, space, entity, causality, and intentionality. Discontinuities on any dimension increase processing difficulty, and effects are additive. This model has been validated through eye-tracking and reading-time studies over 25 years, with Zwaan (2025) recently extending it to include information source tracking. Van den Broek's Landscape Model (1999) adds another crucial concept: standards of coherence — readers maintain varying thresholds of understanding based on motivation, goals, and fatigue, meaning the same text difficulty produces different comprehension depending on reader engagement.

The factors that actually drive comprehension difficulty

Cognitive science has identified at least eight major factors beyond vocabulary and sentence length:

• Idea density: Kintsch & Keenan (1973) showed that recall depends on proposition count, not word count. A 2021 Medical Teacher study found reducing lecture idea density improved comprehension, with L2 speakers 5.96× more likely to achieve mastery on low-density material.
• Coherence and cohesion: Referential overlap (shared concepts across sentences), causal connectives, and temporal markers all affect processing. McNamara & Kintsch (1996) discovered a reverse cohesion effect: high-cohesion text helps low-knowledge readers but can hurt high-knowledge readers by reducing active processing.
• Syntactic complexity: Not just sentence length, but parse tree depth, center-embedding, dependency distance, and clause nesting. Biber, Larsson & Hancock (2024) found grammatical complexity requires a nine-dimension model.
• Background knowledge requirements: Bransford & Johnson (1972) demonstrated that identical text is incomprehensible without relevant prior knowledge. McCarthy et al. (2023) decomposed prior knowledge into amount, accuracy, specificity, and coherence.
• Inference demands: Graesser, Singer & Trabasso (1994) showed readers routinely generate bridging and elaborative inferences to maintain coherence — texts with more implicit connections demand more cognitive work.
• Abstractness: Via Paivio's Dual Coding Theory, concrete words activate both verbal and imagery systems while abstract words engage only verbal processing, making abstract text harder to comprehend and recall.
• Working memory load: Sweller's Cognitive Load Theory (1988) identifies intrinsic load (content complexity), extraneous load (presentation quality), and germane load (schema building). Only 2–4 chunks can be processed simultaneously.
• Discourse structure: Meyer's (1975) research shows that texts organized with clear top-level structures (comparison, cause-effect, problem-solution) are better comprehended and recalled.

Validated multi-level frameworks

The most complete framework is Graesser & McNamara's six-level model (2011): words → syntax → explicit textbase → referential situation model → discourse genre/rhetorical structure → pragmatic communication. Coh-Metrix operationalizes this into five measurable principal components: narrativity, syntactic simplicity, word concreteness, referential cohesion, and deep (causal) cohesion — derived from PCA on 53 measures across 37,520 texts. The Common Core State Standards adopted a complementary three-part model (quantitative measures + qualitative measures + reader/task considerations), explicitly acknowledging that automated metrics alone are insufficient.

---

Traditional readability formulas: what they get wrong

The Flesch-Kincaid Grade Level (1975), Flesch Reading Ease (1948), SMOG (1969), Gunning Fog (1952), Dale-Chall (1948/1995), Coleman-Liau (1975), and Automated Readability Index (1967) all share a fundamental limitation: they measure only 1–2 surface features (sentence length and syllable count or word-list membership) as proxies for difficulty. Classic validation studies reported correlations of r = .7–.8 with observed difficulty (Chall 1958; Fry 1989), but these numbers are misleading for three reasons.

First, the validation criterion was typically cloze testing, which itself measures the very surface features that formulas capture — creating circular validation. Second, different formulas disagree by up to 5 grade levels on the same text (Wang et al., 2013), with SMOG consistently scoring 3.1 grade levels higher than Flesch-Kincaid. Third, using the more rigorous CLEAR corpus (~4,724 texts rated by 1,116 teachers), Crossley et al. (2022) found traditional formulas predict only 23–34% of variance in human-judged reading ease. A 2025 eye-tracking study (Yevtushenko et al.) went further, finding that "existing readability formulas are moderate to poor predictors of reading ease" — and even modern ML systems and frontier LLMs performed poorly at predicting real-time reading difficulty measured via eye movements.

The manipulation vulnerability is particularly damning. Duffy & Kabance tested four versions of identical content with mechanically shortened sentences and simplified words: readability scores dropped by 6 grade levels, but comprehension did not improve. Davison & Kantor (1982) documented the classic case where changes that improved readability scores actually worsened comprehension by destroying cohesion. Redish (2000) captured the absurdity: "Should we be happy if 50% of our readers understand 50% of our documents?" — referring to the 50/50 criterion underlying formula calibration.

The most meaningful post-2000 advancement in formula-based approaches is CAREC (Crossley, Skalicky & Dascalu, 2019), which incorporates word age-of-acquisition, n-gram frequency, cohesion, and sentiment features. CAREC achieved the strongest correlation (r = .580) with CLEAR ratings among all formulas tested. Still, even CAREC explains only ~34% of variance — confirming that counting-based approaches have a ceiling.

---

NLP tools now measure what formulas cannot

Coh-Metrix: the gold standard for multi-level text analysis

Coh-Metrix (Graesser, McNamara et al., 2002–present) computes 108+ indices (200+ in the batch version) organized across 11 categories: descriptive statistics, easability components, referential cohesion, latent semantic analysis, lexical diversity, connectives, situation model, syntactic complexity, syntactic pattern density, word information, and traditional readability. Its five principal components — narrativity, syntactic simplicity, word concreteness, referential cohesion, and deep cohesion — were derived via PCA and account for 67% of variance in text difficulty across grade levels.

Validation studies demonstrate that three Coh-Metrix variables (lexical co-referentiality, word frequency, syntactic complexity) yield predictions of reading difficulty comparable to traditional formulas while "better reflecting the psycholinguistic factors of reading comprehension" (Crossley et al., 2008). The Coh-Metrix L2 Reading Index outperformed traditional formulas for discriminating between levels of simplified L2 texts. However, Coh-Metrix is not open source for English — it's available only as a web tool limited to 15,000 characters.

The Crossley-Kyle-McNamara tool ecosystem

The most comprehensive open set of linguistic analysis tools comes from Scott Crossley, Kristopher Kyle, and Danielle McNamara's research groups, available at linguisticanalysistools.org:

TAALES (Tool for the Automatic Analysis of Lexical Sophistication) provides 400+ indices measuring word frequency, range, n-gram associations, psycholinguistic properties (concreteness, familiarity), and contextual distinctiveness. TAALES indices explained 58% of variance in lexical proficiency scores (Kyle et al., 2018). TAACO (Tool for the Automatic Analysis of Cohesion) computes ~150 indices of local and global cohesion, including LSA, LDA, and word2vec-based semantic similarity. TAASSC measures clausal and phrasal syntactic complexity plus usage-based sophistication indices. TAALED implements the most rigorous lexical diversity measures (MTLD, HD-D). Together, these tools produce 843 total indices — though they are standalone desktop applications, not pip-installable Python packages.

Transformer models push the boundary

The integration of transformer models with handcrafted linguistic features represents the current state of the art. Lee et al. (2021/2022) achieved 99% accuracy on OneStopEnglish using a RoBERTa + Random Forest hybrid, a 20.3% improvement over prior approaches. The critical finding is that hybrid models dramatically outperform either approach alone — transformers capture semantic and contextual features while handcrafted features capture structural and psycholinguistic properties.

GPT-4 shows remarkable zero-shot readability assessment capability. Trott (2024) found GPT-4 Turbo ratings correlated r = 0.76 with human CLEAR judgments — higher than the split-half reliability between random splits of human raters (r = 0.63) — making it the single best predictor of human readability judgments among all methods tested. This suggests LLMs have internalized implicit models of text difficulty, though they remain poorly calibrated for domain-specific texts.

Key benchmark datasets enabling this research include the CLEAR Corpus (~4,724 excerpts with continuous teacher-rated scores, MIT-licensed), Newsela (1,130+ articles with 5 professionally simplified levels), OneStopEnglish (189 Guardian articles at 3 ESL levels, freely available), and WeeBit (3,125 documents across 5 age groups).

---

Legal text difficulty stems from writing style, not complex concepts

The MIT research group of Martínez, Mollica & Gibson has produced the most rigorous empirical work on legal text comprehension. Their 2022 Cognition paper analyzed a corpus of ~10 million words and found that contracts use passive voice 4.7× more frequently than standard English (0.758 vs. 0.160 per sentence). But the single most impactful feature was center-embedding — inserting long definitions mid-sentence — which taxed working memory far beyond what sentence length alone would predict. When center-embedded structures were replaced with separate definitions, laypeople's comprehension improved from ~38% to 45–50% recall.

Their 2023 PNAS follow-up, "Even lawyers don't like legalese," demonstrated that 100+ lawyers also found legalese harder to understand, recalled less from it, rated plain English as higher quality, and said they'd be more likely to hire the writer of plain English versions. Lawyers considered plain English equally enforceable. The persistence of legalese, the authors concluded, is best explained by template copying rather than functional necessity.

For the PlainLicense project specifically, the research on open source license comprehension is sobering. Almeida et al. (2017/2019) surveyed 375 developers and found their understanding of GPL 3.0, LGPL 3.0, and MPL 2.0 matched expert opinion in only 62% of cases — dropping further when multiple licenses were involved. Meanwhile, Obar & Oeldorf-Hirsch (2020) found 74% of users skip privacy policies entirely, with those who "read" them spending an average of 73 seconds on documents requiring 29–32 minutes. McDonald & Cranor (2008) estimated that reading all privacy policies would cost an average user 244 hours per year.

The plain language movement, despite decades of advocacy and the 2010 Plain Writing Act, has produced no measurable improvement in federal law readability (Martínez et al., 2024). The International Plain Language Federation defines its standard functionally: readers should be able to find what they need, understand what they find, and act on that understanding. The most successful model for license comprehension remains Creative Commons' three-layer approach: legal code, human-readable deed, and machine-readable metadata.

For legal-specific NLP, Legal-BERT (Chalkidis et al., 2020) — pre-trained on 12 GB of legal text — outperforms vanilla BERT on legal tasks and is available on HuggingFace (nlpaueb/legal-bert-base-uncased). Recent work on graded legal simplification ("From Jargon to Clarity," 2025) introduced multi-level simplification (Skilled, Intermediate, Basic) with multi-criteria evaluation covering readability, lexical simplicity, semantic preservation, and entailment.

---

Document structure and typography have measurable effects on comprehension

A meta-analysis of 103 signaling studies (Schneider et al., 2018; N = 12,201) found headings and other signals produce a medium effect on retention (g+ = 0.53) and a small-to-medium effect on transfer (g+ = 0.33). Lorch et al. showed that headings don't merely help recall — they restructure readers' mental representations, making signaled topics significantly more prominent in summaries while reducing recall of unsignaled topics.

Graphic organizers show even stronger effects: g+ = 0.59–0.70 for instructor-provided organizers (2025 meta-analysis). Bullet points improve recall of listed items but can reduce recall of surrounding text — they redistribute rather than uniformly enhance attention, and they strip away connective words (because, therefore) that express relationships between ideas.

For typography, the evidence is nuanced. Line length research supports the classic 50–75 character guideline, with 66 characters per line as the traditional optimum, though screen reading may favor slightly longer lines for speed (while users still prefer shorter lines). The serif vs. sans-serif debate is essentially settled at null — no significant difference on modern screens. Line spacing at 1.5× improves reading accuracy by up to 20%. The widely discussed disfluency effect (harder-to-read fonts improving deep processing) has been largely debunked by meta-analysis: disfluent fonts affect metacognitive judgments (d = −0.43) and increase reading time (d = 0.52) but show no overall effect on actual recall (d = −0.01) or transfer (d = 0.03).

The critical gap here is computational: no existing readability tool measures structural or visual features. All tools analyze plain text. This represents a significant opportunity for readscore — HTML/markup analysis could extract heading hierarchy, paragraph length distribution, list frequency, and text density, while no current tool attempts this.

---

Building readscore: a practical implementation roadmap

The Python tool ecosystem

The most efficient foundation for readscore combines four freely available libraries:

TextDescriptives (pip install textdescriptives) is the strongest single starting point. Built on spaCy v3, it computes readability formulas, first-order and second-order coherence (via embedding cosine similarity), dependency distance metrics, POS proportions, and text quality heuristics in a unified pipeline. LFTK (pip install lftk) provides 200+ handcrafted linguistic features including Kuperman age-of-acquisition norms, entity density, and comprehensive POS analysis, organized in a searchable taxonomy. textstat (pip install textstat) offers the most complete implementation of traditional formulas with a useful text_standard() consensus function. lexicalrichness (pip install lexicalrichness) implements all major text-length-independent diversity measures including MTLD, HD-D, and vocd-D.

For psycholinguistic word properties, three databases are essential: Brysbaert et al. (2014) concreteness norms (37,058 words rated 1–5, freely available on OSF), Kuperman et al. (2012) age-of-acquisition norms (30,121 words), and SUBTLEX-US word frequencies (74,286 words with Zipf scale, available from ugent.be). The Python package wordfreq provides convenient access to word frequencies in 40+ languages. All three can be loaded as DataFrames and joined on word lemma to compute mean scores per text.

For coherence measurement beyond TextDescriptives' built-in capabilities, sentence-transformers (pip install sentence-transformers) enables computing cosine similarity between consecutive sentence embeddings using models like all-MiniLM-L6-v2. This captures semantic coherence rather than mere lexical overlap.

A newer entry, ELFEN (pip install elfen, March 2025), adds psycholinguistic features including emotion and morphological analysis, built on polars for speed with large datasets.

The recommended composite architecture

A multi-dimensional understandability score should combine at least eight measurable components:

1. Surface difficulty — Flesch-Kincaid, SMOG via textstat
2. Lexical sophistication — mean word frequency (wordfreq), mean age-of-acquisition (Kuperman norms via LFTK)
3. Lexical diversity — MTLD, HD-D via lexicalrichness
4. Syntactic complexity — mean dependency distance, parse tree depth via TextDescriptives
5. Cohesion — sentence-to-sentence semantic similarity via sentence-transformers
6. Abstractness — mean concreteness via Brysbaert norms
7. Entity/concept density — entities per sentence via spaCy NER
8. Connective density — discourse connectives per sentence via LFTK or custom matching

These dimensions can be combined through PCA (replicating Coh-Metrix's approach of deriving interpretable principal components), weighted scoring with expert-tuned weights, or a machine learning model trained on the CLEAR corpus or Newsela. The hybrid approach — handcrafted features fed alongside transformer embeddings into a gradient boosting model — consistently achieves the best performance in the literature.

What remains genuinely hard to measure

Five dimensions that cognitive science identifies as important resist current computational measurement. Background knowledge requirements cannot be estimated without modeling the reader — a text on quantum entanglement is trivial for a physicist and impenetrable for a layperson regardless of its readability score. Inference demands — how much readers must infer from unstated connections — have partial proxies (causal cohesion gaps in Coh-Metrix) but no reliable automated measure. Pragmatic comprehension (irony, implicature, indirect speech acts) requires cultural and contextual knowledge beyond current NLP. Visual/structural layout effects are well-documented but no tool processes rendered documents. And the reader-text interaction itself — the central construct in all cognitive models — is fundamentally unmeasurable from text alone.

---

Conclusion

The gap between what cognitive science knows about comprehension and what readability tools measure is real but shrinking. The field has moved through three generations: first-generation formulas (1920s–1960s) capturing two surface dimensions; second-generation tools like Lexile adding statistical rigor to the same two dimensions; and third-generation tools like Coh-Metrix measuring discourse coherence, vocabulary depth, and genre across 100+ indices. The emerging fourth generation — hybrid transformer + linguistic feature models — achieves near-ceiling classification accuracy on benchmark datasets.

For readscore, the practical insight is that the infrastructure for multi-dimensional understandability assessment already exists in freely available Python packages — what's missing is integration into a coherent tool with a composite scoring framework. TextDescriptives + LFTK + textstat + sentence-transformers + psycholinguistic norm databases cover approximately 6 of the 8 recommended dimensions with minimal custom code. The remaining challenge is defining appropriate weighting or training a model to combine these dimensions into scores that correlate with actual human comprehension outcomes.

For PlainLicense, the research provides specific, actionable guidance. The single most impactful change for legal text is eliminating center-embedded definitions (Martínez et al., 2022). Active voice, shorter sentences, common vocabulary, and logical organization all help — but even with all these changes, comprehension of legal concepts remains imperfect because readers bring conflicting prior beliefs (Masson & Waldron, 1994). This argues for the Creative Commons model: supplement plain language with concrete examples of what is and isn't permitted, brief explanations of why provisions exist, and structured visual design with clear headings and bulleted permissions/restrictions. The research is unambiguous that current license text fails its audience — developers understand open source licenses only 62% of the time, and 74% of users skip terms entirely.

[ckmtools]

For a robust Plain Language Score, combining multiple established formulas tends to work better than any single metric. Each dimension captures something different:

Traditional readability (sentence + word length):

• Flesch Reading Ease: widely understood baseline
• Gunning Fog: penalizes polysyllabic words more aggressively — good for jargon detection

Vocabulary complexity:

• Dale-Chall: built around a curated list of "familiar" words — works well for legal text because it flags uncommon formal terms
• Coleman-Liau: character-based, less susceptible to hyphenation noise in PDFs

Structure (what classic formulas miss):

• Passive voice ratio
• Nominalizations ("utilization" vs "use", "implementation" vs "implement") — legal text is full of these and they add measurable cognitive load

A practical v1 for license text: weighted average of Flesch + Gunning Fog + a legal jargon frequency score. The jargon component is where you can differentiate — a curated list of terms like "whereas", "hereinafter", "notwithstanding", "indemnify" gives you something domain-specific that generic readability tools don't have.

For CI/CD integration, Python's textstat library covers most of the traditional formulas with no external dependencies.

Disclosing I am building TextLens API — readability, sentiment, keywords via REST, no Python deps — waitlist at https://ckmtools.dev/api/ if useful.