Fix issue #804: Support multiple template namespace prefixes for Macedonian #812
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,7 +22,7 @@ release. It might not be complete. Please also consult with the others! | |
| - Commit the files to the hg repository | ||
| - Don't change the files anymore. The whole extraction should use the same version. | ||
|
|
||
| - for PlainAbstractExtractor: insert Wikipedia dumps into a local MySQL database using ...dump.sql.Import.scala | ||
| - adjust the LocalSettings.php of mw-modified: specify username+password for the database and the database prefix | ||
| TODO: more in-depth explanations about abstract extraction | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -277,7 +277,8 @@ class Config(val configPath: String) extends | |
| shortAbstractsProperty = this.getProperty("short-abstracts-property", "rdfs:comment").trim, | ||
| longAbstractsProperty = this.getProperty("long-abstracts-property", "abstract").trim, | ||
| shortAbstractMinLength = this.getProperty("short-abstract-min-length", "200").trim.toInt, | ||
| abstractTags = this.getProperty("abstract-tags", "query,pages,page,extract").trim, | ||
| removeBrokenBracketsProperty = this.getProperty("remove-broken-brackets-plain-abstracts", "false").trim.toBoolean | ||
| ) | ||
| } match{ | ||
| case Success(s) => s | ||
|
|
@@ -293,7 +294,8 @@ class Config(val configPath: String) extends | |
| writeAnchor = this.getProperty("nif-write-anchor", "false").trim.toBoolean, | ||
| writeLinkAnchor = this.getProperty("nif-write-link-anchor", "true").trim.toBoolean, | ||
| abstractsOnly = this.getProperty("nif-extract-abstract-only", "true").trim.toBoolean, | ||
| cssSelectorMap = this.getClass.getClassLoader.getResource("nifextractionconfig.json"), //static config file in core/src/main/resources | ||
| removeBrokenBracketsProperty = this.getProperty("remove-broken-brackets-html-abstracts", "false").trim.toBoolean | ||
| ) | ||
| } match{ | ||
| case Success(s) => s | ||
|
|
@@ -348,7 +350,8 @@ object Config{ | |
| writeAnchor: Boolean, | ||
| writeLinkAnchor: Boolean, | ||
| abstractsOnly: Boolean, | ||
| cssSelectorMap: URL, | ||
| removeBrokenBracketsProperty: Boolean | ||
| ) | ||
|
|
||
| /** | ||
|
|
@@ -369,11 +372,12 @@ object Config{ | |
| ) | ||
|
|
||
| case class AbstractParameters( | ||
| abstractQuery: String, | ||
| shortAbstractsProperty: String, | ||
| longAbstractsProperty: String, | ||
| shortAbstractMinLength: Int, | ||
| abstractTags: String, | ||
| removeBrokenBracketsProperty: Boolean | ||
| ) | ||
|
|
||
| case class SlackCredentials( | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,7 +13,7 @@ import scala.language.reflectiveCalls | |
| * Created: 5/19/14 9:21 AM | ||
| */ | ||
|
|
||
| class HtmlAbstractExtractor( | ||
| context : { | ||
| def ontology : Ontology | ||
| def language : Language | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -54,7 +54,7 @@ extends PageNodeExtractor | |
|
|
||
| private val language = context.language.wikiCode | ||
|
|
||
| private val logger = Logger.getLogger(classOf[PlainAbstractExtractor].getName) | ||
|
|
||
| //private val apiParametersFormat = "uselang="+language+"&format=xml&action=parse&prop=text&title=%s&text=%s" | ||
| private val apiParametersFormat = "uselang="+language+"&format=xml&action=query&prop=extracts&exintro=&explaintext=&titles=%s" | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,7 +14,7 @@ import scala.language.reflectiveCalls | |
| /** | ||
| * Extracts page html. | ||
| * | ||
| * Based on PlainAbstractExtractor, major difference is the parameter | ||
| * apiParametersFormat = "action=parse&prop=text§ion=0&format=xml&page=%s" | ||
| * | ||
| * This class produces all nif related datasets for the abstract as well as the short-, long-abstracts datasets. | ||
|
|
@@ -69,7 +69,7 @@ class NifExtractor( | |
|
|
||
| object NifExtractor{ | ||
| //TODO check if this function is still relevant | ||
| //copied from PlainAbstractExtractor | ||
| def postProcessExtractedHtml(pageTitle: WikiTitle, text: String): String = | ||
| { | ||
| val startsWithLowercase = | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,13 +1,13 @@ | ||||||||||||||||||||||||
| package org.dbpedia.extraction.mappings | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| import java.util.logging.Logger | ||||||||||||||||||||||||
| import org.dbpedia.extraction.annotations.ExtractorAnnotation | ||||||||||||||||||||||||
| import org.dbpedia.extraction.config.Config | ||||||||||||||||||||||||
| import org.dbpedia.extraction.config.provenance.DBpediaDatasets | ||||||||||||||||||||||||
| import org.dbpedia.extraction.ontology.Ontology | ||||||||||||||||||||||||
| import org.dbpedia.extraction.transform.{Quad, QuadBuilder} | ||||||||||||||||||||||||
| import org.dbpedia.extraction.util.abstracts.AbstractUtils | ||||||||||||||||||||||||
| import org.dbpedia.extraction.util.{Language, MediaWikiConnector, WikiUtil} | ||||||||||||||||||||||||
| import org.dbpedia.extraction.wikiparser._ | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| import scala.language.reflectiveCalls | ||||||||||||||||||||||||
|
|
@@ -30,7 +30,7 @@ import scala.language.reflectiveCalls | |||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| @deprecated("replaced by NifExtractor.scala: which will extract the whole page content including the abstract", "2016-10") | ||||||||||||||||||||||||
| @ExtractorAnnotation("abstract extractor") | ||||||||||||||||||||||||
| class PlainAbstractExtractor( | ||||||||||||||||||||||||
| context : { | ||||||||||||||||||||||||
| def ontology : Ontology | ||||||||||||||||||||||||
| def language : Language | ||||||||||||||||||||||||
|
|
@@ -39,7 +39,7 @@ class AbstractExtractor( | |||||||||||||||||||||||
| ) | ||||||||||||||||||||||||
| extends WikiPageExtractor | ||||||||||||||||||||||||
| { | ||||||||||||||||||||||||
| protected val logger = Logger.getLogger(classOf[PlainAbstractExtractor].getName) | ||||||||||||||||||||||||
| this.getClass.getClassLoader.getResource("myproperties.properties") | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
|
@@ -50,6 +50,8 @@ extends WikiPageExtractor | |||||||||||||||||||||||
| //private val apiParametersFormat = "uselang="+language+"&format=xml&action=parse&prop=text&title=%s&text=%s" | ||||||||||||||||||||||||
| protected val apiParametersFormat = context.configFile.abstractParameters.abstractQuery | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| protected val removeBrokenBrackets = context.configFile.abstractParameters.removeBrokenBracketsProperty | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| // lazy so testing does not need ontology | ||||||||||||||||||||||||
| protected lazy val shortProperty = context.ontology.properties(context.configFile.abstractParameters.shortAbstractsProperty) | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
|
@@ -63,7 +65,6 @@ extends WikiPageExtractor | |||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| private val mwConnector = new MediaWikiConnector(context.configFile.mediawikiConnection, context.configFile.abstractParameters.abstractTags.split(",")) | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| override def extract(pageNode : WikiPage, subjectUri: String): Seq[Quad] = | ||||||||||||||||||||||||
| { | ||||||||||||||||||||||||
| //Only extract abstracts for pages from the Main namespace | ||||||||||||||||||||||||
|
|
@@ -79,16 +80,22 @@ extends WikiPageExtractor | |||||||||||||||||||||||
| // if(abstractWikiText == "") return Seq.empty | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| //Retrieve page text | ||||||||||||||||||||||||
| val text = mwConnector.retrievePage(pageNode.title, apiParametersFormat, pageNode.isRetry) match { | ||||||||||||||||||||||||
| case Some(t) => PlainAbstractExtractor.postProcessExtractedHtml(pageNode.title, replacePatterns(t)) | ||||||||||||||||||||||||
| case None => return Seq.empty | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| val modifiedText = if (removeBrokenBrackets) { | ||||||||||||||||||||||||
| AbstractUtils.removeBrokenBracketsInAbstracts(text) | ||||||||||||||||||||||||
| } else { | ||||||||||||||||||||||||
| text | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| //Create a short version of the abstract | ||||||||||||||||||||||||
| val shortText = short(modifiedText) | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| //Create statements | ||||||||||||||||||||||||
| val quadLong = longQuad(pageNode.uri,modifiedText, pageNode.sourceIri) | ||||||||||||||||||||||||
| val quadShort = shortQuad(pageNode.uri, shortText, pageNode.sourceIri) | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| if (shortText.isEmpty) | ||||||||||||||||||||||||
|
|
@@ -140,7 +147,7 @@ extends WikiPageExtractor | |||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| private def replacePatterns(abst: String): String= { | ||||||||||||||||||||||||
| var ret = abst | ||||||||||||||||||||||||
| for ((regex, replacement) <- PlainAbstractExtractor.patternsToRemove) { | ||||||||||||||||||||||||
| val matches = regex.pattern.matcher(ret) | ||||||||||||||||||||||||
| if (matches.find()) { | ||||||||||||||||||||||||
| ret = matches.replaceAll(replacement) | ||||||||||||||||||||||||
|
|
@@ -205,15 +212,15 @@ extends WikiPageExtractor | |||||||||||||||||||||||
| .filter(renderNode) | ||||||||||||||||||||||||
| .map(_.toWikiText) | ||||||||||||||||||||||||
| .mkString("").trim | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| // decode HTML entities - the result is plain text | ||||||||||||||||||||||||
| decodeHtml(text) | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| */ | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| object PlainAbstractExtractor { | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| //TODO check if this function is still relevant | ||||||||||||||||||||||||
| def postProcessExtractedHtml(pageTitle: WikiTitle, text: String): String = | ||||||||||||||||||||||||
|
|
@@ -243,6 +250,7 @@ object AbstractExtractor { | |||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| val patternsToRemove = List( | ||||||||||||||||||||||||
| """<div style=[^/]*/>""".r -> " ", | ||||||||||||||||||||||||
| """</div>""".r -> " ", | ||||||||||||||||||||||||
| """<normalized>.*<\/normalized>""".r -> "" | ||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||
|
Comment on lines
251
to
255
There was a problem hiding this comment. Make
🔧 Proposed fix (non-greedy DOTALL) val patternsToRemove = List(
"""<div style=[^/]*/>""".r -> " ",
"""</div>""".r -> " ",
- """<normalized>.*<\/normalized>""".r -> ""
+ """(?s)<normalized>.*?</normalized>""".r -> ""
)📝 Committable suggestion
Suggested change
🤖 Prompt for AI Agents
Comment on lines
251
to
255
There was a problem hiding this comment. Use non-greedy matching for The regex 🐛 Proposed fix val patternsToRemove = List(
"""<div style=[^/]*/>""".r -> " ",
"""</div>""".r -> " ",
- """<normalized>.*<\/normalized>""".r -> ""
+ """<normalized>.*?<\/normalized>""".r -> ""
)🤖 Prompt for AI Agents |
||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolve markdownlint issues: bare URLs + missing code‑fence languages.
Line 11 and Line 19 include bare URLs, and the fenced blocks at Lines 27/31/35 are missing language identifiers. There’s also a blank line inside a blockquote (Line 25). These are easy markdownlint failures to fix.
✅ Suggested cleanup
🧰 Tools
🪛 markdownlint-cli2 (0.18.1)
11-11: Bare URL used
(MD034, no-bare-urls)
11-11: Bare URL used
(MD034, no-bare-urls)
11-11: Bare URL used
(MD034, no-bare-urls)
19-19: Bare URL used
(MD034, no-bare-urls)
19-19: Bare URL used
(MD034, no-bare-urls)
25-25: Blank line inside blockquote
(MD028, no-blanks-blockquote)
27-27: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
31-31: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
35-35: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents