Extract termwiki term pairs for our apertium languages, prepare for inclusion in bidix (Bugzilla Bug 2256)

[albbas]

This issue was created automatically with bugzilla2github

Bugzilla Bug 2256

Date: 2016-11-09T11:53:43+01:00
From: @snomos (Originally created by @snomos)
To: @albbas
CC: @albbas, @ftyers, @leneantonsen, @mka055, @snomos, @Trondtr

Last updated: 2018-05-28T09:11:50+02:00

[albbas]

Comment 11653

Date: 2016-11-09 11:53:43 +0100
From: @snomos

We want to leverage the work done in the termwiki for our MT systems, and one way of doing that is to automatically extract all term pairs for the language pairs we have MT systems for:

sme-nob
sme-sma
sme-smj
sme-smn
sme-fin

(please correct me if I missed something)

For both target and source language it is imporant that the term is analysable and generatable, so this must be checked before the candidate pair is constructed. Also, it must be checked that the pair does not exist in the bidix already.

Given this, new pairs should be constructed in bidix format, and added to the apertium svn for the relevant pair, in a special bidic candidate file (there is no such file ATM, so just create one with a descriptive name).

[albbas]

Comment 11654

Date: 2016-11-09 13:07:18 +0100
From: @leneantonsen

Børre has made this script:

New Revision: 133146

Added:
trunk/gt/script/termwiki-expressions.py
Log:
Print se and fi entries from the daily termwiki dump to stdout

The se and fi output are separated by a tab.

If there are several expressions in each language, they are separated by
a comma.

[albbas]

Comment 11968

Date: 2017-02-10 11:00:29 +0100
From: @snomos

Moving satni.org bugs from the risten.no product to the satni.org product.

[albbas]

Comment 12284

Date: 2017-04-04 12:03:33 +0200
From: @tomipieski

*** This bug has been marked as a duplicate of bug #2342 ***

[albbas]

Comment 12296

Date: 2017-04-05 09:14:19 +0200
From: @snomos

This is not a duplicate. The crucial part is:

prepare for inclusion in bidix

That is, you need to format the term pairs as a bidix entry fragment, ie xml. The xml fragment should be sent somewhere, presently I don't know.

I'll add Fran to the Cc list. Maybe he or Lene or Trond have ideas for where to put the bidix fragment.

[albbas]

Comment 12313

Date: 2017-04-06 14:33:56 +0200
From: @tomipieski

Ok. What is bidix? Can you explain?

[albbas]

Comment 12314

Date: 2017-04-06 15:12:53 +0200
From: @snomos

http://wiki.apertium.org/wiki/Northern_Sámi_and_Norwegian/bidix

and

http://wiki.apertium.org/wiki/Bilingual_dictionary

[albbas]

Comment 12345

Date: 2017-04-18 09:19:36 +0200
From: @tomipieski

Termwiki term pairs are prepared in GTHOME/langtech/tools/TermWikiExporter/bidix/sme-nob.dix

[albbas]

Comment 12366

Date: 2017-04-25 10:32:17 +0200
From: @snomos

(In reply to Tomi Pieski from comment #7)

Termwiki term pairs are prepared in
GTHOME/langtech/tools/TermWikiExporter/bidix/sme-nob.dix

$ ll $GTHOME/tools/TermWikiExporter/bidix/
total 6360
-rw-r--r--@ 1 smo036 1907360568 3,1M 18 apr 16:42 sme-nob.dix

It looks reasonable, but there are a couple of questions:

1. you should add the required root element and xmllint the result to verify that the structure is ok
2. have you ensured that all entries in the bidix are recognised by the analyser?

Also, in the initial bug report I requested bidix data for all language pairs for which we have MT systems, and you have so far only added one. I will reopen the bug report. Leave it open until we all agree that we have reached the goal as stated in the initial bug report.

[albbas]

Comment 12368

Date: 2017-04-25 13:30:37 +0200
From: @tomipieski

(In reply to Sjur Nørstebø Moshagen from comment #8)

(In reply to Tomi Pieski from comment #7)

Termwiki term pairs are prepared in
GTHOME/langtech/tools/TermWikiExporter/bidix/sme-nob.dix

$ ll $GTHOME/tools/TermWikiExporter/bidix/
total 6360
-rw-r--r--@ 1 smo036 1907360568 3,1M 18 apr 16:42 sme-nob.dix

It looks reasonable, but there are a couple of questions:

1. you should add the required root element and xmllint the result to verify
   that the structure is ok

Done.

2. have you ensured that all entries in the bidix are recognised by the
   analyser?

Only analysed entries are added.

Also, in the initial bug report I requested bidix data for all language
pairs for which we have MT systems, and you have so far only added one. I
will reopen the bug report. Leave it open until we all agree that we have
reached the goal as stated in the initial bug report.

Leaving this just here.

[albbas]

Comment 12369

Date: 2017-04-25 13:52:34 +0200
From: @snomos

(In reply to Tomi Pieski from comment #9)

1. you should add the required root element and xmllint the result to verify
   that the structure is ok

Done.

Not completely. The correct root element is , and the linting command should be something like:

xmllint --dtdvalid /usr/local/share/apertium/dix.dtd --noout $FILE1 && exit 0;

That is, follow the dtd.

[albbas]

Comment 12370

Date: 2017-04-25 14:02:33 +0200
From: @tomipieski

(In reply to Sjur Nørstebø Moshagen from comment #10)

(In reply to Tomi Pieski from comment #9)

1. you should add the required root element and xmllint the result to verify
   that the structure is ok

Done.

Not completely. The correct root element is , and the linting
command should be something like:

xmllint --dtdvalid /usr/local/share/apertium/dix.dtd --noout $FILE1 && exit
0;

That is, follow the dtd.

Dtd is not found in folder defined above.

[albbas]

Comment 12376

Date: 2017-04-26 16:39:49 +0200
From: @albbas

test

[albbas]

Comment 12377

Date: 2017-04-26 16:41:16 +0200
From: @albbas

(In reply to Børre Gaup from comment #12)

test

one more test

[albbas]

Comment 12378

Date: 2017-04-26 16:42:16 +0200
From: @albbas

(In reply to Børre Gaup from comment #12)

test

one more test