Orthography profile: is i/j i/j always an error ?

[XachaB]

Today, as we were working on the sound comparison study in lexitools, we encountered an error with the form "i/j i/j u ⁴²" (Meixian), resulting in alignments problems. To solve it, you updated the orthography profile, adding ^jiu j iu

Now when playing with the threshold, I encountered another similar case, causing the same error: "i/j i/j ɛ ŋ ²¹". I am guessing that it may be another version of the same problem ? This is from the following entry:

Yangjiang-771_itchy-1,,Yangjiang,771_itchy,jiɛŋ²¹,jiɛŋ²¹,i/j i/j ɛ ŋ ²¹,,Cihui,,,,,癢

which was originally row 29518 of words.tsv. The string jiɛ matches 9 times in that file, two of which are word initial. We find "i/j i/j" only twice in cldf/forms.csv, so it looks like only the initial jiɛ result in problems. I would guess that the solution is to add a rule ^jiɛ j iɛ in the profile, but I could be wrong.

What do you think ?

[LinguList]

Yes! Goid catch.

[XachaB]

Rows where the form contains "w u/w", which come from words starting in "wui" lead to the same error:

Yangjiang-440_fatherinlawwifesfather-1,,Yangjiang,440_fatherinlawwifesfather,wui⁴⁵⁴lou²¹,wui⁴⁵⁴lou²¹,w u/w i ⁴⁵⁴ + l ou ²¹,,Cihui,,,,,外 佬
Yangjiang-466_grandsondaughtersson-1,,Yangjiang,466_grandsondaughtersson,wui⁴⁵⁴ʃaŋ³³,wui⁴⁵⁴ʃaŋ³³,w u/w i ⁴⁵⁴ + ʃ a ŋ ³³,,Cihui,,,,,外 甥
Yangjiang-434_maternalgrandfather-1,,Yangjiang,434_maternalgrandfather,wui⁴⁵⁴kʊŋ³³,wui⁴⁵⁴kʊŋ³³,w u/w i ⁴⁵⁴ + k ʊ ŋ ³³,,Cihui,,,,,外 公
Yangjiang-435_maternalgrandmother-1,,Yangjiang,435_maternalgrandmother,wui⁴⁵⁴pʰo⁴⁴³,wui⁴⁵⁴pʰo⁴⁴³,w u/w i ⁴⁵⁴ + pʰ o ⁴⁴³,,Cihui,,,,,外 婆
Yangjiang-441_motherinlawwifesmother-1,,Yangjiang,441_motherinlawwifesmother,wui⁴⁵⁴mou²¹,wui⁴⁵⁴mou²¹,w u/w i ⁴⁵⁴ + m ou ²¹,,Cihui,,,,,外 母
Yangjiang-463_nephewsistersson-1,,Yangjiang,463_nephewsistersson,wui⁴⁵⁴ʃaŋ³³,wui⁴⁵⁴ʃaŋ³³,w u/w i ⁴⁵⁴ + ʃ a ŋ ³³,,Cihui,,,,,外 甥
Yangjiang-470_outoutside-1,,Yangjiang,470_outoutside,wui⁴⁵⁴tɐi³³,wui⁴⁵⁴tɐi³³,w u/w i ⁴⁵⁴ + t ɐi ³³,,Cihui,,,,,外 低

[LinguList]

In general, these are good catches, but I think we should systematicize them and catch them before making this step-by-step.

First, the Pairwise class in LingPy seems to re-segmentize the data, which is in fact unwanted behavior. This is a bug in the code I wrote for lexitools, and we should treat it as an issue of itself. I know what happens, but not yet where:

In [1]: from lingpy import *                                                                                                                                             

In [2]: ipa2tokens(['t', 't', 'a', 'b'])                                                                                                                                 
Out[2]: ['tt', 'a', 'b']

In [3]: ipa2tokens(' '.join(['t', 't', 'a', 'b']))                                                                                                                       
Out[3]: ['t', 't', 'a', 'b']

In [4]: ipa2tokens(['t', 't', 'a', 'b'], merge_geminates=False)                                                                                                          
Out[4]: ['t', 't', 'a', 'b']

So what happens is that the "merge_geminates" in ipa2tokens keyword is active, although it should NOT be active, since we have the data already segmented. Normally, ipa2tokens should return the pre-segmented string unchanged, if it is passed a string with spaces, and the same would also hold for a list, but the code re-segments the data when it is passed a list. This is generally bad and warrants an issue and a fix on lingpy.

But for lexitools, this can be handled by passing ' '.join(sequence) to Pairwise, as this will force the expected behavior.

[LinguList]

For the Chinese dialect data, we face the problem that it is ambiguous whether i and u are glides or vowels. In a words [ui], it is most likely interpreted as [w i] or [w ui]. We mark this by adding the "slash" notation: [u/w i] or [u/w ui]. However, when we have 'wui' already, our orthoprofile makes a wrong slash notation. So these instances should be identified and then modified in the orothoprofile.

[XachaB]

Noted, thanks. I'll let you figure out what is best for this dataset, and use the fix over in lexitools.
I see it as a good sign that using datasets helps us notice inconsistencies :)