Some taxa on the extinct tree have duplicate IDs

[davidebbo]

In the extinct tree, we use QIDs instead of OTTs as primary ID, since too many OTTs don't exist.

In some cases, we end up with multiple nodes incorrectly having the same QID. For instance, both Тheropoda and Averostra get mapped to Q188438, which is the correct ID for Тheropoda. But there exists a distinct QID for Averostra, Q4828332.

To explain why this happens, let's look at how we get QIDs:

• We start with the Wikipedia (not Wikidata!) page name
• We request its data using the Wikipedia API
• We get the QID from that data
• We also get the taxon's data range from that Wikipedia data (specifically the taxobox)

The challenge is that some Wikipedia pages don't actually exist, which is the situation for Averostra. Instead, going to https://en.wikipedia.org/wiki/Averostra just redirects to https://en.wikipedia.org/wiki/Theropoda. It's not because they are synonyms, but more that some articles end up covering multiple taxa. It's not that uncommon.

We need to better detect this situation and try to somehow map to the correct.

/cc @hyanwong

[hyanwong]

Good triaging, thanks. At a minimum we could check whether there are duplicate QIDs in the tree, and if so, remove any that don't match the wikidata name for that QID. There is a mechanism for referring to internal nodes by name rather than ID, e.g. for vernaculars etc.

[davidebbo]

Yes, that is something we could do.

To add a bit more info, currently I make the following call to get Wikipedia info: https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2&prop=revisions&redirects=1&rvlimit=1&rvprop=content&rvslots=main&titles=Averostra

Note the redirects=1 in there. If you remove that, the same call returns very basic Averostra info with the Wikipedia page ID, but not the QID.

However, I found that if instead of prop=revisions, I use prop=pageprops, I do get the QID. e.g. https://en.wikipedia.org/w/api.php?action=query&format=json&prop=pageprops&titles=Averostra returns:

{
  "batchcomplete": "",
  "query": {
    "pages": {
      "29644404": {
        "pageid": 29644404,
        "ns": 0,
        "title": "Averostra",
        "pageprops": {
          "wikibase-badge-Q70893996": "",
          "wikibase_item": "Q4828332"
        }
      }
    }
  }
}

Note how Q4828332 is the correct Averostra QID. So there is hope here!

On the downside, without a real page behind it, there is no taxobox to get the date range from. But maybe that's a lesser problem.

So I'm thinking we could do something like this:

• For the date range, keep the current logic (with the redirect)
• For the QID, start with a prop=pageprops call without redirect
• If that fails to get a QID, fallback to a prop=pageprops call with redirect

That might cover a lot of the cases where we end up with duplicate IDs today.

[hyanwong]

@davidebbo made a fix, but even with this, I see that e.g. all the Mosasaurus species, which do have their own QIDs, are still given the QID 312131, which is the QID for the genus Mosasaurus.

[davidebbo]

@davidebbo made a fix, but even with this, I see that e.g. all the Mosasaurus species, which do have their own QIDs, are still given the QID 312131, which is the QID for the genus Mosasaurus.

The species don't have individual pages on Wikipedia, and all redirect to the genus. But unlike the case above, the redirection entries does not have a QID. e.g. https://en.wikipedia.org/w/api.php?action=query&format=json&prop=pageprops&titles=Mosasaurus%20missouriensis has no pageprops:

{
  "batchcomplete": "",
  "query": {
    "pages": {
      "33038333": {
        "pageid": 33038333,
        "ns": 0,
        "title": "Mosasaurus missouriensis"
      }
    }
  }
}

So there is no clean link from Wikipedia to QID for the species, causing the above algorithm to fall back.

To address this (without fixing Wikipedia), we'd probably need to derive the mapping by name. e.g. we'd search Wikidata for "Mosasaurus missouriensis", and find that there is a specific page for it: https://www.wikidata.org/wiki/Q135497489