Write a script collecting instrument names from MIMO

[cguo88]

As discussed today, this will probably be much faster than going through this big list manually.

I'm happy to attempt this with AI assistance, also welcoming any help from @PouyaMohseni as suggested, or @JoyfulGen who has also been looking at names on MIMO.

This script should:

• Provide list of all the names in different languages available for each instrument entry (everything except "Elements of musical instruments" from hierarchical view), as well as "broader concept" (category) and "exactly matching concepts" (sometimes provides potential HBS number).
• Check if each entry collected from MIMO already exists in UMIL or Wikidata.

What to do with the collected data:

• If a corresponding entry is already on UMIL, we can enter those names directly on UMIL.
• If a corresponding entry is already on Wikidata, but not yet on UMIL, should we add those names to the existing Wikidata pages, or wait for them to be imported to UMIL first?
• Instruments that we don't yet have anywhere can be set aside and recorded in the giant spreadsheet, for once we have implemented the feature of adding new instruments.
• In the process of entering these names, we should also manually check for any mistakes made by the script (e.g. junk such as "assemblage of electronic modules", etc.)

[cguo88]

After noodling around on ChatGPT for some time, I was able to get a good mockup of the desired excel output. However, I'm quite stuck on getting the web scraping feature to actually work, and output anything to an excel file...

[yinanazhou]

Can you share more details on what you stuck on

[cguo88]

I think the problem is that due to the way that data is stored on MIMO, there isn't a simple way to extract it? Currently, ChatGPT seems to recommend using Selenium and ChromeDriver which I've successfully installed, but still stuck in endless debugging in terms of reading the data from MIMO.

Here is the webpage: https://vocabulary.mimo-international.com/InstrumentsKeywords/en/index
Through the "Hierarchy" tab, I'm hoping to simply exclude everything under "Elements of Musical Instruments".

[yinanazhou]

The easiest way would be manually expand all the parents, and use that link instead. This way you don't need to navigate using code.

[cguo88]

I wasn't able to make that work since the expanded parents aren't preserved when saving the web page/html (and don't provide a unique URL); or did you have a different direction in mind?

I was however able to get some excel output from using a JSON-based approach and building the web page URL format, only problem is that it doesn't seem to include all the instrument page entries (appears to max out at 2026 rows/IDs).

[cguo88]

Update - turns out that it was not too difficult to automate the menu expansion, but I was able to figure out the rest of the data extraction by combing some of the approaches above:

1. Use Playwright to open browser and manually navigate to Hierarchy tab
2. Automated menu expansion which skips "Elements of Musical Instruments" node
3. Extract instrument IDs, names, and links from sidebar (UI crawl), save to a local .json file
4. Use API to fetch translations and editorial notes by ID

Last step would be to check against both UMIL and Wikidata, which are still giving me trouble; any pointers or suggestions would be appreciated! In particular, I'm thinking if we already have a list of all instruments currently in UMIL to date, using one might simplify that part of the process.