Platforms and Ongoing Maintenance (permanent issue for reference)

[MarkGotham]

We have written elsewhere about the workflows for creating and checking scores.

We now need to clarify a new, later stage that might be called “maintenance” (cf #4 ).

Here’s the current plan, based on various recent discussions:

• musescore.com:
  • The scores remain as they are; not replaced or updated unless we spot an actual error in which case replaced manually.
  • This means that they will be created in multiple versions of musescore (v2–). The site supports that and we have been satisfied with how the scores are rendered at the point of initial upload. Changing versions may break this and create considerable maintenance work with no obvious benefit.
• musescore.com -> GitHub.
  • More regular updates needed.
  • Pull new scores only if newly added there, or if (rarely) changed there more recently than here (shouldn't usually happen)
• GitHub:
  • Run batch conversion to recent musescore versions as they stabilise (e.g., 4.2 did not work well enough, 4.5.2 did). Condition met when approximately <1% score fail rate. This ensures the scores are maintained. For example, some scores created in v2 cannot be opened at all in v4 but only via v3. Iteratively converting these scores via intermediate versions is necessary for ongoing maintenance and usability of those scores.
  • On running conversion script across the whole corpus, fix known errors of cases which failed to convert or to which we are alerted.
  • Manual improvements to any scores welcome at any time (timestamped on commit to this repo).
  • Very limited risk of conflict between musescore and GitHub (very few people have access to either and all of them should read this!).
• GitHub -> Zenodo:
  • Batch conversions of the whole repo (to MS4, 5, ...) should be reflected in new release, numbered to match which in turn triggers a new Zenodo record and DOI.
  • Provisionally, plan to run this conversion/release exactly once per MS version. Adjust as needed.
• Public-facing websites (like this one):
  • keep the musescore.com embed as long as that works well (currently True).
  • remove the MS3 option as legacy only no longer maintained. If we find users asking for this, then create an MS3 branch from a relevant point in time and provide links to the scores there (with a caveat of not maintained, may not be latest version)
  • Add to the links (cf Metadata and database #3 )

In future:

• Yes:
  • MEI export from MS >= v4.
  • Apply the same to other OpenScore corpora
• Maybe:
  • Consider applying relevant parts of this to other open corpora beyond OpenScore.
  • Consider an OSF mirror or equivalent to record download numbers of individual files.

[MarkGotham]

@apacha @Budimba @lpugin @magiraud @shoogle @rettinghaus

[Budimba]

All scores I manually checked were in MS3, despite having a .mscz extension. Also, the layouts of some scores were just not acceptable (e.g., title crashing into dedication)
Many hacks were used (e.g., spacing bar), which don't transfer to MusicXML.

[MarkGotham]

All scores I manually checked were in MS3, despite having a .mscz extension. Also, the layouts of some scores were just not acceptable (e.g., title crashing into dedication) Many hacks were used (e.g., spacing bar), which don't transfer to MusicXML.

Thanks for this. Sounds like you're content with the broad plan above and we're moving to details.

So to those details then:

1. version issue = not previously known. How odd. This is despite having been converted using MS4.5.2. You can see the script used at convert_all.py. Perhaps you have a suggestion for how to modify this?
2. layout/crashing: clearly should be fixed. Might restore default layouts be better in general? Or perhaps we need a new style sheet per version (not that we've actually in MS4 yet)? It would be good to make layout robust across MS version changes but automatic fixes per version may be more practical.
3. hacks: "not transferring" is not a problem in itself; leaving an issue unresolved is, which brings us back to 2.

Thanks for all/any suggestions wrt how to proceed.

[rettinghaus]

I'm wondering why use compressed MusicXML (.mxl) and not uncompressed (.musicxml)?

[Budimba]

2. Might restore default layouts be better in general? Or perhaps we need a new style sheet per version (not that we've actually in MS4 yet)? It would be good to make layout robust across MS version changes but automatic fixes per version may be more practical.

The fewer gimmicks used for layout, the more robust across version changes it would be. I suggest rebuilding the headings programmatically and replacing one vertical frame with as many individual text frames as possible:

The first three are text frames with preset gaps; the last is a vertical frame with two texts.

[Budimba]

I'm wondering why use compressed MusicXML (.mxl) and not uncompressed (.musicxml)?

An uncompressed file can be several times larger than a compressed one. For the record, currently the repo is ~800 MB, already exceeding the quota (512 MB) of the place where the database is currently hosted and where automated GitHub syncing is planned (PythonAnywhere). To sync to GitHub, the repo must be cloned.
.mscz is also just .mscx together with some other unimportant files, ZIP-compressed.

[Budimba]

All scores I manually checked were in MS3, despite having a .mscz extension

Sorry, it turns out there were only two such files (the two I checked). Now all .mscz files are in MS4.

1. Perhaps you have a suggestion for how to modify this?

I think Python is overkill for something that can be done with a shell command:
find . -name '*.mscx' -exec sh -c 'QT_QPA_PLATFORM=offscreen <location_of_musescore_binary> -o "${0%mscx}mscz" "$0"' {} \;
If the extension for MS5 is still .mscz, even easier:
find . -name '*.mscz' -exec QT_QPA_PLATFORM=offscreen <location_of_musescore_binary> -o {} {} \;
I haven't looked into Zsh, but from what I gathered find won't be needed.
Also, if you run MuseScore headlessly like this, I believe the issue 'The program didn't exit cleanly' won't appear (otherwise it's certainly a bug).

[shoogle]

currently the repo is ~800 MB, already exceeding the quota (512 MB) of the place where the database is currently hosted and where automated GitHub syncing is planned (PythonAnywhere)

This gives you the repo's full commit history without storing any score files on disk:

git clone --no-checkout https://github.com/OpenScore/Lieder.git
cd Lieder
git reset .
ls -lA        # Shows you have the hidden .git folder and nothing else.

The commit history is inside the hidden .git folder. It's compressed by Git, so it only takes up 170 MB on my machine, and you can use it to restore files from any point in the history.

You can create a web server that browses the commit history like a filesystem, serving up files without ever saving them to disk.

# List scores by Amy Beach:
git ls-files -- 'scores/Beach,_Amy/'        # Any format
git ls-files -- 'scores/Beach,_Amy/*.mscx'  # Only MSCX

# Get full path to a score from its ID:
git ls-files -- 'scores/*/lc6202745.mscx'

# List commits that modified this score:
git log -- 'scores/*/lc6202745.mscx'

# Show how it was modified by a particular commit (i.e. show the diff):
git show e1a544a192 -- 'scores/*/lc6202745.mscx'

# CRUCIAL
# Pass the contents of a score file to another program (e.g. a web server) without saving it to disk:
# NOTE: Replace 'less' with the executable of your program / web server.
git show "HEAD:$(git ls-files 'scores/*/lc6202745.mscx')" | less        # Current version
git show "111c0d6f1e:$(git ls-files 'scores/*/lc6202745.mscx')" | less  # Historic version

# Fetch new commits without saving score files to disk:
git fetch origin main
git reset origin/main  # In this case avoid using --hard or --soft

You only need to save a score to disk if you want to:

1. Edit the file, or;
2. View it in a program that doesn't support reading from stdin.

When that's the case, you only need to save that one file and not any of the others.

# Save score file to disk (so you can edit it):
git checkout -- 'scores/*/lc6202745.mscx'             # Current version
git checkout 111c0d6f1e -- 'scores/*/lc6202745.mscx'  # Historic version

# Edit the file, then:
git add  -- 'scores/*/lc6202745.mscx'
git commit -m 'Your message'
rm -- "$(git ls-files 'scores/*/lc6202745.mscx')"  # Delete file so it's no longer on disk.

See also git clone --sparse and git sparse-checkout.

[Budimba]

This gives you the repo's full commit history without storing any score files on disk:

I don't mind the disk usage on my personal computer, but as @MarkGotham intends the GitHub repo to be the central record, I want to push commits to the repo from a webserver, so approvers can update the repo without installing git and cloning the repo. (The webserver can ask for the (fine-grained) token every time (to be generated), not storing the token; if the service can be hosted on the (trusted) fourscore server, then the token can be stored there.) Can I make commits and push without storing the 800 MB? This may include:

• git pull without downloading scores
• Untrack an uploaded score so deleting it won't make git think it needs to be deleted remotely too

This automated remote git business feels clunky and insecure. I may prefer the approver left the scores in the Proton Drive folder and I'll upload them to GitHub locally every week.

[MarkGotham]

Wrt compressed/uncompressed (@rettinghaus , @Budimba ...),

First, the move from MS3 -> MS4 strongly nudges us towards compressed. Uncompressed is “experimental” for MS4.

Second, can any of us say that we have used uncompressed files to keep track of versioning details? It’s a possible use case for sure, but not one we’ve actually used, I think. It might would be different if we were talking about a light-weight format like krn.

We may, in future:

• Batch export to MEI
• Use MEI for meaningful version control along with wider enhancements to the sophistication of source-encoding comparison (thank you in advance, @mei-friend @wergo @musicog !)

Unless I’m missing something in the above, I find @Budimba 's argument about repo size / use cases convincing and am inclined towards compressed mscz and mxl, while considering a future with MEI.

[MarkGotham]

I suggest rebuilding the headings programmatically and replacing one vertical frame with as many individual text frames as possible:

Sounds good @Budimba with the caveat that we've preserved score page layout and this is easily broken in such cases.

The programmatic solution to that would be to reduce stave size until it fits again. Another can of worms ;)

[Budimba]

we've preserved score page layout and this is easily broken in such cases

How about instead:

• Batch-convert all files to PDF headlessly
• Programmatically extract the 1st pages and stitch them into a 1400-page PDF; also make a list of musescore_id's
• Scroll through every page, mark the page number of problematic pages
• Translate those page numbers to their musescore_id's
• Manually correct them

Or, more easily:

• Batch-convert old files to PDF; programmatically record numbers of pages
• Programmatically swap headings
• Batch-convert new files to PDF; programmatically record numbers of pages
• See which files now have more pages; correct them manually. If still those many pages, assume the layouts are okay

By the way, I looked at a random score and the layout was already broken (widened measures added a new line and thus a new page). Its .mscx was fine, and my manually converted .mscz was also fine. I don't know how that happened. In the future do we also lock the number of measures each line?
I then looked at a few more random scores and found that this isn't a common phenomenon.

In the future, 'Scale with staff size' of all heading elements should be unchecked, so the amount of space the heading takes remains consistent, no matter how dense the music is.

[MarkGotham]

Yes this layout across versions business is a big topic. Scores created by many people across many MS versions mean these issues can accumulate. One understandable case is that we don't have all stave and page breaks marks: some are currently some missing when they "just worked" in the MS version used at the time and the transcriber, reviewer, and checker were all satisfied with the layout. Looking at the scores online shows how it looked to that team. Including all intended stave and page brakes in the encoding should have been policy from the start. That was an oversight.

I agree that we should now solve this by:

• Specifying all intended stave and page breaks (even when they "just work")
• ... then lock format.
• Have some style-based programmatic solution to identify issues (yes, perhaps by recording # bars per stave etc.) and fix as many as possible programmatically
• Manually fix those that remain (if it's a practical number, not 1300+)

It's highly likely that this will raise other layout issues accruing over time and MS versions.

This all should probably come first, then after that, once style is broadly "fixed"

• Version release on Zenodo with all scores working well in MS4.
• Run the batch MuseScore -> MEI and check how effective that is for this repertoire.
• Use the downstream MEI for version control.
• Consider adding the MEI version (using verovio) for an additional on-page embedding of the public-facing sites.

[shoogle]

Can I make commits and push without storing the 800 MB?

That's exactly what git sparse-checkout is designed to do. It's similar to clone --no-checkout, except:

• git status won't show the "missing" local files as deleted.
• You can do git pull origin main instead of git fetch origin main && git reset origin/main.

This should probably be the recommended way to interact with the repository:

git clone --sparse https://github.com/OpenScore/Lieder.git
cd Lieder
ls -lA  # Shows you have the hidden .git folder and top-level files (README, LICENSE, .gitignore, etc.) only.
git sparse-checkout list  # Empty at the moment (no directories have been added).

Most of the time you'll use it like the --no-checkout method, using stdin instead of saving files to disk:

# Extract a score file from the Git history and pass it to another program without saving it to disk:
git show "HEAD:$(git ls-files 'scores/*/lc6202745.mscx')" | less        # Current version
git show "111c0d6f1e:$(git ls-files 'scores/*/lc6202745.mscx')" | less  # Historic version

When you do need to save a score to disk (e.g. for editing or viewing in a program that can't read from stdin):

# Get path to the score in question:
git ls-files -- 'scores/*/lc6202745.mscx'
# Output: scores/White,_Maude_Valérie/_/To_Mary/lc6202745.mscx

# Add its directory to the sparse checkout:
git sparse-checkout add scores/White,_Maude_Valérie/_/To_Mary/  # Saves all decendants to disk.

# Edit the score, then:
git add -- 'scores/*/lc6202745.mscx'
git commit -m "Your message"
git push origin main
git sparse-checkout set ''  # Deletes local files and ignores the deletion so remote files are not deleted.