app-directory prototype

[keighrim]

Future todo list;

• handling of "third-party" app submission  #32
• name conflicts in app directory entries #33
• document directory behaviors #34
• add proper jekyll config/theme  #42
• app entry design (https://github.com/clamsproject/apps/blob/bac068feb283ede4563eb2e8b3e0da5fc86ed473/docs/spacy-wrapper/v1/index.md)

---

(original comment)
Moving the below comment from clamsproject/wfe-galaxy#2, as it seems more suitable in this repo.

---

@marcverhagen commented on Oct 8
A slightly different, but overlapping, issue is what needs to be in the application's metadata and/or the toolshed to make this work. Most of the things that go into tool.xml qualify (but maybe not input and output name). And somewhere the script needs to know where the repositories are. I was first thinking that could be part of the metadata, but I know think that maybe it should be in the toolshed and that the process that creates the toolshed adds the repository URL.

Not sure whether anybody gave some thoughts as to what the toolshed would look like, so I will just throw out some loose thoughts for discussion:

• The toolshed, code and data, is stored in a Github repository and published from there.
• The published toolshed has a URL like https://apps.clams.ai.
• There is an entry for each app (for example https://apps.clams.ai/kaldi).
• There is a sub entry for each version (for example https://apps.clams.ai/kaldi/0.2.4).
• Either the app URL or the version URL is used as an identifier in the app metadata. If it is the former then we also need a version number in the app metadata.
• A toolshed app entry is built automatically from an app repository.
• For each app version the toolshed contains:
  • a link to the GitHub repository for the app and a particular release
  • the Dockerfile
  • a metadata file in JSON or YAML format
  • Docker file and metadata file each have their own URL.
• In the metadata of a MMIF view, our applications refer to the toolshed entry. Third party tools may use some other URI.

It is a bit weird to call this a toolshed because what we put in the shed are app, but appshed sounds like ape shit.

[keighrim]

I'm pushing for the term app directory instead of toolsheld, as you mentioned that what we are putting inside are apps.

On top of the points @marcverhagen mentioned in the above comment, I'm adding some more of my thoughts;

• In addition to app id and app version, a vendor id also be a good component to have to reduce name conflicts, like https://apps.clams.ai/edu.brandeis.llc/kaldi/x.y.z vs https://apps.clams.ai/hisptas/kaldi/a.b.c
  • We need some rules on how to name.
• App developers use github pull requests for publication, we write some automation scripts (using github actions or setting up our own jenkins) to fully or mostly fully automate the publication process.
  • We use github template for pull request submission to specify required information (vendor id, app id, app version, app repo address, etc), developers must submit a PR with all required information.
  • The PR body is then parsed by one of our scripts to pull out those information. Once the parsing is done, following automation pipeline starts;
• build automations
  • Automatically check name conflicts. If any of the id or version conflicts, automatically reject the PR with some meaningful comment.
  • Automatically check required files (I'm thinking Dockerfile, but maybe some more) in the app repo, and reject if they don't exist.
  • Automatically build the app image from Dockerfilein the repo, start a container and query for app metadata, upload the image to the docker hub or the github container registry. Reject the PR if the app metadata is invalid, or docker build fails.
• publication automations
  • Automatically generate a HTML version of the app metadata in docs directory (e.g. docs/edu.brandeis.llc/kaldi/x.y.z/index.html) and add a commit to the PR.
    • We can also automatically update the app list somewhere under the docs (e.g. docs/apps.html, docs/vendors.html, and/or docs/index.html) .
  • At this point, we might want to manually review the PR before merging.
  • Once merged to the main branch, github will publish the HTML version of the app metadata, under the exact IRI of the full app identifier.

[marcverhagen]

There may be a need for namespaces in the directory, but I am not yet convinced. It is probably not right to use the term vendor by the way. Are we really the vendor of https://apps.clams.ai/edu.brandeis.llc/kaldi/x.y.z and is hipstas really the vendor of https://apps.clams.ai/hisptas/kaldi/a.b.c? We have a distinction between tools and apps. The tools are from vendors and they should be mentioned in the app metadata. We do the apps and if somebody else does an app they can get their own app directory. If all that is true then we do not need namespaces for vendor reasons. However, we could choose to use subdirectories video, audio, images and text or any other logical division.

For the build and publication automation, I would suggest to do a few by hand first to see what automation steps make sense. What there is above in the bullet points makes sense in general, but I will for now admit that I am not a great fan of github actions after my first failed attempt to understand them.

Some remarks on the bullets:

• Yes, there should at the least be one manual review step.
• We should think about whether we want a separation between app building, appliance building and publication. You could imagine the first two being complete independent from the third.
• The prose above seems to be aimed at app repositories. Somewhere there needs to be a step to export information to the app directory repository. We could make that a pull from the app directory and then it would check wether the app name is allowed. (maybe this is exactly what was proposed though).
• In any case, this again makes it a good thing to make access to metadata easier than starting a container.

[angus-lherrou]

Perhaps something like author or maintainer would make sense instead of vendor at the app level

[keighrim]

Automatically generate a HTML version of the app metadata in docs directory (e.g. docs/edu.brandeis.llc/kaldi/x.y.z/index.html) and add a commit to the PR.

This can't be done if the PR sent from a third party fork, where clams-bot do not have push permit.

[keighrim]

(With all the recent hard work on the app metadata implementation, app template implementation, and some help from ghcr and gha)

---

Introducing the first prototype app directory, made available by

• 40ab9b0
• app directory integration clams-python#137 (will be release as a part of next clams-python release)

And the first app registration is submitted at #31 via https://github.com/clamsproject/app-spacy-wrapper/actions/runs/4973069451.
(spacy-wrapper got the update a bit earlier as my testbed)

---

Now looking back at the initial checklist from @marcverhagen ;

• The toolshed, code and data, is stored in a Github repository and published from there.
• The published toolshed has a URL like https://apps.clams.ai.
  • keighrim: having trouble accessing our DNS company at the moment, till I configure the subdomiain rule, the directory is at https://clams.ai/apps/
• There is an entry for each app (for example https://apps.clams.ai/kaldi).
• There is a sub entry for each version (for example https://apps.clams.ai/kaldi/0.2.4).
• Either the app URL or the version URL is used as an identifier in the app metadata.
  • keighrim: we use the version URL
• If it is the former then we also need a version number in the app metadata.
• A toolshed app entry is built automatically from an app repository.
  • keighrim: fully automated by a GHA workflow, but needs two human reviews before merging into the public website
• For each app version the toolshed contains:
  • a link to the GitHub repository for the app and a particular release
  • the Dockerfile keighrim: even better, we have pre-built images in ghcr, also fully automated by the GHA workflow
  • a metadata file in JSON or YAML format
  • Docker file prebuilt image and metadata file each have their own URL.
    • keighrim: https://..../appname/version/metadata.json for metadata ghcr.io/clamsproject/appname:version for images
• In the metadata of a MMIF view, our applications refer to the toolshed entry. Third party tools may use some other URI. (representation of the producer in view metadata mmif#153)

and mine;

• In addition to app id and app version, a vendor id also be a good component to have to reduce name conflicts, like https://apps.clams.ai/edu.brandeis.llc/kaldi/x.y.z vs https://apps.clams.ai/hisptas/kaldi/a.b.c
  • We need some rules on how to name.
    • keighrim: we no longer uses vendors in the app identifier scheme. However, current automation only works with first-party apps (due to github authentication used in automation workflows). We need to think about how to handle thrid-party app submissions. But my current thought is that we keep https://apps.clams.ai/ base url as first-party only. Thrid party apps should have their own app identifier with their own base url (doesn't matter a web page is behind the identifier or not).
• App developers use github pull requests for publication, we write some automation scripts (using github actions or setting up our own jenkins) to fully or mostly fully automate the publication process.
  • We use github template for pull request submission to specify required information (vendor id, app id, app version, app repo address, etc), developers must submit a PR with all required information.
    • keighrim: I used a GHA workflow for building webpages. Currently there are three ways to submit an app (or technically, call the GHA workflow
      1. call from other workflow (this is built-in in the clams develop app template (app directory integration clams-python#137))
      2. submit an issue in this repository with NAME, TAG, CONTAINER variables (using an issue template might be a good way to do this)
      3. manually invoke the workflow at https://github.com/clamsproject/apps/actions/workflows/register.yml -> Run workflow button.
  • The PR body is then parsed by one of our scripts to pull out those information. Once the parsing is done, following automation pipeline starts;
    • keighrim: https://github.com/clamsproject/apps/blob/68b1811cf56262007245c13d52cfb8cc2034e92e/scripts/parse_issue.py
• build automations
  • Automatically check name conflicts. If any of the id or version conflicts, automatically reject the PR with some meaningful comment.
    • keighrim: current implementation doesn't reject name conflicts. There are two cases
      1. If the identical (in terms of appmetadata) app submitted twice, the topic branch isn't created (because there's "nothing to commit") and in turn, PR does not get created (the workflow fails to finish).
      2. If a different (again, in terms of appmetadata) app is submitted with the same repo+tag, the new version will be updating existing html and json files, but we have a chance to manually review them before published.
  • Automatically check required files (I'm thinking Dockerfile, but maybe some more) in the app repo, and reject if they don't exist.
    • keighrim: delegated to the app develop template
  • Automatically build the app image from Dockerfile in the repo, start a container and query for app metadata, upload the image to the docker hub or the github container registry. Reject the PR if the app metadata is invalid, or docker build fails.
• publication automations
  • Automatically generate a HTML version of the app metadata in docs directory (e.g. docs/edu.brandeis.llc/kaldi/x.y.z/index.html) and add a commit to the PR.
    • keighrim: https://github.com/clamsproject/apps/blob/68b1811cf56262007245c13d52cfb8cc2034e92e/scripts/metadata_markdown.py
    • We can also automatically update the app list somewhere under the docs (e.g. docs/apps.html, docs/vendors.html, and/or docs/index.html) .
      • keighrim: https://raw.githubusercontent.com/clamsproject/apps/68b1811cf56262007245c13d52cfb8cc2034e92e/docs/index.md + https://github.com/clamsproject/apps/blob/68b1811cf56262007245c13d52cfb8cc2034e92e/scripts/index_page_data.py + https://github.com/clamsproject/apps/blob/68b1811cf56262007245c13d52cfb8cc2034e92e/docs/apps.json.html
  • At this point, we might want to manually review the PR before merging.
  • Once merged to the main branch, github will publish the HTML version of the app metadata, under the exact IRI of the full app identifier.