One location for Pulsar endpoint information

[mira-miracoli]

@mtangaro was asking me where to collect information about the pulsar endpoints.
I think a group_vars/pulsar_endpoints.yml file would be a good possibility to:

• collect all the information in one place
• directly use the yaml file for defining variables that can be re-used in templates like destinations.yml.j2 and job_conf.yml.j2
• implement a semi automated workflow for adding new Pulsar endpoints
• maybe even implement a gh workflow to create a overview on the ESG page

[bgruening]

@mira-miracoli can Marco work on this?

[mira-miracoli]

of course 🙏🏼

[mtangaro]

Hi all :)
So I think we have all needed information to work on this and @gm-ds can work on this.

• we have endpoints info from influxdb (@sanjaysrikakulam am I right?)
• we can collect info from not tracked endpoints
• we are monitoring all endpoints: https://github.com/usegalaxy-it/monitor-usegalaxy-it

So, we should to finally decide which information to display:

• endpoints
• resources
• status
• test jobs
• ...

And where? A page in the galaxy radio telescope or a galaxy-hub pace?

ps. sorry for the delay, we had a lot of debug for our test scripts.

[sanjaysrikakulam]

Hey @mtangaro! What endpoints' info (measurements) are you looking for? If you give me a list of then I can tell you whether they are available already.

[mtangaro]

Hey @mtangaro! What endpoints' info (measurements) are you looking for? If you give me a list of then I can tell you whether they are available already.

Hi Sanjay thank you for the help.
I think we should display very few info about endpoints, just resource availability:

• cpu number
• ram size
• storage size
• gpu availability

[sanjaysrikakulam]

Of each Pulsar endpoint? No, we do not have this static info on InfluxDB. They are available only here

[mtangaro]

So we need to collect them manually and/or keep the destination file up-to-date. Just to understand, is this an information needed by the metascheduler?

[sanjaysrikakulam]

In all scenarios, the TPV destination file must be updated, as Galaxy + TPV cannot send jobs to the respective endpoints without it.

For the metascheduler: Since TPV is the one sending or talking to the TPV Broker, TPV will already have the details about the available destinations at that point. So, TPV will simply forward that data (a Python object) to the TPV Broker to make advanced decisions with the static info + "real-time" info, which it will fetch from the InfluxDB.

[mtangaro]

So static info, like resources, from the destination file and real-time info from influx.

[sanjaysrikakulam]

Yes.

[martenson]

So iiuc we are talking about where to host the static site of https://monitor.usegalaxy.it/ and two of the considered approaches are to put it on the hub (https://galaxyproject.org/) or on the GJR (https://gjr.metacentrum.cz/).

Both of these seem viable to me and I also wanted to point out we can likely set up push automation for the hub (and not issue PRs daily). Thematically it also fits well with GJR and I can easily imagine the interactive map to occupy e.g. only half of the screen estate and the rest be dedicated to the live monitoring.