utilities tool

[Jo-stfc]

utilities tool

[Jo-stfc]

utilities/ - folder containing the new scripts
utilities/hostlist.txt - newline separated list of hosts over which the scripts are run
utilities/logsearch.sh - use: logsearch.sh search_string | search unified xrootd and gridftp logs and ss -a for specified string
utilities/serviceop.sh - use: serviceop.sh {start/stop/restart} service_name | run systemctl on the service over all hosts in the hostlist
utilities/aquilon - aquilon utilities
utilities/aquilon/hostlist.txt - another copy of the hostlist to be used in aquilon
utilities/aquilon/compilehosts.sh - use: compilehost.sh | run aq compile on all hosts in the hostlist file
utilities/aquilon/checkhost.sh - use: checkhost search_term | run aq show_host and searches for the term specified. using sandbox as the search term lets you check if it's in a sandbox

[Tom-Byrne]

Lists of hosts are going to go out of date - do we really need these to be up on github?

[Jo-stfc]

these can be updated edited as necessary - i've used reduced subsets for specific uses. We could keep the full list on github or do something else

[Tom-Byrne]

To make this more generic, it may be more appropriate to include instructions on how to generate an up to date list of hosts e.g. aq search_host --personality <personality> > hostlist.txt

[Jo-stfc]

open to suggestions on managing this

[Tom-Byrne]

The usage of aquilon 'metadata' in our ceph clusters management has a few loosely defined principles:

1. The source of truth for what a host is doing is its current personality. Anything else can and will go out of date.
2. All production hosts with a particular personality should be doing the same thing, so any bulk operation should be done (at minimum) to all hosts with the same personality.

Basing tooling around these two things minimizes the amount of places where information can go stale, and reduces the chance of mistakes being made. It also makes our lives easier as we aren't having to understand what "service functionality" a list of hosts is representing, and then having to check that the list isn't out of date before continuing.

For this example, it's not obvious what the hosts on this list are doing now (e.g. are these all Echo gateways, or just a subset?), and it will be even less obvious in 6 months time when some of them are doing different things, and there are new hosts that should be on this list.

If you absolutely need a list of hosts spanning multiple personalities for some part of your workflow, this PR could contain the script to generate a temporary list from a set of personalities. There should be a piece of operational documentation that keeps track of what sets of personalities implement an overarching functionality that may need to be operated on in bulk (e.g. these sets of personalities provide external access to X), so someone can generate an up-to-date list if they need to reuse the tooling.

How does this sound?

[Jo-stfc]

sure, i'll get something that fetches a list based on personality instead

[Tom-Byrne]

Can this functionality be replaced by aq compile --personality?

[Jo-stfc]

I've used this when hosts are across multiple personalities, if they're all the same personality you can use the aq compile --personality command instead