Auto Upgrade

Automatically check for available xyOps and xySat upgrades, generate a job report, optionally send an email notification, and optionally queue background upgrades using the built-in xyOps admin APIs.

This is a xyOps Event Plugin packaged as a pure Node.js / npx marketplace plugin. It is designed to be scheduled nightly or weekly so your cluster stays current without manual version checking.

Features

• Pure Node.js / npx plugin.
• Checks for available upgrades for conductor servers and/or worker servers.
• Generates a Markdown upgrade report in the job output.
• Can send email notifications when upgrades are available.
• Can queue background upgrades for conductors or workers.
• Uses the same documented xyOps APIs that the xyOps UI uses.
• Safe by design: it does not SSH into servers or run undocumented upgrade commands.

Requirements

• npx (this comes pre-installed with xyOps and xySat)
• Network access from the job runner to your xyOps primary conductor
• A valid xyOps API key stored in a Secret Vault as XYOPS_API_KEY
• xyOps must not be running in Air-Gapped Mode

Overview

The plugin can be used in three different ways:

• Report Only: Check for upgrades and generate a job report.
• Report + Email: Check for upgrades and send a notification email.
• Report + Upgrade: Check for upgrades and queue the actual upgrades in the background.

You can also choose whether to check:

• Conductor servers only
• Worker servers only
• Both conductors and workers

When upgrades are found, the plugin writes a Markdown report into the job output containing outdated servers and release notes for the latest xyOps / xySat versions.

Secret Vault / API Key Setup

Create an API key in xyOps, then store it in a Secret Vault:

1. In the xyOps UI, click API Keys from the sidebar and create a new API key.
2. Grant the key the minimum privileges you need:
   • Report only: no special privilege beyond a valid API key.
   • Send emails: add the Send Emails privilege.
   • Run upgrades: add the Administrator privilege.
3. In the xyOps UI, click Secrets from the sidebar and create a new Secret Vault.
4. Assign the Secret Vault to this Plugin, or to the specific Event that will use it.
5. Add this variable to the vault: XYOPS_API_KEY, and paste in your API Key as the value.

Plugin Parameters

These parameters appear in the xyOps Event editor:

Parameter	Description
Check Conductor Versions	Check the xyOps version on all conductor servers.
Check Server Versions	Check the xySat version on all worker servers.
Send Email	Send email containing information about outdated servers.
Perform Upgrades	Queue upgrades to the latest stable release (requires full admin privileges).
Email Addresses	Comma-separated destination list for notification emails.
Stagger Upgrades (sec)	Delay between background upgrade requests, in seconds.

Perform Upgrade Logic

When Perform Upgrades is enabled, the plugin intentionally upgrades only one server type per job run:

• If one or more conductor servers need upgrades, only conductors are upgraded.
• If no conductor upgrades are needed, but one or more worker servers need upgrades, then workers are upgraded.
• If conductors are being upgraded on this run, worker upgrades are deferred until the next scheduled run.

This behavior is intentional for two reasons:

• The upgrade APIs queue background work and return immediately, so the job cannot wait for one upgrade wave to fully complete before starting another.
• It is safer and easier operationally to split conductor and worker upgrades across separate scheduled runs, such as on separate nights.

Scheduling Recommendation

After importing or installing the plugin, create an Event that uses Auto Upgrade, configure the parameters you want, and schedule it to run nightly or weekly.

Nightly is recommended for most installations. Weekly is also reasonable if you prefer a slower maintenance cadence.

A common setup is:

• Check both conductors and workers
• Send email notifications
• Enable background upgrades
• Schedule the event overnight during a low-traffic window

If you use the "Send Email" feature by itself, it is recommended to target the "Auto Upgrade" event at a single server (not a group of servers). The reason is, the plugin writes out a "state" file to the OS temp directory on the server it is running on, to prevent sending duplicate emails for the same software versions.

It should be pointed out that you do not need to target the "Auto Upgrade" event at all the servers you want to upgrade. It doesn't work that way. It actually only needs to run on a single server. It queries xyOps via the API for a list of all conductors and servers in your cluster.

xyOps Upgrade Behavior

It is generally recommended to schedule upgrades during a time window when no jobs are running. However, if you have jobs running constantly 24x7, here is what you can expect:

Conductor Upgrade Behavior

If you have a multi-conductor setup:

• The primary conductor runs upgrades of all peer conductors first, with a "stagger" delay (default 60 seconds).
• After those are complete, it upgrades itself.
• Primary conductor downtime is usually around 5 seconds.
• Jobs are not paused or aborted for any upgrade operations.
  • xySat handles this downtime seamlessly by "holding" all job updates and completions until it reconnects to a primary.
• If the primary upgrade takes longer than 10 seconds, an election is held and a backup conductor takes over.
  • xySat handles this as well, and reconnects to the new primary and resumes job updates and completions.
• If any jobs were scheduled to run during the downtime, behavior depends on the event settings:
  • If the event has the Catch-Up modifier, it will run any missed jobs after the upgrade completes.
  • Otherwise, the job will be skipped until the next scheduled run.

If you have a single-conductor setup:

• Primary conductor downtime is usually around 5 seconds.
• Jobs are not paused or aborted for any upgrade operations.
  • xySat handles this downtime seamlessly by "holding" all job updates and completions until it reconnects.
• If any jobs were scheduled to run during the downtime, behavior depends on the event settings:
  • If the event has the Catch-Up modifier, it will run any missed jobs after the upgrade completes.
  • Otherwise, the job will be skipped until the next scheduled run.

Server Upgrade Behavior

For upgrading xySat on worker servers:

• Servers are upgraded in sequence, with a "stagger" delay (default 60 seconds).
• For each server, the upgrade request is "queued" until all jobs are complete on the server.
• No jobs are aborted for upgrades.
• Only an idle server will execute an upgrade request.
• Downtime for the server is usually around 5 seconds.
• If jobs are scheduled to run on the server during downtime:
  • If the event is targeted at a server group (recommended), xyOps automatically picks a different server to run the job.
  • If the event is targeted directly at the server, or the group only has one server, and Queuing is enabled, the job will be queued until the server comes back online.
  • If the event has Retries configured (recommended), those kick in, and the job will keep trying until the server comes back online, or the retry limit is hit.
  • Otherwise, the job will fail and show as "aborted" with the error message: "No available servers matching targets."

Docker Warning

If your xyOps and/or xySat servers are running in Docker containers, please note that this Plugin will work, but the upgrades are applied to the xyOps / xySat software "only". The Docker image itself is not upgraded, as this is an internal "self-upgrade" running from inside the container.

It is recommended that you upgrade the Docker images themselves, at least semi-regularly, so you get OS-level package updates and patches when those are available. But image upgrades have to be orchestrated from outside, which is out of scope for this plugin.

Safety and API Usage

This plugin is safe to use because it only calls documented xyOps REST APIs, and those are the same upgrade and release-discovery APIs used by the xyOps UI.

The plugin does not:

• Log into servers directly
• Execute shell commands on conductors or workers
• Download or install releases by itself

Instead, it asks xyOps to perform these actions through the official APIs:

• get_servers
• get_master_releases
• get_satellite_releases
• send_email
• admin_upgrade_masters
• admin_upgrade_workers

Plugin Behavior Notes

• If everything is already up to date, the job succeeds and reports that no action was needed.
• If both Check Conductor Versions and Check Server Versions are disabled, the job fails because there is nothing to do.
• If Send Email is enabled, emails are only sent when Email Addresses is populated.
• To reduce duplicate notifications, the plugin remembers the last xyOps/xySat version combination it emailed and skips resending the same combo again.
• The email body and the job report both include release notes for the newest detected xyOps and xySat versions.

Local Testing

The plugin expects a xyOps job payload on STDIN and emits XYWP / report output on STDOUT. For local testing, start with a report-only run against a real xyOps base URL:

export XYOPS_API_KEY="YOUR_API_KEY_HERE"

cat <<'JSON' | node index.js
{
  "xy": 1,
  "type": "event",
  "id": "jtestupgrade001",
  "event": "etestupgrade001",
  "plugin": "ptestupgrade001",
  "server": "local",
  "base_url": "https://xyops.example.com",
  "params": {
    "check_conductors": true,
    "check_servers": true,
    "send_email": false,
    "do_upgrade": false,
    "email_addrs": "",
    "stagger": 60
  }
}
JSON

Notes:

• Replace https://xyops.example.com with your actual xyOps primary conductor URL.
• Start with send_email: false and do_upgrade: false while testing.

Data Collection

This plugin does not collect telemetry, analytics, or usage metrics.

License

MIT.