PSv2: Pipeline registration for async processing services

[carlosgjs]

Summary

In the asynchonous/pull model, antenna doesn't have an endpoint to call on the services to fetch pipelines. Instead, the service workers poll antenna for work to do. This means that the workers need a mechanism (API?) to advertise their supported pipelines.

See: #1076

Related Issues

Depends on #987

Proposed design

In V1, pipelines are associated with a Project via a ProcessingService. They are fetched from the /info endpoint as a ProcessingServiceInfoResponse
In V2, the processing services can POST a ProcessingServiceInfoResponse to an antenna endpoint during startup? E.g. /api/v2/projects/1/pipelines

[carlosgjs]

@mihow Is this a per-project registration? Does this mean a V2 processing service worker runs against a particular project or set of projects?

[mihow]

Good question. Here is my thinking:

• If admins add a Processing Service, that service and it's Pipeline choices are generally available to all projects (unless it's a test or in beta)
• If a user adds a Processing Service, that service and it's Pipelines are only available to that project (or could be all of that user's projects). We don't want someone experimenting or doing something nefarious and suddenly images from other projects are getting sent to that private service.

How does this affect the queue & workers?

If a Processing Service subscribes to a queue for a given pipeline, it should really only be allowed to process images from projects which have that Processing Service enabled.

We should be able to use our permissions system to filter the images that are returned from the queue API. Check the auth token of the P.S., which projects does it have access to? Which images does the user have access to?

If we continue to use a user's token for the P.S. auth, then we should list the "owner" of the processing service in the table view of processing services.

I'm generating more questions as I write. But I think this amounts to just a couple extra queries and the use of our permissions system in the API queue response.

When registering, use the auth token to add the processing service to all of the user's projects by default, unless a list of projects is specified. The P.S. should not be registered to any projects the user does not have access to. There is already a permission for "can register a processing service" that can be used in the view.

Processing service CLI, specify projects

export ANTENNA_API_TOKEN=1234
ami worker register --project 1 --project 2 --name "AMI Data Companion on DRAC gpu-03"
ami worker register [default to all user's projects]

by default, register & start work in one command?

ami worker [register & start pulling for jobs] --pipeline quebec-moths

To start, create API tests in Antenna only
"I am a processing service, register me"
"Give me jobs"
"Here are results to process"
Try some of the options like unique names vs. not while testing to see how they feel in practice.

antenna/ami/ml/tests.py

Lines 55 to 66 in 0673f46

	def _create_processing_service(self, name: str, endpoint_url: str):
	processing_services_create_url = reverse_with_params("api:processingservice-list")
	self.client.force_authenticate(user=self.user)
	processing_service_data = {
	"project": self.project.pk,
	"name": name,
	"endpoint_url": endpoint_url,
	}
	resp = self.client.post(processing_services_create_url, processing_service_data)
	self.client.force_authenticate(user=None)
	self.assertEqual(resp.status_code, 201)
	return resp.json()["instance"]

How do we name the processing service?

• Must specify a name for the processing service when launching the worker
• Must be unique - Antenna returns a validation error
• Pass hostname as a parameter to antenna during registration
• Save auth token's user as the processing service owner/user - dedicated user is created first