Skip to content
This repository was archived by the owner on Jul 16, 2020. It is now read-only.

Cloud API

Jump to bottom
Aaron Graubert edited this page Jan 30, 2019 · 1 revision

Endpoints

This page documents the general flow of each of the cloud functions that form the API

Submit

This endpoint is invoked by Lapdog Clients to submit new jobs

  1. Check that the user's credentials are valid and not expired
  2. Check that the user has registered with this Engine and has a Service Account allocated
  3. Check that the user has read/write permissions to the workspace bucket associated with the job
  4. Check that the bucket is associated with the workspace for this job and that the workspace is associated with this Engine
    • If this is the first time a job is run in a given workspace, query FireCloud to ensure the given bucket is, in fact, associated with the given workspace. If so, sign a data blob indicating this association
    • For subsequent submissions, check that the signature exists and is valid
  5. Check that the user has generated a submission data file in the expected location and that the file is less than 1 Gb
  6. Submit a Pipelines Request to start a Compute Instance to run the Cromwell server for this job
  7. Sign a data blob which associates the submission ID with the operation ID returned by the Pipelines Request
  8. Return the operation ID to the Client

Abort

This endpoint is invoked by Lapdog Clients to abort running submissions

  1. Check that the user's credentials are valid and not expired
  2. Check that the user has read/write permissions to the workspace bucket associated with the job
  3. Check that the user has generated a submission data file in the expected location and that the file is less than 1 Gb
  4. Check that the operation ID in the submission data file matches the signature generated when the job was submitted
  5. Sign a data blob indicating that the Cromwell Server should start a graceful shutdown, and send the signature to the server
    • After receiving the graceful shutdown signal, Cromwell will verify the signature's validity then send abort signals to all running workflows every 2 minutes until no workflows are running
    • No new workflows will be started, and once all running workflows have stopped, Cromwell will shut down
  6. If the user requested a "hard" (immediate, non-graceful shutdown) abort the operation managing the Cromwell Server
    • This will immediately shut down the instance as soon as Google processes the request. This may allow workflows which are already running to continue running.

Signature (internal)

This endpoint is invoked by Cromwell servers to check the signatures from an abort signal

Register

This endpoint is invoked by Lapdog Clients when registering to an Engine

  1. Check that the user's credentials are valid and not expired
  2. Check that the user provided a workspace and bucket id, and that the user has read/write permissions to that bucket
  3. Check that the bucket is associated with the provided workspace and that the workspace is associated with this Engine
    • If this is the first time the Engine is encountering this workspace, query FireCloud to ensure the given bucket is, in fact, associated with the given workspace. If so, sign a data blob indicating this association
    • For subsequent invocations, check that the signature exists and is valid
  4. Check if the a service account has already been allocated for this user
    • If not, allocate a new service account
  5. Update Service Account Policy Bindings to allow the Engine to act as this service account when running Pipelines
  6. Update Project Policy Bindings to grant appropriate IAM roles to the user and service account
  7. Generate an access key for this service account
    • The key will be discarded after the function completes, for security
  8. Use key to authenticate the FireCloud API as the new service account:
    • Register the service account as a new FireCloud user
    • Add the service account to the user's proxy group
  9. Return the service account email to the Client

Query

This endpoint is used by Lapdog Clients to query their registration status

  1. Check that the user's credentials are valid and not expired
  2. Return whether or not a service account has been issued for this user

Existence (internal)

This endpoint is used internally and returns a fixed, expected response to determine the status of a Lapdog Engine

Quotas

This endpoint is used by Lapdog Clients to fetch the current quota usage of a Lapdog Engine

  1. Check that the user's credentials are valid and not expired
  2. Check that the user has registered with this Engine and has a Service Account allocated
  3. Return the current quota usage of the Engine

Clone this wiki locally