Support batching up commands

When you have around 60 baremetal nodes attached to a single switch, it
takes a long time to execute all those commands. This gets worse when
you limit the number of concurrent ssh connections.

Here we look to batch up commands to send to the switch together using a
single connection. The results of each port's commands are returned when
available.

This is implemented using etcd as a queueing system. Commands are added
to an input key, then a worker thread processes the available commands
for a particular switch device. We pull off the queue using the version
at which the keys were added, giving a FIFO style queue. The result of
each command set are added to an output key, which the original request
thread is watching. Distributed locks are used to serialise the
processing of commands for each switch device.

Various neat etcd features are used here to alleviate some of the issues
of distributed task coordination, including transactions, leases,
watches, historical key/value tracking, etc.

Co-Authored-By: Mark Goddard <mark@stackhpc.com>

Change-Id: I8c458bbc94df5630cfede5434bcdbe527988059c
(cherry picked from commit 45b237b)

Author: JohnGarbutt

doc/source/configuration.rst

@@ -118,8 +118,9 @@ for the Dell PowerConnect device::
     ngs_switchport_mode = access
 
 Dell PowerConnect devices have been seen to have issues with multiple
-concurrent configuration sessions. See :ref:`synchronization` for details on
-how to limit the number of concurrent active connections to each device.
+concurrent configuration sessions. See :ref:`synchronization` and
+:ref:`batching` for details on how to limit the number of concurrent active
+connections to each device.
 
 for the Brocade FastIron (ICX) device::
 
@@ -201,8 +202,16 @@ connection URL for the backend should be configured as follows::
     [ngs_coordination]
     backend_url = <backend URL>
 
-The default is to limit the number of concurrent active connections to each
-device to one, but the number may be configured per-device as follows::
+The backend URL format includes the Tooz driver as the scheme, with driver
+options passed using query string parameters. For example, to use the
+``etcd3gw`` driver with an API version of ``v3`` and a path to a CA
+certificate::
+
+    [ngs_coordination]
+    backend_url = etcd3+https://etcd.example.com?api_version=v3,ca_cert=/path/to/ca/cert.crt
+
+The default behaviour is to limit the number of concurrent active connections
+to each device to one, but the number may be configured per-device as follows::
 
     [genericswitch:device-hostname]
     ngs_max_connections = <max connections>
@@ -216,6 +225,35 @@ timeout of 60 seconds before failing. This timeout can be configured as follows
     ...
     acquire_timeout = <timeout in seconds>
 
+.. _batching:
+
+Batching
+========
+
+For many network devices there is a significant SSH connection overhead which
+is incurred for each network or port configuration change. In a large scale
+system with many concurrent changes, this overhead adds up quickly. Since the
+Antelope release, the Generic Switch driver includes support to batch up switch
+configuration changes and apply them together using a single SSH connection.
+
+This is implemented using etcd as a queueing system. Commands are added
+to an input key, then a worker thread processes the available commands
+for a particular switch device. We pull off the queue using the version
+at which the keys were added, giving a FIFO style queue. The result of
+each command set are added to an output key, which the original request
+thread is watching. Distributed locks are used to serialise the
+processing of commands for each switch device.
+
+The etcd endpoint is configured using the same ``[ngs_coordination]
+backend_url`` option used in :ref:`synchronization`, with the limitation that
+only ``etcd3gw`` is supported.
+
+Additionally, each device that will use batched configuration should include
+the following option::
+
+    [genericswitch:device-hostname]
+    ngs_batch_requests = True
+
 Disabling Inactive Ports
 ========================