Tutorial
Contents
In the following, we consider the sequencer has been installed as a normal user, not as the root user. See Installation for details.
The sequencer is an engine. It does not do anything by itself. It should therefore be configured properly.
In this first example, we will define a new "command" (ruleset in the sequencer terminology as we will see later) that will perform multiple pings in parallel:
$ sequencer dbcreate $ sequencer dbadd pings cmd ALL ALL 'ping -nq -c 1 %name' NONE NONE 'Parallel Ping' $ sequencer dbshow ruleset | name | types | filter | action | depsfinder | dependson | comments | ------------------------------------------------------------------------------------------------- pings | cmd | ALL | ALL | ping -nq -c 1 %name | NONE | NONE | Parallel Ping | $
The first instruction dbcreate creates the sequencer table. This is where the engine stores its internal stuff. As you may guess, dbshow displays the sequencer table. The second instruction dbadd creates a new ruleset called 'pings'. For the moment, consider a ruleset as a simple command understood by the sequencer. The instruction dbadd requires various parameters:
$ sequencer dbadd Usage: sequencer [options] dbadd ruleset name types filter action depsfinder dependson comments sequencer: error: dbadd: expected 8 arguments, given 0 $
The main important stuff at this stage are the ruleset 'pings' and the action: 'ping -nq -c 1 %name'. Basically, we specify ping to be quiet (-q), to provide numerical output only (-n) and to send only 1 ECHO_REQUEST (see ping(8) for details). The string '%name' will be replaced by the sequencer on execution.
With such a configuration, the sequencer now understand our 'pings' ruleset and list it as a shortcut action:
$ sequencer
Usage: sequencer [global_options] <action> [action_options] <action parameters>
<normal actions>:
chain: Chain 'depmake', 'seqmake' and 'seqexec' actions.
dbadd: Add a rule
dbchecksum: Display ruleset checksums
dbcopy: Copy a rule or a complete ruleset
dbcreate: Create the sequencer table.
dbdrop: Remove the sequencer table
dbremove: Remove a rule or a complete ruleset
dbshow: Show rulesets
dbupdate: Update a rule
depmake: Compute from the given ruleset the dependency graph of the given components.
graphrules: Display the rules graph related to the given ruleset.
knowntypes: Display the types known by the given ruleset.
seqexec: Execute the given instructions sequence.
seqmake: Compute an instructions sequence from the given dependency graph.
<shortcut actions (equivalent to 'chain <shortcut>')>:
pings
sequencer: error: main: wrong number of arguments.
$
So we can use the shortcut 'pings' directly as in:
$ sequencer pings tx[1-3,99,102-105,202] [0-3].pool.ntp.org tx1#exotic@alien/cmd: PING tx1 (67.215.77.132) 56(84) bytes of data. tx1#exotic@alien/cmd: tx1#exotic@alien/cmd: --- tx1 ping statistics --- tx1#exotic@alien/cmd: 1 packets transmitted, 1 received, 0% packet loss, time 0ms tx1#exotic@alien/cmd: rtt min/avg/max/mdev = 51.825/51.825/51.825/0.000 ms tx3#exotic@alien/cmd: PING tx3 (67.215.77.132) 56(84) bytes of data. ... 2.pool.ntp.org#exotic@alien/cmd: ERROR - 2.pool.ntp.org#exotic@alien/cmd: [rc=1] [No output on stderr] 0.pool.ntp.org#exotic@alien/cmd: PING 0.pool.ntp.org (89.188.26.129) 56(84) bytes of data. 0.pool.ntp.org#exotic@alien/cmd: 0.pool.ntp.org#exotic@alien/cmd: --- 0.pool.ntp.org ping statistics --- 0.pool.ntp.org#exotic@alien/cmd: 1 packets transmitted, 0 received, 100% packet loss, time 0ms 0.pool.ntp.org#exotic@alien/cmd: ERROR - 0.pool.ntp.org#exotic@alien/cmd: [rc=1] [No output on stderr] $
The input of a shortcut is a component list: a space separated list of names containing a range expressed between brackets '[]'.
Before understanding the output, we will slightly modify our action: currently, ping is far too verbose for our purpose despite the quiet flag (-q). We just want to know who is alive and who is not. Therefore, instead of doing:
ping -nq -c 1 %name
we will use something a bit more complex like:
/bin/bash -c 'ping -nq -c 1 %name > /dev/null && echo Alive || echo Unreachable'
By the way this will introduce how a ruleset can be modified using dbupdate:
$ sequencer dbupdate Usage: sequencer [options] dbupdate [--nodeps] ruleset_name rule_name <column1>=<value1> <column2>=<value2>... sequencer: error: dbupdate: expected a minimum of 3 arguments, given 0 $
So here, we do:
$ sequencer dbupdate pings cmd action="/bin/bash -c 'ping -nq -c 1 %name > /dev/null && echo Alive || echo Unreachable'"
And since we are only interested (currently) by ruleset and action, specify those columns to dbshow so the display remains readable:
$ sequencer dbshow --columns=ruleset,action ruleset | action | ---------------------------------------------------------------------------------------------- pings | /bin/bash -c 'ping -nq -c 1 %name > /dev/null && echo Alive || echo Unreachable' | $
Back to the execution we now have a much better output:
$ sequencer pings tx[1-3,99,102-105,202] [0-3].pool.ntp.org tx1#exotic@alien/cmd: Alive tx202#exotic@alien/cmd: Alive tx2#exotic@alien/cmd: Alive tx105#exotic@alien/cmd: Alive tx104#exotic@alien/cmd: Alive tx3#exotic@alien/cmd: Alive tx102#exotic@alien/cmd: Alive tx103#exotic@alien/cmd: Alive tx99#exotic@alien/cmd: Alive 0.pool.ntp.org#exotic@alien/cmd: Alive 2.pool.ntp.org#exotic@alien/cmd: Alive 3.pool.ntp.org#exotic@alien/cmd: Alive 1.pool.ntp.org#exotic@alien/cmd: Alive $
With a big list of components, the output may well become unreadable. Using ClusterShell clubak feature we might still have something readable:
$ sequencer pings tx[1-255] [0-255].pool.ntp.org|clubak -c --------------- [0-255].pool.ntp.org#exotic@alien/cmd,tx[1-99,201-255]#exotic@alien/cmd (410) --------------- Alive --------------- tx[100-200]#exotic@alien/cmd (101) --------------- Unreachable $
You may wonder from where does this exotic@alien come from? In a
big set of components, given a simple component name such as node234,
it might not be easy to guess if the given component is a compute
node, a lustre metadata server, a login node, or if it hosts an NFS
server (daemon) or an Apache httpd server for example.
By the way, the sequencer main purpose is to compute an order of execution between actions according to criteria attached to components such as hosts, switch, pdu/talim, disk arrays and so on.
Therefore, for each component given to the sequencer, a type and a category is first guessed by a specific component called the guesser. Those informations are used for defining an order between different actions (more on that later).
Therefore, a component, from the sequencer point of view always has the following format:
name#type@category
Example of names include:
host23#compute@node: a compute node called 'host23' in a clustertx23#login@computer: a login node called 'tx23' in a clusterda34#sfa10k@disk_array: a DDN SFA10K disk array called 'da34' in a clusteresw2-2#eth@switch: an ethernet switch called 'esw2-2'cc3#compute@rack: a rack of compute nodes called 'cc3'nfs1#nfsd@soft: the nfsd daemon that is running on host 'nfs1'www1#backup@vms: the backup virtual machine called 'www1'www#server@group: the group of servers called 'www'.
Of course the meaning of a given type and category is up to you. If you specify the type and the category of a component in the input list, the default guesser will pass them directly to the sequencer. Since for the moment, we do not use neither 'type' nor 'category' in our ruleset, but only the component name, we can specify any type and category and the ruleset will still work:
$ sequencer pings tx100#foo@bar tx100#foo@bar/cmd: Alive $
By default, the sequencer provides a basic guesser that returns type=exotic and category=alien for any component given without a specific type and category.
The guesser is just a convenient way to specify a single name, and to let the framework guess what its type and category are before applying the ruleset. In most cases, you will probably implement your own guesser so it will fetch types and categories from any backend representing your cluster or data center. The backend might be a database, plain file, a process, or whatever you are confortable with (see Advanced Usage for details on how to implement a guesser). For the moment however, we will explicitely specify types and categories where required.
Finally, when an action is executed on a component, it might produce some output (on stdout and/or stderr). As we will see, a ruleset can contain several rules. Each rule defines an action. Therefore, in order to identify which action has produced a given output, the sequencer uses the following format:
name#type@category/rulename: output
In our case, we only have a single rule in our ruleset 'pings'. This rule has been given the name 'cmd'. Hence the output shown above.
We will extend our example, so the shortcut pings will understand
a group of hosts. For that purpose we will introduce the concept of
dependencies which is the heart of the sequencer. we will define
three distincts groups:
-
clients: representing clients hosts; -
servers: representing server hosts; -
world: all (known) hosts
By the way, we will define the group world as made of groups clients
and servers. The objective is to allow the following instructions:
$ sequencer pings clients#fake@group $ sequencer pings servers#fake@group $ sequencer pings world#fake@group
First, we introduce a new rule called 'group' into our ruleset:
$ sequencer dbadd pings group 'fake@group' ALL NONE 'find_deps_for %id' 'group,cmd' 'Grouping IPs'
The output of dbshow is too wide, so we limit the length of the action column:
$ sequencer dbshow --columns=action:8
ruleset | name | types | filter | action | depsfinder | dependson | comments |
---------------------------------------------------------------------------------------------------
pings | cmd | ALL | ALL | /b..{0} | NONE | NONE | Parallel Ping |
pings | group | fake@group | ALL | NONE | find_deps_for %id | cmd,group | Grouping IPs |
Legend:
0: /bin/bash -c 'ping -nq -c 1 %name > /dev/null && echo Alive || echo Unreachable'
$
As shown, the new rule will only be applied on component with the
following form: name#fake@group. It has no action. However, it has a
depsfinder: it is a script that tells for a given component which
other components it depends on. In our case, this script is
find_deps_for and it takes only one parameter: a component
identifier (that is of the form name#type@category). When
called, this script should output on its UNIX stdout one component
identifier per line.
In our example, this script is very simple (and available here):
#!/bin/bash
if test $# -ne 1;then
echo "Usage: $(basename $0) id"
exit 1;
fi
id=$1
if test $id == "servers#fake@group"; then
echo -e "nfs1#server@host\nnfs2#server@host\nwww1#server@host\nwww2#server@host"
elif test $id == "clients#fake@group"; then
echo -e "login1#client@host\nlogin2#client@host\nlogin3#client@host\nlogin4#client@host"
elif test $id == "world#fake@group"; then
echo -e "servers#fake@group\nclients#fake@group"
else echo "Unknown id: $id" >&2
fi
If the given id is either clients#fake@group or
servers#fake@group the script outputs the related hosts one per
line. If id is world#fake@group, this script outputs
clients#fake@group and servers#fake@group (otherwise, it exits
with an error message).
Create a file with the content above, adapt
it to your own configuration (with real hostnames for both clients
and servers) and save it as find_deps_for:
$ cat > ~/.local/bin/find_deps_for <copy/past>
Don't forget to change the executable bit:
$ chmod +x ~/.local/bin/find_deps_for
In the sequencer table, the dependson colums list the rules the
actual matching rule may depend on. For each identifier returned by
the depsfinder of a mathing rule, the sequencer looks if it matches
one rule in the dependson column.
In our case, the rule group depends on rules cmd and also on
itself. Our depsfinder may return both nodes for which rule cmd
should apply and groups for which rule group should apply. Note
however, that since rule cmd matches anything (both types and
filter are set to ALL), it also matches our
clients#fake@group and servers#fake@group. We do not want to
ping those components obviously. Therefore, we will update our cmd
rule as follow:
$ sequencer dbupdate pings cmd filter='%id !~ .*fake@group$'
Displaying the table gives (note how we hide the comments column
completely):
$ sequencer dbshow --columns=action:8,comments:0
ruleset | name | types | filter | action | depsfinder | dependson |
-------------------------------------------------------------------------------------------------
pings | cmd | ALL | %id !~ .*fake@group$ | /b..{0} | NONE | NONE |
pings | group | fake@group | ALL | NONE | find_deps_for %id | cmd,group |
Legend:
0: /bin/bash -c 'ping -nq -c 1 %name > /dev/null && echo Alive || echo Unreachable'
$
Now the cmd rule will not match (operator: '!~') component
identifier that ends with fake@group excluding our clients and
servers groups. The other matching operator is '=~'. Filter
can also be done through scripts. See depmake (1) man pages for
details.
With such a configuration, the execution gives:
$ sequencer pings world#fake@group login1#client@host/cmd: Alive login4#client@host/cmd: Alive login3#client@host/cmd: Alive www1#server@host/cmd: Alive www2#server@host/cmd: Alive nfs2#server@host/cmd: Alive login2#client@host/cmd: Alive nfs1#server@host/cmd: Alive $
It might not be easy to understand the dependencies between rules of a
given ruleset when looking to the sequencer table. The sequencer can
export a DOT format
representation of the rules graph. For example, the sequencer
table for the ruleset pings is currently:
$ sequencer dbshow pings --columns=name,types,filter,depsfinder,dependson name | types | filter | depsfinder | dependson | ---------------------------------------------------------------------------- cmd | ALL | %id !~ .*fake@group$ | NONE | NONE | group | fake@group | ALL | find_deps_for %id | cmd,group | $
Notice how we explicitely defined the list of columns we want to display.
The rules graph can be generated using the graphrules sequencer
action:
$ sequencer graphrules pings -o /tmp/pings.rg.dot $
The file /tmp/pings.rg.dot is a valid DOT file that can be parsed by any DOT reader such as Graphviz:
$ dot -T x11 /tmp/pings.rg.dot
The following graph should be displayed:
The sequencer can also produce a DOT graphical representation of the computed sequence of actions (called the actions graph):
$ sequencer pings world#fake@group --actionsgraphto=/tmp/pings.ag.dot www2#server@host/cmd: Alive login2#client@host/cmd: Alive login3#client@host/cmd: Alive nfs2#server@host/cmd: Alive nfs1#server@host/cmd: Alive www1#server@host/cmd: Alive login1#client@host/cmd: Alive login4#client@host/cmd: Alive $
The file /tmp/pings.ag.dot is a valid DOT file that can be parsed by any DOT reader such as Graphviz:
$ dot -T x11 -Kneato /tmp/pings.ag.dot
Note: we use the ``neato`` layout here. See ``dot (1)`` man page for details.
The following graph should be displayed:
[[pings.ag.jpg|alt=Actions graph DOT representation of the pings ruleset with the world#fake@group input list]]
As shown, despite our dependencies (world#fake@group depends on
both clients#fake@group and servers#fake@group for example),
none appears in the final execution.
The reason is that the sequencer is actually made of three distinct stages. In this tutorial, we have used the chaining feature: it chains all stages in one turn transparently. This mode of operation is called the blackbox mode: it hides most internal execution details and is very convenient for day to day use.
Hovewer, the sequencer also provides an incremental mode of operation which is described in Advanced Usage.
But to make it clear:
- the first stage -- alias the Dependency Graph Maker -- computes the dependency graph according to a given ruleset and a components input list;
- the second stage -- alias the Instructions Sequence Maker -- computes an instructions sequence from a dependency graph that may have been computed by the first stage (but this is not mandatory);
- the third stage -- alias the Instructions Sequencer Executor -- executes an instructions sequence that may have been computed by the second stage (but this is also not required).
The second stage may remove components that do not have any related
actions. In our example, this is the case for components world,
clients and servers. The sequencer provides a way to export
the dependency graph, that is the output of the first stage:
$ sequencer pings world#fake@group --depgraphto=/tmp/pings.dg.dot login1#client@host/cmd: Alive login3#client@host/cmd: Alive login4#client@host/cmd: Alive www2#server@host/cmd: Alive login2#client@host/cmd: Alive nfs2#server@host/cmd: Alive www1#server@host/cmd: Alive nfs1#server@host/cmd: Alive $
The file /tmp/pings.dg.dot is a valid DOT file that can be parsed by any DOT reader such as Graphviz:
$ dot -T x11 -Kdot /tmp/pings.dg.dot
Note: we use the ``dot`` layout here. See ``dot (1)`` man page for details.
The following graph should be displayed:
[[pings.dg.jpg|alt=Dependency graph DOT representation of the pings ruleset with the world#fake@group input list]]
And here, we clearly see our components world, clients and
servers with their related dependencies as expected.
At that stage, you may have a basic idea of what the sequencer can do for you (if this is not the case, please let us know: contact us).
You might move forward to the Advanced Usage section.