From 3a1f3cfd337d1e2eef7c4b092b25eb3a3c9501b1 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Wed, 11 May 2022 17:42:12 +0200 Subject: [PATCH 1/9] initial draft app docs Signed-off-by: R.I.Pienaar --- docs/content/development/appbuilder/_index.md | 330 ++++++++++++++++++ docs/content/development/middleware/_index.md | 38 +- 2 files changed, 349 insertions(+), 19 deletions(-) create mode 100644 docs/content/development/appbuilder/_index.md diff --git a/docs/content/development/appbuilder/_index.md b/docs/content/development/appbuilder/_index.md new file mode 100644 index 0000000..02a3405 --- /dev/null +++ b/docs/content/development/appbuilder/_index.md @@ -0,0 +1,330 @@ ++++ +title = "Application Builder" +weight = 30 ++++ + +Teams often find they have a number of `choria req` and other related commands that they execute frequently while +managing a large Choria deployment. These commands are hard to discover or share, frequently sat in runbooks where +they quickly get out of date. + +We want to make it easy to create per team CLI macro style tools that invoke Choria RPC Agents, Interact with Key-Value +buckets or call other external helpers. + +The Application Builder allows you to describe a CLI in a YAML file and will automatically build a `choria` like CLI +for you based on this input definition. + +The application will feel comfortable within any Choria deployment and it's entire feature set will be discoverable +using `--help`. + +Initially we support calling Choria Agents, Interacting with Choria Key-Value Store and executing external commands. + +## Exploratory Example + +Here we see a short example CLI built using this framework: + +```nohighlight +$ opscmd +usage: opscmd [] [ ...] + +Example Inc Operations Tools + +Flags: + --help Show context-sensitive help (also try --help-long and + --help-man). + +Commands: + help [...] + Show help. + + fleet info [] + Get Choria Versions, broker distribution and more + + provisioning reprovision [] + Re-provision running nodes + + Reprovisioning will remove the node from the network and reset it's + configuration to factory default, it will then get a new + configuration, credentials and more from the Choria Provisioner. + + provisioning restart [] + Restarts running nodes + + provisioner force-election + Force a leader election + + provisioner leader + Obtain the name of the current cluster leader + + machines [] + Query Autonomous Agents +``` + +We'll focus on the `machines` example a bit as it shows the current sweet spot behavior: + +```nohighlight +$ opscmd machines --help +usage: opscmd machines [] + +Query Autonomous Agents + +Flags: + --help Show context-sensitive help (also try + --help-long and --help-man). + --senders List only the names of matching nodes + --json Render results as JSON + --table Render results as a table + --display=DISPLAY Display only a subset of results (ok, failed, all, none) + --ne=VERSION Matches instances that are not this version + --le=VERSION Matches instances that are older than or same as + --lt=VERSION Matches instances that are older than + --ge=VERSION Matches instances that are newer than or same as + --gt=VERSION Matches instances that are older than + --eq=VERSION Matches instances that are same as + +Args: + Autonomous Agent to report on +``` + +We see we have a fairly standard `choria req` like CLI here, but, we have a number of extra flags like `--ne` and `--gt`. + +Many of the standard fields here are opt-in, you can elect to not have discovery flags (not shown), filters, +output formats (shown) or batching enabled (not shown). + +When one runs: `opscmd machines example --ne 1.0.0 --senders` a list of nodes where the Autonomous Agent `example` is not +at version `1.0.0`. This is equivalent to running: + +```nohighlight +choria req choria_util machine_state machine=example --filter-replies 'ok() && semver(data("version"), "!= 1.0.0")' --senders +``` + +To create this command we had to create the following bit of YAML (abstract): + +```yaml +name: opscmd +description: Example Inc Operations Tools +version: 0.0.1 +author: https://wiki.example.com/ + +commands: +- name: machines + description: Query Autonomous Agents + type: rpc + std_filters: false + output_formats_flags: true + display_flag: true + batch_flags: false + request: + agent: choria_util + action: machine_state + params: + name: "{{.Arguments.machine}}" + + arguments: + - name: machine + description: Autonomous Agent to report on + required: true + + flags: + - name: ne + description: Matches instances that are not this version + place_holder: VERSION + filter: ok() && semver(data("version"), "!= {{.Flags.ne}}") +``` + +We place this file in `/etc/choria/builder/opscmd-app.yaml` and create a symlink `/usr/bin/opscmd` that points to the +`choria` binary, anyone who invokes `opscmd` will now run the custom application. + +## Configuration + +A configuration file `applications.yaml` can be made that will be read and values made available in templates using +`{{ .Config.value | require "value is required in configuration" }}`. + +The file should be a YAML file that is all string values. + +```yaml +# example applications.yaml +password: s3cret +``` + +The configuration file and application definitions can be put in `.`, `~/.config/choria/builder` or `/etc/choria/builder`. + +Once a definition is placed in any of the above locations in a file like `opscmd-app.yaml`, simply make a symlink from +`/usr/bin/opscmd` (or anywhere in your path) to `/usr/bin/choria`. + +## Reference + +The Specification YAML file is validated using a JSON Schema, as a temporary location this Schema is in the +[Choria source repository](https://raw.githubusercontent.com/choria-io/go-choria/main/internal/fs/schemas/builder.json), +once this feature is complete it will be stored in our usual schema repository. + +Each Specification must have these fields: + +```yaml +# CLI tool name, must be all lower case, no spaces or numbers. Can have dash and underscore +name: opscmd + +# Human friendly description shown on initial --help page +description: Example Inc Operations Tools + +# Version reported in opscmd --version +version: 0.0.1 + +# Any author hint, opscmd --help-man makes a man page that includes this +author: https://wiki.example.com/ +``` + +### Generic Command + +All commands types have at least these behaviors so we'll show the generic options and then what makes commands types +unique in specific sections. + +```yaml +commands: + - name: example + description: An example sub command + # Can also be called as 'opscmd ex' or 'opscmd e' + aliases: + - ex + - e + # The type of command this is, one of rpc, exec, kv or parent + type: x + commands: [] # sub commands that are any valid command + + # arguments are like 'opscmd command argument' they do not require dashes before and + # tend to be reserved for things that are required, though the last ones can all be + # optional + arguments: + - name: item + description: The item to act on + required: true + + # Flags are passed like --value=x, today we only support string value flags, they can + # can be required if needed + flags: + - name: value + description: Sets a value + # in --help this looks like --value=VALUE by default, --value=VAL with this + placeholder: VAL + required: false +``` + +### `parent` Command Type + +The `parent` command type does not do anything, it just allows you to create a set of related sub commands. + +Arguments and Flags are ignored. + +```yaml +commands: + - name: provisioning + description: Fleet provisioning actions + type: parent + commands: [] # here list rpc, kv, exec or parent commands. +``` + +### `exec` Command Type + +The `exec` command type will just run some other command. When the limitations placed on this framework prevents +you from doing something, or you just have a shell script already to do something, you can call them from here. + +In the example below when someone runs `opscmd install --version 1.2.3` it will invoke `/usr/local/bin/install.rb 1.2.3`, +the version argument is required. + +```yaml +commands: + - name: install + description: Installs and Configured something + type: exec + command: /usr/local/bin/install.rb --version {{ .Arguments.version }} + arguments: + - name: version + description: The version to install + required: true +``` + +### `kv` Command Type + +The `kv` command can Put, Get or Delete data from a Choria Key-Value bucket. + +The example below will show a current leader for something using Choria Leader Election and force a Leader Election by +deleting some data. It also shows creating some data. + +When creating data we support template expansion to read values from Flags, Arguments or Configuration. + +```yaml +commands: + - name: app + description: Manages app + type: parent + commands: + - name: leader + description: Obtain the name of the current cluster leader + type: kv + action: get + bucket: CHORIA_LEADER_ELECTION + key: app + + - name: force-election + description: Force a leader election + type: kv + action: del + bucket: CHORIA_LEADER_ELECTION + key: app + + - name: upgrade + description: Triggers an upgrade to a new version + type: kv + action: put + bucket: APP_CONFIG + key: version + value: "{{ .Arguments.version }}" + arguments: + - name: version + description: The version to deploy + required: true +``` + +### `rpc` Command Type + +The `rpc` command type invokes Choria RPC Agents, it renders data like `choria req` and supports almost 100% compatible +flags as `choria req`, essentially it's a frontend with configurable defaults. + +In the example below we initial the `machine_state` action on `choria_util` agent with properties built from command line +arguments, flags, and we set some defaults. + + * `std_filters` will enable `-C`, `-I`, `-S`, `-F` and friends + * `output_format_flags` will enable `--json`, `--table`, `--senders` etc + * `display_flag` will enable `--display` + * `batch_flags` will enable `--batch` and `--batch-sleep` + +All of these match their `choria req` counter-parts and all are off by default. + +This type will have a number of features added to it over the following days including the ability to force display, +batching, output format and filters + +```yaml +commands: + - name: machines + description: Query Autonomous Agents + type: rpc + std_filters: true + output_formats_flags: true + display_flag: true + batch_flags: true + request: + agent: choria_util + action: machine_state + params: + name: "{{.Arguments.machine}}" + + arguments: + - name: machine + description: Autonomous Agent to report on + required: true + + flags: + - name: ne + description: Matches instances that are not this version + place_holder: VERSION + filter: ok() && semver(data("version"), "!= {{.Flags.ne}}") +``` diff --git a/docs/content/development/middleware/_index.md b/docs/content/development/middleware/_index.md index fdee084..e6313df 100644 --- a/docs/content/development/middleware/_index.md +++ b/docs/content/development/middleware/_index.md @@ -1,6 +1,6 @@ +++ title = "Middleware" -weight = 30 +weight = 40 +++ Choria communicates almost exclusively over the middleware, this document describes the target names and messages you might find on the middleware. You might use this for debuging or learning purposes using the `choria tool pub` and `choria tool sub` commands. @@ -15,13 +15,13 @@ This is the core RPC related message targets, this is how machines discover node These are namespaced on *collective* allowing one to make something like a VLAN - where servers and clients in a particular Subcollective can only find others configured with the same. -|Target|Description|Schema| -|------|-----------|------| -|`collective.reply.>`|Replies to requests made by clients, example *mcollective.reply.dev1.example.net.c2a764e6013a44adb848904ff7d74ff4*|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:reply:1](https://choria.io/schemas/choria/protocol/v1/secure_reply.json) payload| -|`collective.broadcast.agent.>`|Requests from clients to specific agents broadcasted to all servers interested|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:request:1](https://choria.io/schemas/choria/protocol/v1/secure_request.json) payload| -|`collective.broadcast.service.>`|Requests from clients to specific Service Agents broadcasted to all servers interested, handled by only 1 of those hosting the Service.|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:request:1](https://choria.io/schemas/choria/protocol/v1/secure_request.json) payload| -|`collective.broadcast.agent.registration`|The default destination for registration messages when no custom targets are configured|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json)| -|`collective.node.>`|Requests from clients to specific nodes regardless of the agent aka *directed*|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:request:1](https://choria.io/schemas/choria/protocol/v1/secure_request.json) payload| +| Target | Description | Schema | +|-------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `collective.reply.>` | Replies to requests made by clients, example *mcollective.reply.dev1.example.net.c2a764e6013a44adb848904ff7d74ff4* | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:reply:1](https://choria.io/schemas/choria/protocol/v1/secure_reply.json) payload | +| `collective.broadcast.agent.>` | Requests from clients to specific agents broadcasted to all servers interested | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:request:1](https://choria.io/schemas/choria/protocol/v1/secure_request.json) payload | +| `collective.broadcast.service.>` | Requests from clients to specific Service Agents broadcasted to all servers interested, handled by only 1 of those hosting the Service. | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:request:1](https://choria.io/schemas/choria/protocol/v1/secure_request.json) payload | +| `collective.broadcast.agent.registration` | The default destination for registration messages when no custom targets are configured | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) | +| `collective.node.>` | Requests from clients to specific nodes regardless of the agent aka *directed* | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:request:1](https://choria.io/schemas/choria/protocol/v1/secure_request.json) payload | ### Federated @@ -31,10 +31,10 @@ Likewise replies are received on a different target and the Federation Broker wi These are namespaced on *collective* here, each Federation *Member Collective* has a unique name. -|Target|Description|Schema| -|------|-----------|------| -|`choria.federation.collective.collective`|When federated, replies to clients from server that would typically go to `collective.reply.>` are directed here instead|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:reply:1](https://choria.io/schemas/choria/protocol/v1/secure_reply.json) payload| -|`choria.federation.collective.federation`|When federated, requests to servers that would typically go to *collective.broadcast.>* or *collective.node>* goes here instead|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:request:1](https://choria.io/schemas/choria/protocol/v1/secure_request.json) payload| +| Target | Description | Schema | +|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `choria.federation.collective.collective` | When federated, replies to clients from server that would typically go to `collective.reply.>` are directed here instead | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:reply:1](https://choria.io/schemas/choria/protocol/v1/secure_reply.json) payload | +| `choria.federation.collective.federation` | When federated, requests to servers that would typically go to *collective.broadcast.>* or *collective.node>* goes here instead | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:secure:request:1](https://choria.io/schemas/choria/protocol/v1/secure_request.json) payload | ## Lifecycle @@ -43,13 +43,13 @@ Lifecycle messages are small events the various Choria components publish indica Read our [introductory blog post about events](https://choria.io/blog/post/2019/01/03/lifecycle/). -|Target|Description|Schema| -|------|-----------|------| -|`choria.lifecycle.event.TYPE.COMPONENT`|A specific event from a specific component, example *choria.lifecycle.event.startup.server* when a server starts|[various](https://github.com/choria-io/schemas/tree/master/choria/lifecycle/v1)| +| Target | Description | Schema | +|-----------------------------------------|------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------| +| `choria.lifecycle.event.TYPE.COMPONENT` | A specific event from a specific component, example *choria.lifecycle.event.startup.server* when a server starts | [various](https://github.com/choria-io/schemas/tree/master/choria/lifecycle/v1) | ## Others -|Target|Description|Schema| -|------|-----------|------| -|*>*|Custom registration target set using *plugin.choria.registration.file_content.target*|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:registration:filecontent:1](https://choria.io/schemas/choria/registration/v1/filecontent.json) payload| -|*>*|Custom sources for consuming messages by the adapters like those set by *plugin.choria.adapter.xx.ingest.topic*|[choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a payload depending on the producer settings| +| Target | Description | Schema | +|--------|-----------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `>` | Custom registration target set using *plugin.choria.registration.file_content.target* | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a [choria:registration:filecontent:1](https://choria.io/schemas/choria/registration/v1/filecontent.json) payload | +| `>` | Custom sources for consuming messages by the adapters like those set by *plugin.choria.adapter.xx.ingest.topic* | [choria:transport:1](https://choria.io/schemas/choria/protocol/v1/transport.json) with a payload depending on the producer settings | From ce0200e1c14a0a0da4d6fccacbd166c9360000d7 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Thu, 12 May 2022 17:52:16 +0200 Subject: [PATCH 2/9] update for latest schema Signed-off-by: R.I.Pienaar --- docs/content/development/appbuilder/_index.md | 44 +++++++++++++++++-- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/docs/content/development/appbuilder/_index.md b/docs/content/development/appbuilder/_index.md index 02a3405..9911af8 100644 --- a/docs/content/development/appbuilder/_index.md +++ b/docs/content/development/appbuilder/_index.md @@ -18,6 +18,12 @@ using `--help`. Initially we support calling Choria Agents, Interacting with Choria Key-Value Store and executing external commands. +{{% notice warning %}} +This is an emerging proof of concept feature that will be introduced in Choria 0.26.0. It's subject to change, and, we +might even entirely remove it should this prove to be too restrictive as a replacement for the old application framework. +{{% /notice %}} + + ## Exploratory Example Here we see a short example CLI built using this framework: @@ -110,7 +116,7 @@ commands: description: Query Autonomous Agents type: rpc std_filters: false - output_formats_flags: true + output_format_flags: true display_flag: true batch_flags: false request: @@ -249,7 +255,7 @@ The `kv` command can Put, Get or Delete data from a Choria Key-Value bucket. The example below will show a current leader for something using Choria Leader Election and force a Leader Election by deleting some data. It also shows creating some data. -When creating data we support template expansion to read values from Flags, Arguments or Configuration. +When creating data we support template expansion to read values from `.Flags`, `.Arguments` or `.Configuration`. ```yaml commands: @@ -308,7 +314,7 @@ commands: description: Query Autonomous Agents type: rpc std_filters: true - output_formats_flags: true + output_format_flags: true display_flag: true batch_flags: true request: @@ -328,3 +334,35 @@ commands: place_holder: VERSION filter: ok() && semver(data("version"), "!= {{.Flags.ne}}") ``` + +While above, we enabled flags like `--batch`, we can instead disable `--batch` entirely and force it to a specific value, +we can do the same with a few other options shown here. In all cases when you set it specifically you cannot also set +the option to enable matching flags from being set: + +```yaml +commands: + - name: machines + description: Query Autonomous Agents + type: rpc + + # sets batch settings, prevents batch_flags from being added to the cli options + batch: 10 + batch_sleep: 60 + + # sets display mode overriding the DDL, prevents display_flag from being used + display: none + + # disables the progress bar + no_progress: true + + # forces JSON output, prevents output_format_flags from being used + output_format: json + + # sets specific discovery filters and options, prevent std_filters from being used + filter: + facts: [operatingsystem=CentOS] + discovery_method: choria + classes: [puppet] + + # as before +``` From f4a6bd92cf8b1376cd9c640fcfdd96b45b275c6c Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Thu, 12 May 2022 18:35:23 +0200 Subject: [PATCH 3/9] minor tweaks Signed-off-by: R.I.Pienaar --- docs/content/development/appbuilder/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/development/appbuilder/_index.md b/docs/content/development/appbuilder/_index.md index 9911af8..3a41359 100644 --- a/docs/content/development/appbuilder/_index.md +++ b/docs/content/development/appbuilder/_index.md @@ -19,7 +19,7 @@ using `--help`. Initially we support calling Choria Agents, Interacting with Choria Key-Value Store and executing external commands. {{% notice warning %}} -This is an emerging proof of concept feature that will be introduced in Choria 0.26.0. It's subject to change, and, we +This is a proof of concept feature that will be introduced in Choria 0.26.0. It's subject to change and, we might even entirely remove it should this prove to be too restrictive as a replacement for the old application framework. {{% /notice %}} From 16d362853d63e3db2700fa605d48a8f1d6674dc6 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Sat, 14 May 2022 20:04:44 +0200 Subject: [PATCH 4/9] update for recent additions Signed-off-by: R.I.Pienaar --- docs/content/development/appbuilder/_index.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/content/development/appbuilder/_index.md b/docs/content/development/appbuilder/_index.md index 3a41359..4c4c0a5 100644 --- a/docs/content/development/appbuilder/_index.md +++ b/docs/content/development/appbuilder/_index.md @@ -196,6 +196,9 @@ commands: type: x commands: [] # sub commands that are any valid command + # Prompts for a y/n before performing the action, no prompt if not set + confirm_prompt: Do you really want to perform this action + # arguments are like 'opscmd command argument' they do not require dashes before and # tend to be reserved for things that are required, though the last ones can all be # optional @@ -250,7 +253,7 @@ commands: ### `kv` Command Type -The `kv` command can Put, Get or Delete data from a Choria Key-Value bucket. +The `kv` command can Put, Get, Delete or History data from a Choria Key-Value bucket. The example below will show a current leader for something using Choria Leader Election and force a Leader Election by deleting some data. It also shows creating some data. @@ -288,6 +291,13 @@ commands: - name: version description: The version to deploy required: true + + - name: history + description: View recent deployed versions + type: kv + action: history + bucket: APP_CONFIG + key: version ``` ### `rpc` Command Type @@ -302,6 +312,7 @@ arguments, flags, and we set some defaults. * `output_format_flags` will enable `--json`, `--table`, `--senders` etc * `display_flag` will enable `--display` * `batch_flags` will enable `--batch` and `--batch-sleep` + * `all_nodes_confirm_prompt` will prompt for confirmation when no filters are given - meaning all nodes will be acted on All of these match their `choria req` counter-parts and all are off by default. @@ -317,10 +328,11 @@ commands: output_format_flags: true display_flag: true batch_flags: true + all_nodes_confirm_prompt: Really act on all nodes without a filter request: agent: choria_util action: machine_state - params: + inputs: name: "{{.Arguments.machine}}" arguments: From 7e5b7c91b90be5d92340462b07f5addcaf4af949 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Sun, 15 May 2022 12:48:44 +0200 Subject: [PATCH 5/9] document functions, document filter options merge Signed-off-by: R.I.Pienaar --- docs/content/development/appbuilder/_index.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/content/development/appbuilder/_index.md b/docs/content/development/appbuilder/_index.md index 4c4c0a5..12701a9 100644 --- a/docs/content/development/appbuilder/_index.md +++ b/docs/content/development/appbuilder/_index.md @@ -157,6 +157,15 @@ The configuration file and application definitions can be put in `.`, `~/.config Once a definition is placed in any of the above locations in a file like `opscmd-app.yaml`, simply make a symlink from `/usr/bin/opscmd` (or anywhere in your path) to `/usr/bin/choria`. +## Templates + +Some strings like commands, inputs and more are parsed via the Go template language. We have a few functions to help +with common tasks. + + * `require` - requires a string is set, mostly used for accessing configuration data `{{ .Config.value | require "value must be set"}}`, here we see how to set a custom error message + * `escape` - perform shell escaping on the value, useful for executing external commands with user input `command: /bin/upgrade.sh "{{ .Arguments.version | escape }}"` + * `read_filter` - reads the contents of a file, you can use this to read files given on the CLI as arguments or flags + ## Reference The Specification YAML file is validated using a JSON Schema, as a temporary location this Schema is in the @@ -348,8 +357,8 @@ commands: ``` While above, we enabled flags like `--batch`, we can instead disable `--batch` entirely and force it to a specific value, -we can do the same with a few other options shown here. In all cases when you set it specifically you cannot also set -the option to enable matching flags from being set: +we can do the same with a few other options shown here. In all cases except `filter` when you set it specifically you cannot +also set the option to enable matching flags from being set, in the case of `filter` it will merge with `std_filter`: ```yaml commands: @@ -370,7 +379,7 @@ commands: # forces JSON output, prevents output_format_flags from being used output_format: json - # sets specific discovery filters and options, prevent std_filters from being used + # sets specific discovery filters and options, will be merged with std_options filter: facts: [operatingsystem=CentOS] discovery_method: choria From 6663060ce63ab8d509157ccba2e72a9515ee2d58 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Mon, 16 May 2022 18:19:49 +0200 Subject: [PATCH 6/9] document the discovery command type Signed-off-by: R.I.Pienaar --- docs/content/development/appbuilder/_index.md | 38 ++++++++++++++++++- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/docs/content/development/appbuilder/_index.md b/docs/content/development/appbuilder/_index.md index 12701a9..5056824 100644 --- a/docs/content/development/appbuilder/_index.md +++ b/docs/content/development/appbuilder/_index.md @@ -16,7 +16,8 @@ for you based on this input definition. The application will feel comfortable within any Choria deployment and it's entire feature set will be discoverable using `--help`. -Initially we support calling Choria Agents, Interacting with Choria Key-Value Store and executing external commands. +Initially we support calling Choria Agents, Interacting with Choria Key-Value Store, Performing Choria Discovery, Fleet +Executing external commands. {{% notice warning %}} This is a proof of concept feature that will be introduced in Choria 0.26.0. It's subject to change and, we @@ -309,12 +310,43 @@ commands: key: version ``` +### `discover` Command Type + +The `discover` command type invokes Choria Discovery, it renders data as lists of nodes in one node per line or JSON format. + +In the example below we find all machines where a specific filter matches, we allow additional filters to be set on the CLI. + +```yaml +commands: + - name: list + description: List the servers running the service + type: discover + std_filters: true + filter: + classes: ["profiles::nats"] +``` + +The filter can take all standard Choria filters including picking the discovery method and discovery options. + + * `collective` - target a specific collective + * `facts` - a list of fact filters like `["region=us-east-1"]`, matches `-W` + * `agents` - a list of agents that should be on the node like `["rpcutil"]`, matches `-A` + * `classes` - a list of configuration management tags like `["apache"]`, matched `-C` + * `identities` - a list of identities to match like `["/db.example.net/"]`, matches `-I` + * `combined` - a list of combined filters like `["apache", "region=us-east-1"]`, matches `-W` + * `compound` - a compound filter to apply like `choria().version="0.25.1"`, matches `-S` + * `discovery_method` - sets one of the discovery methods to use + * `discovery_timeout` - time to allow discovery to run, in seconds + * `dynamic_discovery_timeout` - use a window of reply times to speed up discovery + * `nodes_file` - read targets from a file in yaml, json or plain text formats + * `discovery_options` - options to pass to the discovery method as a hash of strings + ### `rpc` Command Type The `rpc` command type invokes Choria RPC Agents, it renders data like `choria req` and supports almost 100% compatible flags as `choria req`, essentially it's a frontend with configurable defaults. -In the example below we initial the `machine_state` action on `choria_util` agent with properties built from command line +In the example below we perform a RPC request to the `machine_state` action on `choria_util` agent with properties built from command line arguments, flags, and we set some defaults. * `std_filters` will enable `-C`, `-I`, `-S`, `-F` and friends @@ -338,6 +370,8 @@ commands: display_flag: true batch_flags: true all_nodes_confirm_prompt: Really act on all nodes without a filter + filter: + facts: ["team=ops"] request: agent: choria_util action: machine_state From f46442bf083069c6247384d419080d38b56b1248 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Tue, 17 May 2022 15:39:04 +0200 Subject: [PATCH 7/9] document rpc transforms Signed-off-by: R.I.Pienaar --- docs/content/development/appbuilder/_index.md | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/docs/content/development/appbuilder/_index.md b/docs/content/development/appbuilder/_index.md index 5056824..639c49f 100644 --- a/docs/content/development/appbuilder/_index.md +++ b/docs/content/development/appbuilder/_index.md @@ -421,3 +421,32 @@ commands: # as before ``` + +Data can be transformed using JQ (specifically we use [gojq](https://github.com/itchyny/gojq)), the data that will be processed +using JQ will be exactly the structure you might find using `choria req --json`. Here we retrieve the status of an autonomous +agent and transform the data: + +```yaml +commands: + - name: state + description: Obtain the state of the service operator + type: rpc + std_filters: true + transform: + query: '.replies | .[] | select(.statuscode==0) | .sender + ": " + .data.state' + filter: + classes: ["profiles::nats"] + request: + agent: choria_util + action: machine_state + inputs: + name: nats +``` + +The output will look something like this: + +```nohighlight +n1-example: RUN +n2-example: RUN +n3-example: MAINTENANCE +``` From c8acfb0b94c03d2a9e11f5e614331a2bc52ed5d2 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Tue, 17 May 2022 17:41:11 +0200 Subject: [PATCH 8/9] (#1665) move filter to a more appropriate place Signed-off-by: R.I.Pienaar --- docs/content/development/appbuilder/_index.md | 23 +++++++++++-------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/content/development/appbuilder/_index.md b/docs/content/development/appbuilder/_index.md index 639c49f..fecc86d 100644 --- a/docs/content/development/appbuilder/_index.md +++ b/docs/content/development/appbuilder/_index.md @@ -370,11 +370,11 @@ commands: display_flag: true batch_flags: true all_nodes_confirm_prompt: Really act on all nodes without a filter - filter: - facts: ["team=ops"] request: agent: choria_util action: machine_state + filter: + facts: ["team=ops"] inputs: name: "{{.Arguments.machine}}" @@ -387,7 +387,7 @@ commands: - name: ne description: Matches instances that are not this version place_holder: VERSION - filter: ok() && semver(data("version"), "!= {{.Flags.ne}}") + reply_filter: ok() && semver(data("version"), "!= {{.Flags.ne}}") ``` While above, we enabled flags like `--batch`, we can instead disable `--batch` entirely and force it to a specific value, @@ -413,11 +413,14 @@ commands: # forces JSON output, prevents output_format_flags from being used output_format: json - # sets specific discovery filters and options, will be merged with std_options - filter: - facts: [operatingsystem=CentOS] - discovery_method: choria - classes: [puppet] + request: + agent: rpcutil + action: ping + # sets specific discovery filters and options, will be merged with std_options + filter: + facts: [operatingsystem=CentOS] + discovery_method: choria + classes: [puppet] # as before ``` @@ -434,11 +437,11 @@ commands: std_filters: true transform: query: '.replies | .[] | select(.statuscode==0) | .sender + ": " + .data.state' - filter: - classes: ["profiles::nats"] request: agent: choria_util action: machine_state + filter: + classes: ["profiles::nats"] inputs: name: nats ``` From cafcec4db73d4693284cf259f342f7e197d9ef40 Mon Sep 17 00:00:00 2001 From: "R.I.Pienaar" Date: Wed, 18 May 2022 10:29:26 +0200 Subject: [PATCH 9/9] document semver in compound filters Signed-off-by: R.I.Pienaar --- docs/content/concepts/discovery/_index.md | 24 ++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/content/concepts/discovery/_index.md b/docs/content/concepts/discovery/_index.md index 53d0e25..33099a9 100644 --- a/docs/content/concepts/discovery/_index.md +++ b/docs/content/concepts/discovery/_index.md @@ -182,19 +182,20 @@ $ choria discovery -S '(with("customer=acme") && with("environment=staging")) || Within the expressions we have defined some variables: -|Variable|Description| -|--------|-----------| -|`agents`|List of known agents| -|`classes`|List of classes this machine belongs to| -|`facts`|Facts for the machine as raw JSON| +| Variable | Description | +|-----------|-----------------------------------------| +| `agents` | List of known agents | +| `classes` | List of classes this machine belongs to | +| `facts` | Facts for the machine as raw JSON | And we made a few functions available: -|Function|Description| -|--------|-----------| -|`with` |Equivalent of a `-W` filter - class and fact matches combined with regular expression support| -|`fact` |Retrieves a fact from the nested fact data using [GJSON path syntax](https://github.com/tidwall/gjson/blob/master/SYNTAX.md)| -|`include`|Checks if an array includes a specific element| +| Function | Description | +|-----------|------------------------------------------------------------------------------------------------------------------------------| +| `with` | Equivalent of a `-W` filter - class and fact matches combined with regular expression support | +| `fact` | Retrieves a fact from the nested fact data using [GJSON path syntax](https://github.com/tidwall/gjson/blob/master/SYNTAX.md) | +| `include` | Checks if an array includes a specific element | +| `semver` | Performs semver version matching (since 0.26.0) | We can go really deep as here: @@ -204,7 +205,8 @@ with('apache') and # class or agent 'apache' with('fnumber=1.2') and # fact fnumber with a float value equals 1.2 fact('nested.string') matches('h.llo') and # lookup a fact 'nested.string' and regex match it with 'h.llo' include(fact('sarray'), '1') and # check if the 'sarray' fact - a array of strings - include a value '1' - include(fact('iarray'), 1) # check if the 'iarray' fact - a array of ints - include a value 1 + include(fact('iarray'), 1) and # check if the 'iarray' fact - a array of ints - include a value 1 + semver(fact('facterversion'), '>= 4.0.0') # matches the version of facter to ``` Here the *include* command is a basic function to check if an array contains something and *fact()* just looks up a fact using GJSON.