Skip to content

Commit 2585cdd

Browse files
eschabelleschabell
authored
Update the Amazon output plugin docs with YAML configuration examples. Fixes #1890. (#1891)
* Adding YAML examples and cleanup to Amazon CloudWatch output plugin. Part of issue #1890. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Adding YAML examples and cleanup to Amazon Kinesis Data Firehose output plugin. Part of issue #1890. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Adding YAML examples and cleanup to Amazon Kinesis Data Streams output plugin. Part of issue #1890. Signed-off-by: Eric D. Schabell <eric@schabell.org> * Adding YAML examples and cleanup to Amazon S3 output plugin. Part of issue #1890. Signed-off-by: Eric D. Schabell <eric@schabell.org> --------- Signed-off-by: Eric D. Schabell <eric@schabell.org>
1 parent 81b3909 commit 2585cdd

File tree

4 files changed

+532
-103
lines changed

4 files changed

+532
-103
lines changed

‎pipeline/outputs/cloudwatch.md‎

Lines changed: 172 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -45,15 +45,33 @@ In order to send records into Amazon Cloudwatch, you can run the plugin from the
4545

4646
The **cloudwatch** plugin, can read the parameters from the command line through the **-p** argument (property), e.g:
4747

48-
```
48+
```shell
4949
fluent-bit -i cpu -o cloudwatch_logs -p log_group_name=group -p log_stream_name=stream -p region=us-west-2 -m '*' -f 1
5050
```
5151

5252
### Configuration File
5353

54-
In your main configuration file append the following _Output_ section:
55-
54+
In your main configuration file append the following:
55+
56+
{% tabs %}
57+
{% tab title="fluent-bit.yaml" %}
58+
59+
```yaml
60+
pipeline:
61+
62+
outputs:
63+
- name: cloudwatch_logs
64+
match: '*'
65+
region: us-east-1
66+
log_group_name: fluent-bit-cloudwatch
67+
log_stream_prefix: from-fluent-bit-
68+
auto_create_group: on
5669
```
70+
71+
{% endtab %}
72+
{% tab title="fluent-bit.conf" %}
73+
74+
```text
5775
[OUTPUT]
5876
Name cloudwatch_logs
5977
Match *
@@ -62,22 +80,56 @@ In your main configuration file append the following _Output_ section:
6280
log_stream_prefix from-fluent-bit-
6381
auto_create_group On
6482
```
65-
#### Intergration with Localstack (Cloudwatch Logs)
6683

67-
For an instance of Localstack running at `http://localhost:4566`, the following configuration needs to be added to the `[OUTPUT]` section:
84+
{% endtab %}
85+
{% endtabs %}
86+
87+
#### Integration with Localstack (Cloudwatch Logs)
88+
89+
For an instance of `Localstack` running at `http://localhost:4566`, the following configuration is needed:
90+
91+
{% tabs %}
92+
{% tab title="fluent-bit.yaml" %}
93+
94+
```yaml
95+
pipeline:
96+
97+
outputs:
98+
- name: cloudwatch_logs
99+
match: '*'
100+
region: us-east-1
101+
log_group_name: fluent-bit-cloudwatch
102+
log_stream_prefix: from-fluent-bit-
103+
auto_create_group: on
104+
endpoint: localhost
105+
port: 4566
106+
```
107+
108+
{% endtab %}
109+
{% tab title="fluent-bit.conf" %}
68110
69111
```text
70-
endpoint localhost
71-
port 4566
112+
[OUTPUT]
113+
Name cloudwatch_logs
114+
Match *
115+
region us-east-1
116+
log_group_name fluent-bit-cloudwatch
117+
log_stream_prefix from-fluent-bit-
118+
auto_create_group On
119+
endpoint localhost
120+
port 4566
72121
```
73122

123+
{% endtab %}
124+
{% endtabs %}
125+
74126
Any testing credentials can be exported as local variables, such as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.
75127

76128
### Permissions
77129

78130
The following AWS IAM permissions are required to use this plugin:
79131

80-
```
132+
```json
81133
{
82134
"Version": "2012-10-17",
83135
"Statement": [{
@@ -100,7 +152,7 @@ Here is an example usage, for a common use case- templating log group and stream
100152

101153
Recall that the kubernetes filter can add metadata which will look like the following:
102154

103-
```
155+
```text
104156
kubernetes: {
105157
annotations: {
106158
"kubernetes.io/psp": "eks.privileged"
@@ -121,9 +173,29 @@ kubernetes: {
121173

122174
Using record\_accessor, we can build a template based on this object.
123175

124-
Here is our output configuration:
125-
176+
Here is our configuration:
177+
178+
{% tabs %}
179+
{% tab title="fluent-bit.yaml" %}
180+
181+
```yaml
182+
pipeline:
183+
184+
outputs:
185+
- name: cloudwatch_logs
186+
match: '*'
187+
region: us-east-1
188+
log_group_name: fallback-group
189+
log_stream_prefix: fallback-stream
190+
auto_create_group: on
191+
log_group_template: application-logs-$kubernetes['host'].$kubernetes['namespace_name']
192+
log_stream_template: $kubernetes['pod_name'].$kubernetes['container_name']
126193
```
194+
195+
{% endtab %}
196+
{% tab title="fluent-bit.conf" %}
197+
198+
```text
127199
[OUTPUT]
128200
Name cloudwatch_logs
129201
Match *
@@ -135,17 +207,20 @@ Here is our output configuration:
135207
log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
136208
```
137209

210+
{% endtab %}
211+
{% endtabs %}
212+
138213
With the above kubernetes metadata, the log group name will be `application-logs-ip-10-1-128-166.us-east-2.compute.internal.my-namespace`. And the log stream name will be `myapp-5468c5d4d7-n2swr.myapp`.
139214

140215
If the kubernetes structure is not found in the log record, then the `log_group_name` and `log_stream_prefix` will be used instead, and Fluent Bit will log an error like:
141216

142-
```
217+
```text
143218
[2022/06/30 06:09:29] [ warn] [record accessor] translation failed, root key=kubernetes
144219
```
145220

146221
#### Limitations of record\_accessor syntax
147222

148-
Notice in the example above, that the template values are separated by dot characters. This is important; the Fluent Bit record\_accessor library has a limitation in the characters that can separate template variables- only dots and commas (`.` and `,`) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.
223+
Notice in the example above, that the template values are separated by dot characters. This is important; the Fluent Bit record\_accessor library has a limitation in the characters that can separate template variables; only dots and commas (`.` and `,`) can come after a template variable. This is because the templating library must parse the template and determine the end of a variable.
149224

150225
Assume that your log records contain the metadata keys `container_name` and `task`. The following would be invalid templates because the two template variables are not separated by commas or dots:
151226

@@ -168,55 +243,130 @@ And the following are valid since they only contain one template variable with n
168243

169244
### Metrics Tutorial
170245

171-
Fluent Bit has different input plugins (cpu, mem, disk, netif) to collect host resource usage metrics. `cloudwatch_logs` output plugin can be used to send these host metrics to CloudWatch in Embedded Metric Format (EMF). If data comes from any of the above mentioned input plugins, `cloudwatch_logs` output plugin will convert them to EMF format and sent to CloudWatch as JSON log. Additionally, if we set `json/emf` as the value of `log_format` config option, CloudWatch will extract custom metrics from embedded JSON payload.
246+
Fluent Bit has different input plugins (cpu, mem, disk, netif) to collect host resource usage metrics. `cloudwatch_logs` output plugin can be used to send these host metrics to CloudWatch in Embedded Metric Format (EMF). If data comes from any of the above-mentioned input plugins, `cloudwatch_logs` output plugin will convert them to EMF format and sent to CloudWatch as JSON log. Additionally, if we set `json/emf` as the value of `log_format` config option, CloudWatch will extract custom metrics from embedded JSON payload.
172247

173248
Note: Right now, only `cpu` and `mem` metrics can be sent to CloudWatch.
174249

175250
For using the `mem` input plugin and sending memory usage metrics to CloudWatch, we can consider the following example config file. Here, we use the `aws` filter which adds `ec2_instance_id` and `az` (availability zone) to the log records. Later, in the output config section, we set `ec2_instance_id` as our metric dimension.
176251

252+
{% tabs %}
253+
{% tab title="fluent-bit.yaml" %}
254+
255+
```yaml
256+
service:
257+
log_level: info
258+
259+
pipeline:
260+
inputs:
261+
- name: mem
262+
tag: mem
263+
264+
filters:
265+
- name: aws
266+
match: '*'
267+
268+
outputs:
269+
- name: cloudwatch_logs
270+
match: '*'
271+
region: us-west-2
272+
log_stream_name: fluent-bit-cloudwatch
273+
log_group_name: fluent-bit-cloudwatch
274+
log_format: json/emf
275+
metric_namespace: fluent-bit-metrics
276+
metric_dimensions: ec2_instance_id
277+
auto_create_group: true
177278
```
279+
280+
{% endtab %}
281+
{% tab title="fluent-bit.conf" %}
282+
283+
```text
178284
[SERVICE]
179285
Log_Level info
180286

181287
[INPUT]
182288
Name mem
183289
Tag mem
184-
290+
185291
[FILTER]
186292
Name aws
187293
Match *
188294

189295
[OUTPUT]
190296
Name cloudwatch_logs
191297
Match *
298+
region us-west-2
192299
log_stream_name fluent-bit-cloudwatch
193300
log_group_name fluent-bit-cloudwatch
194-
region us-west-2
195301
log_format json/emf
196302
metric_namespace fluent-bit-metrics
197303
metric_dimensions ec2_instance_id
198304
auto_create_group true
199305
```
200306

307+
{% endtab %}
308+
{% endtabs %}
309+
201310
The following config will set two dimensions to all of our metrics- `ec2_instance_id` and `az`.
202311

312+
{% tabs %}
313+
{% tab title="fluent-bit.yaml" %}
314+
315+
```yaml
316+
service:
317+
log_level: info
318+
319+
pipeline:
320+
inputs:
321+
- name: mem
322+
tag: mem
323+
324+
filters:
325+
- name: aws
326+
match: '*'
327+
328+
outputs:
329+
- name: cloudwatch_logs
330+
match: '*'
331+
region: us-west-2
332+
log_stream_name: fluent-bit-cloudwatch
333+
log_group_name: fluent-bit-cloudwatch
334+
log_format: json/emf
335+
metric_namespace: fluent-bit-metrics
336+
metric_dimensions: ec2_instance_id,az
337+
auto_create_group: true
203338
```
339+
340+
{% endtab %}
341+
{% tab title="fluent-bit.conf" %}
342+
343+
```text
344+
[SERVICE]
345+
Log_Level info
346+
347+
[INPUT]
348+
Name mem
349+
Tag mem
350+
204351
[FILTER]
205352
Name aws
206353
Match *
207354

208355
[OUTPUT]
209356
Name cloudwatch_logs
210357
Match *
358+
region us-west-2
211359
log_stream_name fluent-bit-cloudwatch
212360
log_group_name fluent-bit-cloudwatch
213-
region us-west-2
214361
log_format json/emf
215362
metric_namespace fluent-bit-metrics
216363
metric_dimensions ec2_instance_id,az
217364
auto_create_group true
218365
```
219366

367+
{% endtab %}
368+
{% endtabs %}
369+
220370
### AWS for Fluent Bit
221371

222372
Amazon distributes a container image with Fluent Bit and these plugins.
@@ -231,19 +381,19 @@ Amazon distributes a container image with Fluent Bit and these plugins.
231381

232382
Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:
233383

234-
```
384+
```shell
235385
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>
236386
```
237387

238388
For example, you can pull the image with latest version by:
239389

240-
```
390+
```shell
241391
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
242392
```
243393

244394
If you see errors for image pull limits, try log into public ECR with your AWS credentials:
245395

246-
```
396+
```shell
247397
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws
248398
```
249399

@@ -257,8 +407,8 @@ You can check the [Amazon ECR Public official doc](https://docs.aws.amazon.com/A
257407

258408
You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:
259409

260-
```
410+
```shell
261411
aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/
262412
```
263413

264-
For more see [the AWS for Fluent Bit github repo](https://github.com/aws/aws-for-fluent-bit#public-images).
414+
For more see [the AWS for Fluent Bit GitHub repo](https://github.com/aws/aws-for-fluent-bit#public-images).

0 commit comments

Comments
 (0)