You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/inputs/tail.md
+37-37Lines changed: 37 additions & 37 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,33 +10,33 @@ The plugin supports the following configuration parameters:
10
10
11
11
| Key | Description | Default |
12
12
| :-- | :---------- | :------ |
13
-
|`Buffer_Chunk_Size`| Set the initial buffer size to read file data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. |`32k`|
14
-
|`Buffer_Max_Size`| Set the limit of the buffer size per monitored file. When a buffer needs to be increased, this value is used to restrict the memory buffer growth. If reading a file exceeds this limit, the file is removed from the monitored file list. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. |`32k`|
15
-
|`Path`| Pattern specifying a specific log file or multiple ones through the use of common wildcards. Allows multiple patterns separated by commas. |_none_|
16
-
|`Path_Key`| If enabled, it appends the name of the monitored file as part of the record. The value assigned becomes the key in the map. |_none_|
17
-
|`Exclude_Path`| Set one or multiple shell patterns separated by commas to exclude files matching certain criteria, For example, `Exclude_Path *.gz,*.zip`. |_none_|
18
-
|`Offset_Key`| If enabled, Fluent Bit appends the offset of the current monitored file as part of the record. The value assigned becomes the key in the map. |_none_|
19
-
|`Read_from_Head`| For new discovered files on start (without a database offset/position), read the content from the head of the file, not tail. |`False`|
20
-
|`Refresh_Interval`| The interval of refreshing the list of watched files in seconds. |`60`|
21
-
|`Rotate_Wait`| Specify the number of extra time in seconds to monitor a file once is rotated in case some pending data is flushed. |`5`|
22
-
|`Ignore_Older`| Ignores files older than `ignore_older`. Supports `m`, `h`, `d` (minutes, hours, days) syntax. | Read all. |
23
-
|`Skip_Long_Lines`| When a monitored file reaches its buffer capacity due to a very long line (`Buffer_Max_Size`), the default behavior is to stop monitoring that file. `Skip_Long_Lines` alter that behavior and instruct Fluent Bit to skip long lines and continue processing other lines that fit into the buffer size. |`Off`|
24
-
|`Skip_Empty_Lines`| Skips empty lines in the log file from any further processing or output. |`Off`|
25
-
|`DB`| Specify the database file to keep track of monitored files and offsets. |_none_|
26
-
|`DB.sync`| Set a default synchronization (I/O) method. This flag affects how the internal SQLite engine do synchronization to disk, for more details about each option see [the SQLite documentation](https://www.sqlite.org/pragma.html#pragma_synchronous). Most scenarios will be fine with `normal` mode. If you need full synchronization after every write operation set `full` mode. `full` has a high I/O performance cost. Values: `Extra`, `Full`, `Normal`, `Off`. |`normal`|
27
-
|`DB.locking`| Specify that the database will be accessed only by Fluent Bit. Enabling this feature helps increase performance when accessing the database but restricts externals tool from querying the content. |`false`|
28
-
|`DB.journal_mode`| Sets the journal mode for databases (`WAL`). Enabling `WAL` provides higher performance. `WAL` isn't compatible with shared network file systems. |`WAL`|
29
-
|`DB.compare_filename`| This option determines whether to review both `inode` and `filename` when retrieving stored file information from the database. `true` verifies both `inode` and `filename`, while `false` checks only the `inode`. To review the `inode` and `filename` in the database, refer [see `keep_state`](#tailing-files-keeping-state). |`false`|
30
-
|`Mem_Buf_Limit`| Set a memory limit that Tail plugin can use when appending data to the engine. If the limit is reached, it will be paused. When the data is flushed it resumes. |_none_|
31
-
|`Exit_On_Eof`| When reading a file will exit as soon as it reach the end of the file. Used for bulk load and tests. |`false`|
32
-
|`Parser`| Specify the name of a parser to interpret the entry as a structured message. |_none_|
33
-
|`Key`| When a message is unstructured (no parser applied), it's appended as a string under the key name `log`. This option lets you define an alternative name for that key. |`log`|
34
-
|`Inotify_Watcher`| Set to `false` to use file stat watcher instead of `inotify`. |`true`|
35
-
|`Tag`| Set a tag with `regexextract` fields that will be placed on lines read. For example, `kube.<namespace_name>.<pod_name>.<container_name>.<container_id>`. Tag expansion is supported: if the tag includes an asterisk (`*`), that asterisk will be replaced with the absolute path of the monitored file, with slashes replaced by dots. See [Workflow of Tail + Kubernetes Filter](../filters/kubernetes.md#workflow-of-tail--kubernetes-filter). |_none_|
36
-
|`Tag_Regex`| Set a regular expression to extract fields from the filename. For example: `(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64})\.log$`. |_none_|
37
-
|`Static_Batch_Size`| Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). |`50M`|
38
-
|`File_Cache_Advise`| Set the `posix_fadvise` in `POSIX_FADV_DONTNEED` mode. This reduces the usage of the kernel file cache. This option is ignored if not running on Linux. |`On`|
39
-
|`Threaded`| Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). |`false`|
13
+
|`buffer_chunk_size`| Set the initial buffer size to read file data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. |`32k`|
14
+
|`buffer_max_size`| Set the limit of the buffer size per monitored file. When a buffer needs to be increased, this value is used to restrict the memory buffer growth. If reading a file exceeds this limit, the file is removed from the monitored file list. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. |`32k`|
15
+
|`path`| Pattern specifying a specific log file or multiple ones through the use of common wildcards. Allows multiple patterns separated by commas. |_none_|
16
+
|`path_key`| If enabled, it appends the name of the monitored file as part of the record. The value assigned becomes the key in the map. |_none_|
17
+
|`exclude_path`| Set one or multiple shell patterns separated by commas to exclude files matching certain criteria, For example, `exclude_path *.gz,*.zip`. |_none_|
18
+
|`offset_key`| If enabled, Fluent Bit appends the offset of the current monitored file as part of the record. The value assigned becomes the key in the map. |_none_|
19
+
|`read_from_head`| For new discovered files on start (without a database offset/position), read the content from the head of the file, not tail. |`false`|
20
+
|`refresh_interval`| The interval of refreshing the list of watched files in seconds. |`60`|
21
+
|`rotate_wait`| Specify the number of extra time in seconds to monitor a file once is rotated in case some pending data is flushed. |`5`|
22
+
|`ignore_older`| Ignores files older than `ignore_older`. Supports `m`, `h`, `d` (minutes, hours, days) syntax. | Read all. |
23
+
|`skip_long_lines`| When a monitored file reaches its buffer capacity due to a very long line (`buffer_max_size`), the default behavior is to stop monitoring that file. `skip_long_lines` alter that behavior and instruct Fluent Bit to skip long lines and continue processing other lines that fit into the buffer size. |`off`|
24
+
|`skip_empty_lines`| Skips empty lines in the log file from any further processing or output. |`off`|
25
+
|`db`| Specify the database file to keep track of monitored files and offsets. |_none_|
26
+
|`db.sync`| Set a default synchronization (I/O) method. This flag affects how the internal SQLite engine do synchronization to disk, for more details about each option see [the SQLite documentation](https://www.sqlite.org/pragma.html#pragma_synchronous). Most scenarios will be fine with `normal` mode. If you need full synchronization after every write operation set `full` mode. `full` has a high I/O performance cost. Values: `extra`, `full`, `normal`, `off`. |`normal`|
27
+
|`db.locking`| Specify that the database will be accessed only by Fluent Bit. Enabling this feature helps increase performance when accessing the database but restricts externals tool from querying the content. |`false`|
28
+
|`db.journal_mode`| Sets the journal mode for databases (`wal`). Enabling `wal` provides higher performance. `wal` isn't compatible with shared network file systems. |`wal`|
29
+
|`db.compare_filename`| This option determines whether to review both `inode` and `filename` when retrieving stored file information from the database. `true` verifies both `inode` and `filename`, while `false` checks only the `inode`. To review the `inode` and `filename` in the database, refer [see `keep_state`](#tailing-files-keeping-state). |`false`|
30
+
|`mem_buf_limit`| Set a memory limit that Tail plugin can use when appending data to the engine. If the limit is reached, it will be paused. When the data is flushed it resumes. |_none_|
31
+
|`exit_on_eof`| When reading a file will exit as soon as it reach the end of the file. Used for bulk load and tests. |`false`|
32
+
|`parser`| Specify the name of a parser to interpret the entry as a structured message. |_none_|
33
+
|`key`| When a message is unstructured (no parser applied), it's appended as a string under the key name `log`. This option lets you define an alternative name for that key. |`log`|
34
+
|`inotify_watcher`| Set to `false` to use file stat watcher instead of `inotify`. |`true`|
35
+
|`tag`| Set a tag with `regexextract` fields that will be placed on lines read. For example, `kube.<namespace_name>.<pod_name>.<container_name>.<container_id>`. Tag expansion is supported: if the tag includes an asterisk (`*`), that asterisk will be replaced with the absolute path of the monitored file, with slashes replaced by dots. See [Workflow of Tail + Kubernetes Filter](../filters/kubernetes.md#workflow-of-tail--kubernetes-filter). |_none_|
36
+
|`tag_regex`| Set a regular expression to extract fields from the filename. For example: `(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64})\.log$`. |_none_|
37
+
|`static_batch_size`| Set the maximum number of bytes to process per iteration for the monitored static files (files that already exist upon Fluent Bit start). |`50M`|
38
+
|`file_cache_advise`| Set the `posix_fadvise` in `POSIX_FADV_DONTNEED` mode. This reduces the usage of the kernel file cache. This option is ignored if not running on Linux. |`on`|
39
+
|`threaded`| Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). |`false`|
40
40
41
41
## Buffers and memory management
42
42
@@ -131,7 +131,7 @@ Fluent Bit 1.8 and later supports multiline core capabilities for the Tail input
131
131
- Multiline Core
132
132
- Old Multiline
133
133
134
-
### Multiline Core (v1.8)
134
+
### Multiline Core
135
135
136
136
The new multiline core is exposed by the following configuration:
137
137
@@ -148,7 +148,7 @@ The new multiline core is exposed by the following configuration:
148
148
-`multiline_flush`
149
149
-`docker_mode`
150
150
151
-
### Multiline and containers (v1.8)
151
+
### Multiline and containers
152
152
153
153
If you are running Fluent Bit to process logs coming from containers like Docker or CRI, you can use the built-in modes. This helps reassemble multiline messages originally split by Docker or CRI:
154
154
@@ -188,20 +188,20 @@ For the old multiline configuration, the following options exist to configure th
188
188
189
189
| Key | Description | Default |
190
190
| :--- | :--- | :--- |
191
-
|`Multiline`| If enabled, the plugin will try to discover multiline messages and use the proper parsers to compose the outgoing messages. When this option is enabled the Parser option isn't used. |`Off`|
192
-
|`Multiline_Flush`| Wait period time in seconds to process queued multiline messages. |`4`|
193
-
|`Parser_Firstline`| Name of the parser that matches the beginning of a multiline message. The regular expression defined in the parser must include a group name (named `capture`), and the value of the last match group must be a string. |_none_|
194
-
|`Parser_N`| Optional. Extra parser to interpret and structure multiline entries. This option can be used to define multiple parsers. For example, `Parser_1 ab1`, `Parser_2 ab2`, `Parser_N abN`. |_none_|
191
+
|`multiline`| If enabled, the plugin will try to discover multiline messages and use the proper parsers to compose the outgoing messages. When this option is enabled the Parser option isn't used. |`off`|
192
+
|`multiline_flush`| Wait period time in seconds to process queued multiline messages. |`4`|
193
+
|`parser_firstline`| Name of the parser that matches the beginning of a multiline message. The regular expression defined in the parser must include a group name (named `capture`), and the value of the last match group must be a string. |_none_|
194
+
|`parser_N`| Optional. Extra parser to interpret and structure multiline entries. This option can be used to define multiple parsers. For example, `parser_1 ab1`, `parser_2 ab2`, `parser_N abN`. |_none_|
195
195
196
196
### Old Docker mode configuration parameters
197
197
198
198
Docker mode exists to recombine JSON log lines split by the Docker daemon due to its line length limit. To use this feature, configure the tail plugin with the corresponding parser and then enable Docker mode:
199
199
200
200
| Key | Description | Default |
201
201
| :--- | :--- | :--- |
202
-
|`Docker_Mode`| If enabled, the plugin will recombine split Docker log lines before passing them to any parser. This mode can't be used at the same time as Multiline. |`Off`|
203
-
|`Docker_Mode_Flush`| Wait period time in seconds to flush queued unfinished split lines. |`4`|
204
-
|`Docker_Mode_Parser`| Specify an optional parser for the first line of the Docker multiline mode. The parser name to be specified must be registered in the `parsers.conf` file. |_none_|
202
+
|`docker_mode`| If enabled, the plugin will recombine split Docker log lines before passing them to any parser. This mode can't be used at the same time as Multiline. |`Off`|
203
+
|`docker_mode_flush`| Wait period time in seconds to flush queued unfinished split lines. |`4`|
204
+
|`docker_mode_parser`| Specify an optional parser for the first line of the Docker multiline mode. The parser name to be specified must be registered in the `parsers.conf` file. |_none_|
205
205
206
206
## Get started
207
207
@@ -264,7 +264,7 @@ Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something
264
264
at com.myproject.module.MyProject.main(MyProject.java:6)
265
265
```
266
266
267
-
Specify a `Parser_Firstline` parameter that matches the first line of a multiline event. When a match is made, Fluent Bit reads all future lines until another match with `Parser_Firstline` is made.
267
+
Specify a `parser_firstline` parameter that matches the first line of a multiline event. When a match is made, Fluent Bit reads all future lines until another match with `parser_firstline` is made.
268
268
269
269
In this case you can use the following parser, which extracts the time as `time` and the remaining portion of the multiline as `log`.
0 commit comments