diff --git a/content/en/logs/log_configuration/parsing.md b/content/en/logs/log_configuration/parsing.md index 672a99e05107c..a4ac6c8d9072c 100644 --- a/content/en/logs/log_configuration/parsing.md +++ b/content/en/logs/log_configuration/parsing.md @@ -57,7 +57,12 @@ MyParsingRule %{word:user} connected on %{date("MM/dd/yyyy"):date} After processing, the following structured log is generated: -{{< img src="logs/processing/processors/_parser.png" alt="Parsing example 1" style="width:80%;">}} +```json +{ + "users": "john", + "date": 1575590400000 +} +``` **Note**: @@ -80,7 +85,7 @@ Here is a list of all the matchers and filters natively implemented by Datadog: {{< tabs >}} {{% tab "Matchers" %}} -`date("pattern"[, "timezoneId"[, "localeId"]])` +`date("pattern"[, "timezoneId"[, "localeId"]])` : Matches a date with the specified pattern and parses to produce a Unix timestamp. [See the date Matcher examples](#parsing-dates). `regex("pattern")` @@ -216,21 +221,19 @@ Here is a list of all the matchers and filters natively implemented by Datadog: ## Advanced settings -At the bottom of your Grok processor tiles, there is an **Advanced Settings** section: - -{{< img src="logs/processing/parsing/advanced_settings.png" alt="Advanced Settings" style="width:80%;">}} +Use the **Advanced Settings** section at the bottom of your Grok processor to parse a specific attribute instead of the default `message` attribute, or to define helper rules that reuse common patterns across multiple parsing rules. ### Parsing a specific text attribute Use the **Extract from** field to apply your Grok processor on a given text attribute instead of the default `message` attribute. -For example, consider a log containing a `command.line` attribute that should be parsed as a key-value. You could parse this log as follows: +For example, consider a log containing a `command.line` attribute that should be parsed as a key-value. Extract from `command.line` to parse its contents and create structured attributes from the command data. -{{< img src="logs/processing/parsing/parsing_attribute.png" alt="Parsing Command Line" style="width:80%;">}} +{{< img src="/logs/processing/parsing/grok_advanced_settings_extract.png" alt="Advanced Settings with Extract from command.line attribute example" style="width:80%;">}} -### Using helper rules to factorize multiple parsing rules +### Using helper rules to reuse common patterns -Use the **Helper Rules** field to define tokens for your parsing rules. Helper rules help you to factorize Grok patterns across your parsing rules. This is useful when you have several rules in the same Grok parser that use the same tokens. +Use the **Helper Rules** field to define tokens for your parsing rules. Helper rules let you reuse common Grok patterns across your parsing rules. This is useful when you have several rules in the same Grok parser that use the same tokens. Example for a classic unstructured log: @@ -252,8 +255,6 @@ connection connected on %{date("MM/dd/yyyy"):connect_date} server on server %{notSpace:server.name} in %{notSpace:server.env} ``` -{{< img src="logs/processing/parsing/helper_rules.png" alt="helper rules" style="width:80%;">}} - ## Examples Some examples demonstrating how to use parsers: @@ -292,12 +293,18 @@ user=john connect_date=11/08/2017 id=123 action=click rule %{data::keyvalue} ``` -{{< img src="logs/processing/parsing/parsing_example_2.png" alt="Parsing example 2" style="width:80%;">}} - You don't need to specify the name of your parameters as they are already contained in the log. If you add an **extract** attribute `my_attribute` in your rule pattern you will see: -{{< img src="logs/processing/parsing/parsing_example_2_bis.png" alt="Parsing example 2 bis" style="width:80%;">}} +```json +{ + "my_attribute": { + "user": "john", + "id": 123, + "action": "click" + } +} +``` If `=` is not the default separator between your key and values, add a parameter in your parsing rule with a separator. @@ -313,8 +320,6 @@ user: john connect_date: 11/08/2017 id: 123 action: click rule %{data::keyvalue(": ")} ``` -{{< img src="logs/processing/parsing/key_value_parser.png" alt="Key value parser" style="width:80%;" >}} - If logs contain special characters in an attribute value, such as `/` in a url for instance, add it to the allowlist in the parsing rule: **Log:** @@ -329,8 +334,6 @@ url=https://app.datadoghq.com/event/stream user=john rule %{data::keyvalue("=","/:")} ``` -{{< img src="logs/processing/parsing/key_value_allowlist.png" alt="Key value allowlist" style="width:80%;" >}} - Other examples: | **Raw string** | **Parsing rule** | **Result** | diff --git a/content/en/logs/log_configuration/processors.md b/content/en/logs/log_configuration/processors.md index 134d672d83ca2..1981c42b4eccb 100644 --- a/content/en/logs/log_configuration/processors.md +++ b/content/en/logs/log_configuration/processors.md @@ -35,20 +35,25 @@ In [log configuration settings][1], you can configure processors such as the [Gr ## Grok parser -Create custom grok rules to parse the full message or a specific attribute of your raw event. For more information, see the [parsing section][2]. As a best practice, it is recommended to use at most 10 parsing rules within a grok processor. - -{{< tabs >}} -{{% tab "UI" %}} - -Define the Grok processor on the [**Pipelines** page][1]: +Create custom grok rules to parse the full message or a specific attribute of your raw event. As a best practice, limit your grok parser to 10 parsing rules. For more information on Grok syntax and parsing rules, see [Parsing][10]. {{< img src="logs/log_configuration/processor/grok_parser.png" alt="Grok Parser" style="width:80%;" >}} -Click **Parse my logs** to kickstart a set of three parsing rules for the logs flowing through the underlying pipeline. Refine attribute naming from there, and add new rules for other type of logs if needed. This feature requires that the corresponding logs are being indexed, and actually flowing in—you can temporarily deactivate or sample down exclusion filters to make this work for you. +{{< tabs >}} +{{% tab "UI" %}} -Select a sample by clicking on it to trigger its evaluation against the parsing rule and display the result at the bottom of the screen. +Define the Grok processor on the [**Pipelines** page][1]. To configure Grok parsing rules: -Up to five samples can be saved with the processor, and each sample can be up to 5000 characters in length. All samples show a status (`match` or `no match`), which highlights if one of the parsing rules of the grok parser matches the sample. +1. Click **Parse my logs** to automatically generate a set of three parsing rules based on the logs flowing through the pipeline. + **Note**: This feature requires that the corresponding logs are indexed and actively flowing in. You can temporarily deactivate or sample down exclusion filters to allow the feature to detect logs. +1. **Log Samples**: Add up to five sample logs (up to 5000 characters each) to test your parsing rules. +1. **Define parsing rules**: Write your parsing rules in the rule editor. As you define rules, the Grok parser provides syntax assistance: + - **Matcher suggestions**: Type a rule name followed by `%{`. A dropdown appears with available matchers (such as `word`, `integer`, `ip`, `date`). Select a matcher from the list to insert it into your rule.
+ ``` + MyParsingRule %{ + ``` + - **Filter suggestions**: When adding a filter with `:`, a dropdown shows compatible filters for the selected matcher. +1. **Test your rules**: Select a sample by clicking on it to trigger its evaluation against the parsing rule and display the result at the bottom of the screen. All samples show a status (`match` or `no match`), which highlights if one of the parsing rules of the grok parser matches the sample. [1]: https://app.datadoghq.com/logs/pipelines {{% /tab %}} @@ -1047,7 +1052,7 @@ The Decoder processor translates binary-to-text encoded string fields (such as B 2. Select the source encoding: Choose the binary-to-text encoding of the source: `base64` or `base16/hex`. 2. For `Base16/Hex`: Choose the output format: `string (UTF-8)` or `integer`. 3. Set the target attribute: Enter the attribute path to store the decoded result. - + {{< img src="logs/log_configuration/processor/decoder-processor.png" alt="Decoder processor - Append" style="width:80%;" >}} {{% /tab %}} @@ -1075,3 +1080,4 @@ For more information, see [Threat Intelligence][9]. [7]: /integrations/guide/reference-tables/ [8]: /tracing/other_telemetry/connect_logs_and_traces/ [9]: /security/threat_intelligence/ +[10]: /logs/log_configuration/parsing/?tab=matchers diff --git a/static/images/logs/processing/parsing/grok_advanced_settings_extract.png b/static/images/logs/processing/parsing/grok_advanced_settings_extract.png new file mode 100644 index 0000000000000..ce07ef832bcc6 Binary files /dev/null and b/static/images/logs/processing/parsing/grok_advanced_settings_extract.png differ