Support multiline config values, for things like extended regular expressions

[hlein]

Is your feature request related to a problem? Please describe.

For example, complex regexes get painful to maintain; for exceedingly complex ones they are write-only. Being able to use extended regexes with embedded whitespace ((?x) or perl's /x modifier) makes a huge difference, allowing each token to be on its own line, embedded comments, diffs are more granular, etc.

Describe the solution you'd like

I think all(?) this needs is some way to do multi-line values in config file statements, which as far as I can see is not currently supported.

Onigmo used by fluent-bit supports ONIG_OPTION_EXTEND, (?x), etc. A one-line Regex that uses (?x) with embedded whitespace does seem to behave correctly, it's just without being able to make the value multi-line, most of the benefit is lost.

Describe alternatives you've considered

For now, I'm going to experiment with a preprocessor to glue multiline regexes together. Squashing newlines to spaces should work fine based on my limited (?x) tests so far.

But it would be great if the config parser could handle this natively.

Maybe some generic support for \ at end-of-line being a line-wrap indicator?

Maybe an Include directive that is valid in the value portion of a key value pair, and reads the whole thing in as the value?

Maybe special key names that indicate a multline value, where say, each line indented as much or more than the beginning of the key-name's first value is considered part of the same value? (This feels both the most special-case and the most prone to surprising failures.)

(As far as I know none of those are currently supported, I'd love to be wrong.)

[patrick-stephens]

The yaml configuration should support multiline, you can configure multiline lua that way for example.

[hlein]

The yaml configuration should support multiline, you can configure multiline lua that way for example.

You are correct! And it looks like providing a .yaml version of a .conf file Just Works? So for example this works:

$ cat parsers-test.yaml 
parsers:
- name: syslog-test
  format: regex
  regex: '(?x)
          ^
           (?:\<(?<pri>[0-9]+)\>)?
           (?<time>[^\s]*\s{1,2} [^\s]*\s [^\s]*)\s
           (?<host>[^\s]*)\s
           (?<ident>[a-zA-Z0-9_\/\.\-]*)
           (?:\[(?<pid>[0-9]+)\])?
           (?:[^\:]*\:)?
           \s*
           (?<message>.*)
          $'
  time_key: time
  time_format: '%b %d %H:%M:%S'
  time_keep: off

And:

$ echo "Nov  9 03:10:02 foo syslogd[123]: 1.5.1: restart (remote reception)." | \
    fluent-bit -q -R parsers-test.yaml -i stdin -p parser=syslog-test -o stdout -p format=json_lines
{"date":1762657802.0,"host":"foo","ident":"syslogd","pid":"123","message":"1.5.1: restart (remote reception)."}

...So I think I can just use parsers.yaml; may submit some PRs to convert a couple in conf/parser*.conf followed by revisions to them.

[hlein]

Ah, for the sake of future readers, better to use:

  regex: |
    (?x)
    ^
    ...

...Because that way, # comments are handled correctly - I guess the multiline quoted string gets squashed into a single line before being handed off to the regex library, but | makes it Do What I Mean.

[patrick-stephens]

Yeah it's yaml so be ready to fight those syntax errors! It uses libyaml to do the underlying work.

There's also some restrictions, e.g. no support for anchors.

Be good to update the docs as well with good examples if you have them?

[hlein]

Submitted a PR for my WIP, #11168

I'd be happy to work on documentation and Dockerfile updates, but - looks like FLB_CONFIG_YAML is optional, so we can't count on it being available, therefore can't just replace existing parser*.conf file use.

Having both .conf and .yaml versions of the same things in-tree long-term sounds error-prone. Maybe just conf/parsers.conf needs to be preserved, and other conf/parser*.conf files could be retired? Maybe if the docker images - which are probably how most users actually deploy fluent-bit - are guaranteed to have libyaml, then maintaining backwards-compat in the default configs is not as big a concern?

[hlein]

Having both .conf and .yaml versions of the same things in-tree long-term sounds error-prone. Maybe just conf/parsers.conf needs to be preserved, and other conf/parser*.conf files could be retired? Maybe if the docker images - which are probably how most users actually deploy fluent-bit - are guaranteed to have libyaml, then maintaining backwards-compat in the default configs is not as big a concern?

I've been finding and fixing issues in parsers.yaml post-conversion, but the same issues are present in parsers.conf (see hlein#2 , which I will retarget against upstream once the earlier #11168 lands).

So that brings up the question of what should the long-term plan for parsers.conf be? Do the fixes that are easy to see and verify in multiline .yaml need to be propagated back to it? Or could we potentially gut parsers.conf, just not remove it completely for the sake of things that assume it exists?

[patrick-stephens]

I think the intention is to move to YAML configuration, ideally we backport if it's simple but if not I'd say it's fine to go with just YAML.

[hlein]

I think the intention is to move to YAML configuration, ideally we backport if it's simple but if not I'd say it's fine to go with just YAML.

OK cool.

I will convert any references to parser*.conf to parser*.yaml wherever it seems straightforward, and see which .conf files are then ready for removal. FLB_CONFIG_YAML is on by default, and I don't see any of the builds explicitly disabling it. If someone builds w/o it, that's on them.

[hlein]

I will convert any references to parser*.conf to parser*.yaml wherever it seems straightforward, and see which .conf files are then ready for removal.

Done, #11168 updated