Removing lables from record for Loki

[mmohamed]

Describe the bug:
Unable to remove labels from records before sending to Loki

Expected behaviour:
Able to drop label (try with remove_keys on Output and Flow level)

Steps to reproduce the bug:

• Install last release (6.1.0) and configure a Loki as output following this example : https://kube-logging.dev/docs/examples/loki-nginx/
• All kubernetes labels are sent and unable to drop/delete some labels

Additional context:
NA

Environment details:

• Kubernetes version (e.g. v1.15.2): v1.29.4
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc): OnPremise
• logging-operator version (e.g. 2.1.1): 6.1.0
• Install method (e.g. helm or static manifests): Helm
• Logs from the misbehaving component (and any other relevant logs): ---
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: stream-line-log-flow
spec:
  filters:
  - tag_normaliser: {}
  - record_modifier:
      records:
      - origin: stream-log-line
      remove_keys: label2
  - record_transformer:
      remove_keys: label3
  localOutputRefs:
  - logging
  match:
  - select:
      labels:
        type: stream-log-line

---

apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: logging
spec:
  loki:
    buffer:
      timekey: 1m
      timekey_use_utc: true
      timekey_wait: 15s
    configure_kubernetes_labels: true
    remove_keys: ["label1"]
    password:
      value: xxxxxx
    tenant: xxxxxx
    url: https://xxxxxxxx
    username:
      value: xxxxxx

/kind bug

[mmohamed]

Finally, found by my self :)

With configure_kubernetes_labels: true, you need to drop keys (orignal kubernetes labels) before extraction and transformation to loki labels in the output configuration.
Kubernetes labels are an associative array in kuberenetes key of record, to drop one or more is like :

remove_keys: $.kubernetes.labels.label2
#or
remove_keys: "$['kubernetes']['labels']['label2']"

For spectial chars, one way :

remove_keys: "$['kubernetes']['labels']['app.kubernetes.io/managed-by']"

In the end, my flow will be:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: stream-line-log-flow
spec:
  filters:
  - tag_normaliser: {}
  - record_modifier:
      records:
      - origin: stream-log-line
      remove_keys: "$['kubernetes']['labels']['app.kubernetes.io/managed-by']"
  - record_transformer:
      remove_keys: $.kubernetes.labels.label3
  localOutputRefs:
  - logging-prd
  match:
  - select:
      labels:
        type: stream-log-line

@references: https://docs.fluentd.org/plugin-helper-overview/api-plugin-helper-record_accessor

Best,