Added new parameter: FILEBEAT_COMPRESSION_LEVEL

#143

Author: Chris Wiechmann

CHANGELOG.md

@@ -7,6 +7,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 ## [Unreleased] 
 ### Added
 - Traffic-Monitor Authorization should support Multi-Organization [#141](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/141)
+- New optional parameter: `FILEBEAT_COMPRESSION_LEVEL` for Filebeat to Logstash compression rate [#143](https://github.com/Axway-API-Management-Plus/apigateway-openlogging-elk/issues/143)
 
 ## [3.4.0] 2021-09-02
 ### Fixed

UPDATE.md

@@ -56,7 +56,7 @@ On the other hand, the API builder Docker image, as a central component of the s
 | 3.3.1 | [X](#api-builderlogstashmemcached) | -                                  | -                                  | -             | -               | -               | [X](#parameters)|-                   | 7.12.1  |            |
 | 3.3.2 | [X](#api-builderlogstashmemcached) | -                                  | -                                  | -             | -               | -               | -               |-                   | 7.12.1  |            |
 | 3.4.0 | [X](#api-builderlogstashmemcached) | [X](#api-builderlogstashmemcached) | -                                  | -             | -               | [X](#dashboards)| [X](#parameters)|[X](#elastic-config)| 7.14.0  |            |
-| 3.5.0 | [X](#api-builderlogstashmemcached) | -                                  | -                                  | -             | -               | -               | -               |-                   | 7.14.0  |Unreleased  |
+| 3.5.0 | [X](#api-builderlogstashmemcached) | -                                  | -                                  | -             | -               | -               | [X](#parameters)|-                   | 7.14.0  |Unreleased  |
 
 ### Update from Version 1.0.0
 

env-sample

@@ -378,6 +378,14 @@ ELASTIC_VERSION=7.14.0
 # Defaults to false.
 # DISABLE_SETUP_FLOWS=true
 
+# ----------------------------------------------------------------------------------------------
+# The gzip compression level. Setting this value to 0 disables compression. The compression level 
+# must be in the range of 1 (best speed) to 9 (best compression).
+# Increasing the compression level will reduce the network usage but will increase the CPU usage.
+# Used-By: Filebeat
+# Defaults to 3
+# FILEBEAT_COMPRESSION_LEVEL=6
+
 ################################################################################################
 #                         Metricbeat
 ################################################################################################

filebeat/docker-compose.filebeat.yml

@@ -22,6 +22,7 @@ services:
       - GATEWAY_REGION=${GATEWAY_REGION}
       - FILEBEAT_WORKER=${FILEBEAT_WORKER}
       - SELF_MONITORING_ENABLED=${SELF_MONITORING_ENABLED}
+      - FILEBEAT_COMPRESSION_LEVEL=${FILEBEAT_COMPRESSION_LEVEL:3}
     ports:
       - 5066:5066
     volumes:

filebeat/filebeat.yml

@@ -79,6 +79,7 @@ output.logstash:
   hosts: ${LOGSTASH_HOSTS}
   worker: ${FILEBEAT_WORKERS:2}
   bulk_max_size: ${FILEBEAT_BULK_MAX_SIZE:3072}
+  compression_level: ${FILEBEAT_COMPRESSION_LEVEL:3}
   loadbalance: true
   # TTL is only required for a Kubernetes deployment
   # ttl: 2m

helm/templates/elasticApimFilebeat/filebeat-config.yaml

@@ -14,5 +14,7 @@ data:
   ELASTICSEARCH_CLUSTER_UUID: {{ .Values.filebeat.elasticsearchClusterUUID | quote }}
   ELASTICSEARCH_CRT: {{ required "The path to the CA for Logstash to Elasticsearch communication is missing" .Values.global.elasticsearchCa | quote }}
   FILEBEAT_WORKER: {{ .Values.filebeat.filebeatWorker | quote }}
+  FILEBEAT_COMPRESSION_LEVEL: {{ .Values.filebeat.compressionLevel | quote }}
   SELF_MONITORING_ENABLED: {{ .Values.global.selfMonitoringEnabled | quote }}
+  
 {{- end }}

helm/values.yaml

@@ -495,6 +495,10 @@ filebeat:
   elasticsearchClusterUUID: ""
   # The number of Filebeat worker threads sending data to Logstash. See parameter: FILEBEAT_WORKER in env-sample
   filebeatWorker: 2
+  # The gzip compression level. Setting this value to 0 disables compression. The compression level must be in the 
+  # range of 1 (best speed) to 9 (best compression). Increasing the compression level will reduce the network usage but will 
+  # increase the CPU usage.
+  compressionLevel: 3
   # Injects the environment variables from the ConfigMaps and Secrets into the 
   # Filebeat container. Specify your own ConfigMaps or Secrets if you don't
   # provide Configuration and Secrets as part of this values.yaml.
@@ -644,6 +648,7 @@ filebeat:
         hosts: ${LOGSTASH_HOSTS}
         worker: ${FILEBEAT_WORKERS:2}
         bulk_max_size: ${FILEBEAT_BULK_MAX_SIZE:3072}
+        compression_level: ${FILEBEAT_COMPRESSION_LEVEL:3}
         loadbalance: true
         ttl: 2m
         # Required to make TTL working