refactor(logstash): drop PKCS#8 filename shim, use native PKCS#8 generation

CHANGELOG.md

@@ -20,6 +20,13 @@ First release of the `oddly.elasticstack` collection, forked from
 - Certificate expiry warnings and tag-driven renewal (`--tags certificates`,
   `--tags renew_es_cert`, `--tags renew_ca`).
 - Logstash standalone certificate mode for independent deployments.
+- Logstash role writes the unencrypted PKCS#8 PEM key directly to
+  `<hostname>.key` (the `-pkcs8.key` filename is gone). Standalone mode now
+  uses `community.crypto.openssl_privatekey` (PKCS#8 native) instead of
+  `openssl genrsa` + a separate `openssl pkcs8` conversion step. The new
+  `logstash_tls_copy_certs` variable lets external mode reference cert files
+  in place instead of copying them — useful with certmonger or cert-manager
+  where the renewal tool rotates the files and restarts Logstash out of band.
 - Beats Filebeat `filestream` input type for 9.x (replacing deprecated `log`).
 - Logstash `elastic_agent` input plugin support.
 - Elasticsearch `cluster_settings` for runtime cluster configuration via API.

docs/guide/tls.md

@@ -13,7 +13,7 @@ graph TD
     CA -->|"P12 keystore<br/>per node"| ES_T["ES Transport :9300<br/>Node-to-node encryption"]
     CA -->|"P12 keystore<br/>per node"| ES_H["ES HTTP :9200<br/>Client-to-node encryption"]
     CA -->|"P12 keystore"| KB["Kibana<br/>ES connection + optional HTTPS"]
-    CA -->|"PEM cert + PKCS8 key<br/>+ P12 keystore"| LS["Logstash<br/>Beats input + ES output"]
+    CA -->|"PEM cert + key<br/>+ P12 keystore"| LS["Logstash<br/>Beats input + ES output"]
     CA -->|"PEM cert + key<br/>per host"| BT["Beats<br/>Logstash/ES output"]
 
     ES_T -.->|"mutual TLS"| ES_T
@@ -41,7 +41,7 @@ The collection uses different formats depending on what each service expects nat
 | Elasticsearch (HTTP) | PKCS12 | `<hostname>-http.p12` |
 | Kibana | PKCS12 | `<hostname>-kibana.p12` |
 | Logstash (ES output) | PKCS12 | `keystore.pfx` |
-| Logstash (Beats input) | PEM | `<hostname>.crt` + `<hostname>-pkcs8.key` |
+| Logstash (Beats input) | PEM | `<hostname>-server.crt` + `<hostname>.key` |
 | Beats | PEM | `<hostname>-beats.crt` + `<hostname>-beats.key` |
 
 When using external certificates, both PEM (`.crt`, `.pem`) and PKCS12 (`.p12`, `.pfx`) are accepted. Format is auto-detected by probing the file content with `openssl`, not from the file extension.
@@ -177,6 +177,36 @@ You don't always need to set every variable. The roles apply sensible defaults:
 
 By default (`*_tls_remote_src: false`), certificate files are on the Ansible controller and get copied to each managed node. Set `*_tls_remote_src: true` when files are already on the managed nodes — provisioned by certbot, cloud-init, Vault agent, or a configuration management tool.
 
+### Logstash with certmonger / cert-manager (hands-off rotation)
+
+For Logstash specifically, certificate copies can get out of sync with automatic renewals. When certmonger or cert-manager rotates the cert, the copy under `/etc/logstash/certs/` becomes stale until the next Ansible run.
+
+Set `logstash_tls_copy_certs: false` to skip the copy. The pipeline config then references the original paths directly, and Logstash picks up the new cert on the next restart — which the renewal tool can trigger itself.
+
+```yaml title="group_vars/all.yml"
+logstash_cert_source: external
+logstash_tls_copy_certs: false
+logstash_tls_certificate_file: /etc/pki/logstash/server.crt
+logstash_tls_key_file: /etc/pki/logstash/server.key
+logstash_tls_ca_file: /etc/pki/logstash/ca.crt
+```
+
+Logstash runs as the `logstash` user and must be able to read the key. For certmonger, request the cert with the right ownership and restart hook:
+
+```bash
+getcert request \
+  -f /etc/pki/logstash/server.crt -k /etc/pki/logstash/server.key \
+  -o root:logstash -m 0640 \
+  -O root:logstash -M 0644 \
+  -C 'systemctl try-reload-or-restart logstash' \
+  -c local -I logstash
+```
+
+The `-o root:logstash -m 0640` ensures Logstash can read the key; certmonger preserves this ownership across every renewal. The `-C` post-save hook replaces the Ansible re-run. Verified on Debian with certmonger 0.79 and tested on both certmonger and openssl-generated keys, which both produce PKCS#8 PEM directly.
+
+!!! note
+    This path does not generate `keystore.pfx`. If you need the Elasticsearch output's P12 keystore (`logstash_output_elasticsearch: true` with security), either leave `logstash_tls_copy_certs: true` or set `logstash_output_elasticsearch: false` and manage the ES output yourself.
+
 ## Mode 3: Inline PEM content from a secrets manager
 
 When certificates come from HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or any system that provides certificate content rather than files, use the `*_content` variables. They take precedence over file paths.
@@ -335,6 +365,7 @@ beats_cert_source: elasticsearch_ca   # or external
 | | `logstash_input_beats_ssl` | inherited | TLS on Beats input |
 | | `logstash_tls_certificate_file` | `""` | Certificate path (external) |
 | | `logstash_tls_remote_src` | `false` | Certs on managed node |
+| | `logstash_tls_copy_certs` | `true` | Copy external certs into `logstash_certs_dir`; set `false` for hands-off rotation |
 | | `logstash_cert_force_regenerate` | `false` | Force cert regen |
 | **Beats** | `beats_cert_source` | `elasticsearch_ca` | `elasticsearch_ca` or `external` |
 | | `beats_security` | `false` | Enable TLS (opt-in) |

docs/reference/logstash.md

@@ -404,6 +404,7 @@ logstash_cert_force_regenerate: false
 # logstash_tls_key_file: "/path/to/server.key"
 # logstash_tls_ca_file: "/path/to/ca.crt"
 # logstash_tls_remote_src: false
+# logstash_tls_copy_certs: true   # set to false for hands-off rotation (certmonger, cert-manager)
 ```
 
 `logstash_cert_source`
@@ -414,7 +415,7 @@ logstash_cert_force_regenerate: false
     - **`external`** — uses certificate files you provide via `logstash_tls_certificate_file`, `logstash_tls_key_file`, and optionally `logstash_tls_ca_file`. The role copies them into place but does NOT create the ES user/role (assumes you manage that separately).
 
 `logstash_certs_dir`
-:   Directory on the Logstash host where TLS certificates, keys, and the CA bundle are stored. The role creates this directory and writes the PEM certificate, PKCS8 key, P12 keystore, and CA certificate here.
+:   Directory on the Logstash host where TLS certificates, keys, and the CA bundle are stored. The role creates this directory and writes the PEM certificate, the unencrypted PKCS#8 PEM key, the P12 keystore for the Elasticsearch output, and the CA certificate here.
 
 `logstash_tls_key_passphrase`
 :   Passphrase used when generating the P12 keystore for the Elasticsearch output plugin. Also used as the `ssl_keystore_password` in the output config.
@@ -432,11 +433,14 @@ logstash_cert_force_regenerate: false
 :   Force TLS certificate regeneration on the next run, even if current certificates are still valid. Useful after a CA rotation or if you suspect a key compromise. The role resets this to `false` internally after regeneration.
 
 `logstash_tls_certificate_file` / `logstash_tls_key_file` / `logstash_tls_ca_file`
-:   Paths to externally-managed certificate files. Only used when `logstash_cert_source: external`. The role copies these into `logstash_certs_dir`.
+:   Paths to externally-managed certificate files. Only used when `logstash_cert_source: external`. The role copies these into `logstash_certs_dir` by default; set `logstash_tls_copy_certs: false` to reference them in place instead.
 
 `logstash_tls_remote_src`
 :   When `true`, the external certificate files are already on the remote host and are copied locally (no upload from the Ansible controller). Defaults to `false`.
 
+`logstash_tls_copy_certs`
+:   Whether to copy the external `logstash_tls_*_file` paths into `logstash_certs_dir`. Defaults to `true`, which preserves the existing copy-and-rename flow. Set to `false` for a hands-off setup where the pipeline config references the original paths directly — useful for certmonger, cert-manager, or any tool that rotates the key files out-of-band. The `logstash` user must be able to read the key; typical on-disk permissions are `root:logstash 0640`.
+
 ### Dead Letter Queue
 
 ```yaml
@@ -610,12 +614,18 @@ The standard pipeline uses three config files in `/etc/logstash/conf.d/main/`:
 
 Logstash loads `.conf` files alphabetically, so the numbering ensures correct execution order. When you set `logstash_custom_pipeline`, the role writes a single `pipeline.conf` and removes the three numbered files. Switching back from custom to standard mode removes `pipeline.conf`.
 
-### PKCS8 key requirement
+### Key format
+
+The Beats and Elastic Agent inputs need an unencrypted PKCS#8 PEM key (`-----BEGIN PRIVATE KEY-----`) per the upstream plugin contract. The Elasticsearch output uses a P12 keystore. The role produces both from the same certificate:
+
+- **`<hostname>.key`** — unencrypted PKCS#8 PEM, read by the Beats / Elastic Agent inputs
+- **`keystore.pfx`** — P12 keystore (cert + key), read by the Elasticsearch output plugin
 
-Logstash input plugins (Beats, Elastic Agent) require an unencrypted PKCS8 key, while the Elasticsearch output plugin uses a P12 keystore. The role generates both formats from the same certificate:
+Where the key comes from depends on the mode:
 
-- **P12 cert** — copied as `keystore.pfx` for the ES output plugin
-- **PEM cert** — extracted from a ZIP, with the encrypted key converted to unencrypted PKCS8 via `openssl pkcs8 -topk8 -nocrypt`
+- **`elasticsearch_ca`** — `elasticsearch-certutil --pem` emits an encrypted PKCS#1 key inside a zip. The role unpacks it and decrypts to PKCS#8 PEM in one `openssl pkcs8 -topk8 -nocrypt` step, writing directly to `<hostname>.key`.
+- **`standalone`** — generated with `community.crypto.openssl_privatekey`, which emits PKCS#8 PEM natively. No post-processing.
+- **`external`** — the file you supply at `logstash_tls_key_file` is used as-is. Most modern key generators (certmonger, `openssl genpkey`, `openssl genrsa` on OpenSSL 3.0+) emit PKCS#8 by default. If you have a legacy PKCS#1 key, convert it once with `openssl pkcs8 -topk8 -nocrypt`.
 
 ### ES 9.x vs 8.x SSL syntax
 
@@ -626,8 +636,8 @@ The Logstash input and output configuration templates use different SSL paramete
     ```
     # Input (Beats / Elastic Agent)
     ssl_enabled => true
-    ssl_certificate => "/etc/logstash/certs/..."
-    ssl_key => "/etc/logstash/certs/...-pkcs8.key"
+    ssl_certificate => "/etc/logstash/certs/<hostname>-server.crt"
+    ssl_key => "/etc/logstash/certs/<hostname>.key"
     ssl_client_authentication => required
     ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
 
@@ -643,8 +653,8 @@ The Logstash input and output configuration templates use different SSL paramete
     ```
     # Input (Beats / Elastic Agent)
     ssl => true
-    ssl_certificate => "/etc/logstash/certs/..."
-    ssl_key => "/etc/logstash/certs/...-pkcs8.key"
+    ssl_certificate => "/etc/logstash/certs/<hostname>-server.crt"
+    ssl_key => "/etc/logstash/certs/<hostname>.key"
     ssl_verify_mode => force_peer
     ssl_certificate_authorities => ["/etc/logstash/certs/ca.crt"]
 
@@ -655,7 +665,7 @@ The Logstash input and output configuration templates use different SSL paramete
     cacert => "/etc/logstash/certs/ca.crt"
     ```
 
-The template switches automatically based on `elasticstack_release | int >= 9`.
+The template switches automatically based on `elasticstack_release | int >= 9`. With `logstash_cert_source: external` and `logstash_tls_copy_certs: false`, the `ssl_certificate`, `ssl_key`, and `ssl_certificate_authorities` values point at the paths you supplied rather than `logstash_certs_dir`.
 
 ### Event enrichment (ident stamping)
 

molecule/logstash_ssl/converge.yml

@@ -18,7 +18,7 @@
     # External certificate configuration
     logstash_cert_source: external
     logstash_tls_certificate_file: /tmp/test-certs/server.crt
-    logstash_tls_key_file: /tmp/test-certs/server-pkcs8.key
+    logstash_tls_key_file: /tmp/test-certs/server.key
     logstash_tls_ca_file: /tmp/test-certs/ca.crt
     logstash_tls_remote_src: true
     logstash_extra_outputs: |

molecule/logstash_ssl/verify.yml

@@ -24,11 +24,11 @@
       register: server_cert
       failed_when: not server_cert.stat.exists
 
-    - name: Check PKCS8 key exists
+    - name: Check server key exists
       ansible.builtin.stat:
-        path: "/etc/logstash/certs/{{ inventory_hostname }}-pkcs8.key"
-      register: pkcs8_key
-      failed_when: not pkcs8_key.stat.exists
+        path: "/etc/logstash/certs/{{ inventory_hostname }}.key"
+      register: server_key
+      failed_when: not server_key.stat.exists
 
     - name: Check CA certificate exists
       ansible.builtin.stat:
@@ -40,10 +40,24 @@
       ansible.builtin.assert:
         that:
           - server_cert.stat.mode == '0640'
-          - pkcs8_key.stat.mode == '0640'
+          - server_key.stat.mode == '0640'
           - ca_cert.stat.mode == '0640'
         fail_msg: "Certificate file permissions are not correct (expected 0640)"
 
+    - name: Verify server key is unencrypted PKCS#8 PEM (not PKCS#1, not encrypted)
+      ansible.builtin.command: head -1 /etc/logstash/certs/{{ inventory_hostname }}.key
+      register: _server_key_header
+      changed_when: false
+
+    - name: Assert PKCS#8 header
+      ansible.builtin.assert:
+        that:
+          - _server_key_header.stdout == '-----BEGIN PRIVATE KEY-----'
+        fail_msg: >-
+          Expected unencrypted PKCS#8 PEM (-----BEGIN PRIVATE KEY-----),
+          got {{ _server_key_header.stdout }}. The beats/elastic_agent input
+          requires PKCS#8 per upstream Logstash docs.
+
     # --- Input verification (elastic_agent with SSL) ---
 
     - name: Read input configuration

molecule/logstash_standalone_certs/converge.yml

@@ -20,7 +20,7 @@
     # External certificate configuration
     logstash_cert_source: external
     logstash_tls_certificate_file: /tmp/test-certs/server.crt
-    logstash_tls_key_file: /tmp/test-certs/server-pkcs8.key
+    logstash_tls_key_file: /tmp/test-certs/server.key
     logstash_tls_ca_file: /tmp/test-certs/ca.crt
     logstash_tls_remote_src: true
     logstash_extra_outputs: |

molecule/logstash_standalone_certs/verify.yml

@@ -37,7 +37,7 @@
 
     - name: Check server key was copied
       ansible.builtin.stat:
-        path: "/etc/logstash/certs/{{ ansible_facts.hostname }}-pkcs8.key"
+        path: "/etc/logstash/certs/{{ ansible_facts.hostname }}.key"
       register: server_key
 
     - name: Assert server key exists
@@ -46,6 +46,19 @@
           - server_key.stat.exists
         fail_msg: "Server key not found in /etc/logstash/certs/"
 
+    - name: Verify server key is unencrypted PKCS#8 PEM
+      ansible.builtin.command: head -1 /etc/logstash/certs/{{ ansible_facts.hostname }}.key
+      register: _server_key_header
+      changed_when: false
+
+    - name: Assert PKCS#8 header
+      ansible.builtin.assert:
+        that:
+          - _server_key_header.stdout == '-----BEGIN PRIVATE KEY-----'
+        fail_msg: >-
+          Expected unencrypted PKCS#8 PEM (-----BEGIN PRIVATE KEY-----),
+          got {{ _server_key_header.stdout }}.
+
     - name: Check CA certificate was copied
       ansible.builtin.stat:
         path: /etc/logstash/certs/ca.crt

molecule/shared/generate_test_certs.yml

@@ -1,6 +1,6 @@
 ---
 # Generate a self-signed CA and server certificate for testing TLS.
-# Outputs to /tmp/test-certs/: ca.crt, server.crt, server-pkcs8.key
+# Outputs to /tmp/test-certs/: ca.crt, server.crt, server.key (PKCS#8 PEM)
 - name: Install openssl and cryptography for certificate generation
   ansible.builtin.package:
     name:
@@ -18,6 +18,7 @@
   community.crypto.openssl_privatekey:
     path: /tmp/test-certs/ca.key
     size: 2048
+    format: pkcs8
 
 - name: Generate CA CSR
   community.crypto.openssl_csr:
@@ -41,10 +42,12 @@
     provider: selfsigned
     selfsigned_not_after: "+365d"
 
-- name: Generate server private key
+- name: Generate server private key (PKCS#8 PEM, unencrypted)
   community.crypto.openssl_privatekey:
     path: /tmp/test-certs/server.key
     size: 2048
+    format: pkcs8
+    mode: "0640"
 
 - name: Generate server CSR
   community.crypto.openssl_csr:
@@ -64,16 +67,3 @@
     ownca_privatekey_path: /tmp/test-certs/ca.key
     provider: ownca
     ownca_not_after: "+365d"
-
-- name: Create unencrypted PKCS8 key
-  ansible.builtin.command: >-
-    openssl pkcs8 -topk8 -nocrypt
-    -in /tmp/test-certs/server.key
-    -out /tmp/test-certs/server-pkcs8.key
-  args:
-    creates: /tmp/test-certs/server-pkcs8.key
-
-- name: Set key permissions
-  ansible.builtin.file:
-    path: /tmp/test-certs/server-pkcs8.key
-    mode: "0640"

roles/logstash/defaults/main.yml

@@ -200,6 +200,17 @@ logstash_cert_source: elasticsearch_ca
 # logstash_tls_ca_file: "/path/to/ca.crt"
 # logstash_tls_remote_src: false  # Set true if cert files are on remote host
 
+# @var logstash_tls_copy_certs:description: >
+#   Copy externally-managed certificate files into `logstash_certs_dir` (default).
+#   Set to `false` to reference the `logstash_tls_*_file` paths directly from the
+#   pipeline config — Logstash reads them in place. Use this for certmonger,
+#   cert-manager, or any renewal system where the cert files update out-of-band
+#   and you don't want to re-run Ansible to pick up rotations. Logstash must have
+#   read access to the files (typically `root:logstash 0640` and a
+#   world-traversable parent directory).
+# @end
+logstash_tls_copy_certs: true
+
 # @var logstash_certs_dir:description: Directory on the Logstash host where TLS certificates are stored
 logstash_certs_dir: /etc/logstash/certs