Conversation
Add continuous file watching capability for audit logs using the tail:// URL scheme. This allows monitoring files for new entries and handles log rotation via inode detection and file truncation. Closes falcosecurity#191 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Richard Tweed <RichardoC@users.noreply.github.com>
Add documentation for the new tail:// URL scheme and watchPollIntervalMs configuration option. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Richard Tweed <RichardoC@users.noreply.github.com>
Signed-off-by: Richard Tweed <RichardoC@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: RichardoC The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Rules files suggestionsrulesComparing No changes detected |
irozzo-1A
left a comment
There was a problem hiding this comment.
Hi @RichardoC, thanks for your contribution. I suggest using file instead of tail to have a more standard URI, and using an event based approach instead of polling.
| return k.OpenWebServer(u.Host, u.Path, false) | ||
| case "https": | ||
| return k.OpenWebServer(u.Host, u.Path, true) | ||
| case "tail": |
There was a problem hiding this comment.
| case "tail": | |
| case "file": |
plugins/k8saudit/README.md
Outdated
| This plugin supports consuming Kubernetes Audit Events coming from the [Webhook backend](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#webhook-backend) or from a file. For webhooks, the plugin embeds a web server that listens on a configurable port and accepts POST requests. The posted JSON object comprises one or more events. The web server of the plugin can be configured as part of the plugin's init configuration and open parameters. For files, the plugin expects content to be in [JSONL format](https://jsonlines.org/), where each line represents a JSON object, containing one or more audit events. | ||
|
|
||
| The expected way of using the plugin with Falco is through a Webhook. File reading support can be used with Stratoshark or testing and development. | ||
| The expected way of using the plugin with Falco is through a Webhook. File reading support can be used with Stratoshark or testing and development. The `tail://` scheme enables continuous file watching with log rotation support, useful for reading audit logs written to disk by the API server. |
There was a problem hiding this comment.
| The expected way of using the plugin with Falco is through a Webhook. File reading support can be used with Stratoshark or testing and development. The `tail://` scheme enables continuous file watching with log rotation support, useful for reading audit logs written to disk by the API server. | |
| The expected way of using the plugin with Falco is through a Webhook. File reading support can be used with Stratoshark or testing and development. The `file://` scheme enables continuous file watching with log rotation support, useful for reading audit logs written to disk by the API server. |
plugins/k8saudit/README.md
Outdated
| **Open Parameters**: | ||
| - `http://<host>:<port>/<endpoint>`: Opens an event stream by listening on an HTTP web server | ||
| - `https://<host>:<port>/<endpoint>`: Opens an event stream by listening on an HTTPS web server | ||
| - `tail://<filepath>`: Opens an event stream by continuously watching a file for new audit events, similar to `tail -f`. Handles log rotation (inode changes) and file truncation automatically. Example: `tail:///var/log/kube-apiserver/audit.log` |
There was a problem hiding this comment.
| - `tail://<filepath>`: Opens an event stream by continuously watching a file for new audit events, similar to `tail -f`. Handles log rotation (inode changes) and file truncation automatically. Example: `tail:///var/log/kube-apiserver/audit.log` | |
| - `file://<filepath>`: Opens an event stream by continuously watching a file for new audit events, similar to `tail -f`. Handles log rotation (inode changes) and file truncation automatically. Example: `file:///var/log/kube-apiserver/audit.log` |
| UseAsync bool `json:"useAsync" jsonschema:"title=Use async extraction,description=If true then async extraction optimization is enabled (Default: true),default=true"` | ||
| MaxEventSize uint64 `json:"maxEventSize" jsonschema:"title=Maximum event size,description=Maximum size of single audit event (Default: 262144),default=262144"` | ||
| WebhookMaxBatchSize uint64 `json:"webhookMaxBatchSize" jsonschema:"title=Maximum webhook request size,description=Maximum size of incoming webhook POST request bodies (Default: 12582912),default=12582912"` | ||
| WatchPollIntervalMs uint64 `json:"watchPollIntervalMs" jsonschema:"title=Watch poll interval,description=Polling interval in milliseconds when watching a file with tail:// scheme (Default: 250),default=250"` |
There was a problem hiding this comment.
We can avoid this if we use an event based approach e.g. inotify
| // OpenFileTail opens a source.Instance that continuously watches a file for | ||
| // new K8S Audit Events, similar to "tail -f". It handles log rotation by | ||
| // detecting file truncation or inode changes. | ||
| func (k *Plugin) OpenFileTail(path string) (source.Instance, error) { |
There was a problem hiding this comment.
Polling and opening the file each time is not very efficient way of tailing the file content, what about using fsnotify. You can take inspiration from the snippet below:
package main
import (
"fmt"
"io"
"os"
"github.com/fsnotify/fsnotify"
)
func main() {
filePath := "test.log"
watcher, _ := fsnotify.NewWatcher()
defer watcher.Close()
// 1. Open the file and seek to the end
file, _ := os.Open(filePath)
file.Seek(0, io.SeekEnd)
// 2. Start watching the file for changes
watcher.Add(filePath)
fmt.Println("Tailing test.log... (supports rotation/truncation)")
for {
select {
case event, ok := <-watcher.Events:
if !ok {
return
}
// Handle Write: New data added
if event.Op&fsnotify.Write == fsnotify.Write {
io.Copy(os.Stdout, file)
}
// Handle Rename/Remove: This is Log Rotation
if event.Op&(fsnotify.Rename|fsnotify.Remove) == fsnotify.Remove || event.Op&fsnotify.Rename == fsnotify.Rename {
fmt.Println("\n[File rotated, re-opening...]")
file.Close()
// Wait/Retry until the new file exists
for {
newFile, err := os.Open(filePath)
if err == nil {
file = newFile
watcher.Add(filePath) // Re-watch the new file
break
}
// backoff
}
}
case err, ok := <-watcher.Errors:
if !ok {
return
}
fmt.Println("Error:", err)
}
}
}
Co-authored-by: Iacopo Rozzo <iacopo@sysdig.com> Signed-off-by: Richard Tweed <RichardoC@users.noreply.github.com>
Thank you for the thorough review @irozzo-1A , making those changes now |
|
Claude log for the changes |
Replace polling-based file watching with fsnotify for better efficiency. - Use fsnotify to watch parent directory (per maintainer recommendation) - Rename scheme from tail:// to file:// - Remove watchPollIntervalMs config (no longer needed) - Rename test package from tail to filewatch Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Richard Tweed <RichardoC@users.noreply.github.com>
0613d65 to
c12e1e0
Compare
|
The cargo install issue in ci seems unrelated to my changes |
| require ( | ||
| github.com/alecthomas/jsonschema v0.0.0-20220216202328-9eeeec9d044b | ||
| github.com/falcosecurity/plugin-sdk-go v0.8.3 | ||
| github.com/fsnotify/fsnotify v1.9.0 // indirect |
There was a problem hiding this comment.
Why // indirect? 🤔
This should be a direct dependency since it's explicitly imported in source.go
| limitations under the License. | ||
| */ | ||
|
|
||
| package filewatch_test |
There was a problem hiding this comment.
Since there's no filewatch package, why is this file here? 🤔
What type of PR is this?
/kind feature
Any specific area of the project related to this PR?
/area plugins
What this PR does / why we need it:
Add continuous file watching capability for audit logs using the
tail:// URL scheme. This allows monitoring files for new entries
and handles log rotation via inode detection and file truncation.
Which issue(s) this PR fixes:
Fixes #191
Special notes for your reviewer: Generated with Claude Code. Full transcript attached
claude-log.txt