Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions content/about.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
---
title: About
description: |
NetObserv aka Network Observability is a set of components used to observe network traffic by generating NetFlows from eBPF agents, enrich those flows using a configurable pipeline that is Kubernetes-aware, export them in various ways (logs, metrics, Kafka, IPFIX...), and finally provide a comprehensive visualization tool for making sense of that data, and a CLI. Those components can be used as standalones or deployed in Kubernetes / OpenShift via an integrated Operator.
NetObserv aka Network Observability is a set of components used to observe network traffic by generating network flows from eBPF agents, enrich those flows using a configurable pipeline that is Kubernetes-aware, export them in various ways (logs, metrics, Kafka, IPFIX...), and finally provide a comprehensive visualization tool for making sense of that data, and a CLI. Those components can be used as standalones or deployed in Kubernetes via an operator.
layout: :theme/page
---

# About NetObserv

NetObserv is a set of components used to observe network traffic by generating NetFlows from [eBPF agents](https://github.com/netobserv/netobserv-ebpf-agent), enriching those flows using [a configurable pipeline](https://github.com/netobserv/flowlogs-pipeline/) which is Kubernetes-aware, exporting them in various ways (logs, metrics, Kafka, IPFIX...), and finally providing a comprehensive [visualization tool](https://github.com/netobserv/network-observability-console-plugin/) for making sense of that data, and [a CLI](https://github.com/netobserv/network-observability-cli). Those components can be used as standalones or deployed in Kubernetes / OpenShift via an [integrated Operator](https://github.com/netobserv/network-observability-operator/).
NetObserv is a set of components used to observe network traffic by generating network flows from [eBPF agents](https://github.com/netobserv/netobserv-ebpf-agent), enriching those flows using [a configurable pipeline](https://github.com/netobserv/flowlogs-pipeline/) which is Kubernetes-aware, exporting them in various ways (logs, metrics, Kafka, IPFIX...), and finally providing a comprehensive [visualization tool](https://github.com/netobserv/netobserv-web-console/) for making sense of that data, and [a CLI](https://github.com/netobserv/netobserv-cli). Those components can be used as standalones or deployed in Kubernetes with an [operator](https://github.com/netobserv/netobserv-operator/).

It is known and distributed in Red Hat OpenShift as the [Network Observability operator](https://docs.openshift.com/container-platform/latest/observability/network_observability/network-observability-operator-release-notes.html).
It is fully open-source and, for the most part, CNI-agnostic (some features such as monitoring network policy events are CNI-dependent).

It is known and distributed in Red Hat OpenShift as the [Network Observability operator](https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/network_observability/installing-network-observability-operators).

## Topology view

Expand Down
2 changes: 1 addition & 1 deletion content/posts/2022-12-12-3d_topology.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ One of the next implementations is going to be [sankey chart](https://observable

## We need you !

Feel free to contribute by commenting this post, opening issues in [netobserv console plugin](https://github.com/netobserv/network-observability-console-plugin/issues) or opening pull request in any [netobserv component](https://github.com/netobserv)
Feel free to contribute by commenting this post, opening issues in [netobserv console plugin](https://github.com/netobserv/netobserv-web-console/issues) or opening pull request in any [netobserv component](https://github.com/netobserv)

Tell us more about your expectations, the way you currently solve issues and what could help your daily experience.

Expand Down
4 changes: 2 additions & 2 deletions content/posts/2023-10-02-lokiless.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ For simplicity, ClickHouse is deployed locally and [ktunnel](https://github.com/
### Prerequisites

- An OpenShift or Kubernetes cluster
- [NetObserv operator](https://github.com/netobserv/network-observability-operator) installed (do not install a `FlowCollector` yet).
- [NetObserv operator](https://github.com/netobserv/netobserv-operator) installed (do not install a `FlowCollector` yet).
- ClickHouse binary: grab it as explained in their [quick install guide](https://clickhouse.com/docs/en/install#quick-install).
- [ktunnel](https://github.com/omrikiei/ktunnel) binary.
- Some common tools such as `curl`, `kubectl`, `envsubst`...
Expand Down Expand Up @@ -88,7 +88,7 @@ It creates a `clickhouse` service in the `default` namespace, bridged to your lo

### Prepare Kafka

The steps here are very similar to the [Kafka deployment script](https://github.com/netobserv/network-observability-operator/blob/release-1.4/.mk/development.mk#L54-L63) that we use in NetObserv for development and testing purposes. They use [Strimzi](https://strimzi.io/) - the upstream of AMQ Streams for OpenShift - to get Kafka in the cluster, and a topic named "flows-export" is pre-created.
The steps here are very similar to the [Kafka deployment script](https://github.com/netobserv/netobserv-operator/blob/release-1.4/.mk/development.mk#L54-L63) that we use in NetObserv for development and testing purposes. They use [Strimzi](https://strimzi.io/) - the upstream of AMQ Streams for OpenShift - to get Kafka in the cluster, and a topic named "flows-export" is pre-created.

```bash
# Create a namespace for all the deployments
Expand Down
4 changes: 2 additions & 2 deletions content/posts/2023-12-05-acm.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ If you're running NetObserv 1.5 or above, edit the `FlowCollector` resource, fin
- `workload_egress_packets_total`
- `workload_ingress_packets_total`

This adds metrics used in later steps. [Take a look](https://github.com/netobserv/network-observability-operator/blob/main/docs/Metrics.md) at the available metrics if you want to customize this setup further.
This adds metrics used in later steps. [Take a look](https://github.com/netobserv/netobserv-operator/blob/main/docs/Metrics.md) at the available metrics if you want to customize this setup further.

If you are only interested in metrics, you don't need to install and enable Loki. Read more about that [here](https://cloud.redhat.com/blog/deploying-network-observability-without-loki-an-example-with-clickhouse). But while NetObserv doesn't currently provide an out-of-the-box experience for viewing multi-cluster logs from Loki, these flow logs are still the most detailed and accurate data available when it comes to troubleshooting the network per cluster, providing a finer insight than metrics.

Expand Down Expand Up @@ -255,6 +255,6 @@ We would end up with these two new rules:

Be careful about escaping double-quotes, though it's not very pretty, it is necessary: else you would end up with a parsing error. Also, the `label_replace` chained calls here could be avoided as they look messy, but they make it actually easier to manipulate those metrics later on, in Grafana.

Also, don't forget that NetObserv has [more metrics to show](https://github.com/netobserv/network-observability-operator/blob/main/docs/Metrics.md), especially starting from the coming-soon release 1.5, such as TCP latencies, [packet drop](https://cloud.redhat.com/blog/network-observability-real-time-per-flow-packets-drop) counters and so on. And just for teasing, we are working on a fresh new API in NetObserv that will soon let you build pretty much any metric you want out of flow logs, for even more dashboarding possibilities.
Also, don't forget that NetObserv has [more metrics to show](https://github.com/netobserv/netobserv-operator/blob/main/docs/Metrics.md), especially starting from the coming-soon release 1.5, such as TCP latencies, [packet drop](https://cloud.redhat.com/blog/network-observability-real-time-per-flow-packets-drop) counters and so on. And just for teasing, we are working on a fresh new API in NetObserv that will soon let you build pretty much any metric you want out of flow logs, for even more dashboarding possibilities.

If you want to get in touch with the NetObserv team, you can use our [discussion board](https://github.com/orgs/netobserv/discussions).
2 changes: 1 addition & 1 deletion content/posts/2024-02-28-whats_new_1_5.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ In "Processor configuration", the changes are to enable availability zones, clus
![Processor configuration]({page.image('whats-new-1-5/processor-configuration.png')})
_<div style="text-align: center">Figure 4: Processor configuration</div>_

The full list of predefined metrics is [here](https://github.com/netobserv/network-observability-operator/blob/main/docs/Metrics.md). When you include a metric, it stores it in Prometheus and is available as a Prometheus metric prefixed with "netobserv_". For example, if you add the metric *namespace_egress_bytes_total*, then go to **Observe > Metrics** and enter the PromQL `sum(rate(netobserv_namespace_egress_bytes_total[1m]))`. This should display a single line that is the sum of the average number of egress bytes over one-minute intervals. Select a refresh time in the upper right dropdown if you want the graph to be updated periodically.
The full list of predefined metrics is [here](https://github.com/netobserv/netobserv-operator/blob/main/docs/Metrics.md). When you include a metric, it stores it in Prometheus and is available as a Prometheus metric prefixed with "netobserv_". For example, if you add the metric *namespace_egress_bytes_total*, then go to **Observe > Metrics** and enter the PromQL `sum(rate(netobserv_namespace_egress_bytes_total[1m]))`. This should display a single line that is the sum of the average number of egress bytes over one-minute intervals. Select a refresh time in the upper right dropdown if you want the graph to be updated periodically.

Availability zones and cluster ID will be covered in the traffic flow table section below.

Expand Down
2 changes: 1 addition & 1 deletion content/posts/2024-04-05-agent_metrics_perf.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@ This had a big impact on CPU usage:
- +457% CPU when compared to `cacheMaxFlows=10000`
- +143% CPU when compared to `cacheMaxFlows=2000, cacheTimeout=1s`

To avoid this, it's best to keep `cacheMaxFlows` big enough to minimize the ring buffer ratio usage, in spite of the increased memory usage. Other mitigation options could be limiting the number of generated flows by other means, such as [configuring the agent](https://github.com/netobserv/network-observability-operator/blob/19bcb45d6899f12feb13f94a925a10a4f4c92106/docs/FlowCollector.md#flowcollectorspecagentebpf-1) to add filters on monitored interfaces or to increase the sampling value, if the resulting loss in observability is an acceptable trade-off for you.
To avoid this, it's best to keep `cacheMaxFlows` big enough to minimize the ring buffer ratio usage, in spite of the increased memory usage. Other mitigation options could be limiting the number of generated flows by other means, such as [configuring the agent](https://github.com/netobserv/netobserv-operator/blob/19bcb45d6899f12feb13f94a925a10a4f4c92106/docs/FlowCollector.md#flowcollectorspecagentebpf-1) to add filters on monitored interfaces or to increase the sampling value, if the resulting loss in observability is an acceptable trade-off for you.

## Conclusion

Expand Down
6 changes: 3 additions & 3 deletions content/posts/2024-07-25-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ oc krew install netobserv

### From tar archive

Download the latest version either from [github releases](https://github.com/netobserv/network-observability-cli/releases) or [developers.redhat.com](https://developers.redhat.com/content-gateway/rest/mirror2/pub/openshift-v4/clients/netobserv/latest).
Download the latest version either from [github releases](https://github.com/netobserv/netobserv-cli/releases) or [developers.redhat.com](https://developers.redhat.com/content-gateway/rest/mirror2/pub/openshift-v4/clients/netobserv/latest).

Then follow the instructions on [Installing and using CLI plugins](https://docs.openshift.com/container-platform/4.15/cli_reference/openshift_cli/extending-cli-plugins.html#cli-installing-plugins_cli-extend-plugins) from Openshift documentation.

Expand Down Expand Up @@ -141,7 +141,7 @@ oc netobserv help

```
Netobserv allows you to capture flow and packets from your cluster.
Find more information at: https://github.com/netobserv/network-observability-cli/
Find more information at: https://github.com/netobserv/netobserv-cli/

Syntax: netobserv [flows|packets|cleanup] [filters]

Expand Down Expand Up @@ -172,7 +172,7 @@ options:
```

As you can see, the CLI offers two captures options for now:
- `flows` to get network flows containing [enriched informations](https://github.com/netobserv/network-observability-operator/blob/main/docs/flows-format.adoc#network-flows-format-reference)
- `flows` to get network flows containing [enriched informations](https://github.com/netobserv/netobserv-operator/blob/main/docs/flows-format.adoc#network-flows-format-reference)

On top of this command, you can add extra features and filters. Example to capture `Round Trip Time` of `TCP` flows using port `80`:
```
Expand Down
4 changes: 2 additions & 2 deletions content/posts/2024-10-23-whats_new_1_6.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ To generate flows (or not generate flows) for very specific data, you can specif
protocol: TCP
```

Enter `oc edit flowcollector`. Again, look for the `ebpf` section and add the rest of the lines starting at `flowFilter`. The `cidr` specifies a network address and prefix. The example refers to any traffic, but for a specific network, use a value like `10.0.62.0/24`. The feature is limited to one filter and many of the attributes can only have one value. For more details, see the [flowFilter reference](https://github.com/netobserv/network-observability-operator/blob/main/docs/flowcollector-flows-netobserv-io-v1beta2.adoc#specagentebpfflowfilter).
Enter `oc edit flowcollector`. Again, look for the `ebpf` section and add the rest of the lines starting at `flowFilter`. The `cidr` specifies a network address and prefix. The example refers to any traffic, but for a specific network, use a value like `10.0.62.0/24`. The feature is limited to one filter and many of the attributes can only have one value. For more details, see the [flowFilter reference](https://github.com/netobserv/netobserv-operator/blob/main/docs/flowcollector-flows-netobserv-io-v1beta2.adoc#specagentebpfflowfilter).

### eBPF Agent metrics

Expand Down Expand Up @@ -181,7 +181,7 @@ _<div style="text-align: center">Figure 7: Not 'ip' filter</div>_

In my [Network Observability 1.5 blog](https://developers.redhat.com/articles/2024/03/20/whats-new-network-observability-15), I covered the development preview of the FlowMetrics API. While it's possible to take any [flow data labels](https://docs.openshift.com/container-platform/4.16/observability/network_observability/json-flows-format-reference.html) and turn it into a Prometheus metric, that web page was updated to indicate which fields can be safely used as labels to avoid high cardinality.

This feature is now GA, and includes more filtering options (see [reference](https://github.com/netobserv/network-observability-operator/blob/main/docs/FlowMetric.md)) and support for [dashboards in OpenShift](https://github.com/netobserv/network-observability-operator/blob/main/docs/Metrics.md#charts-openshift-only). That link also has many more examples of using custom metrics.
This feature is now GA, and includes more filtering options (see [reference](https://github.com/netobserv/netobserv-operator/blob/main/docs/FlowMetric.md)) and support for [dashboards in OpenShift](https://github.com/netobserv/netobserv-operator/blob/main/docs/Metrics.md#charts-openshift-only). That link also has many more examples of using custom metrics.


## Network Observability CLI
Expand Down
2 changes: 1 addition & 1 deletion content/posts/2025-02-11-cli_whats_new_1.8.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ key: SrcK8S_Namespace value: openshift.*

![regexes]({page.image('cli-whats-new-1-8/regexes.png')})

Regexes are comma-separated, so you can use multiple at once, such as `--regexes=SrcK8S_Namespace~my-ns,SrcK8S_Name~my-app`. Refer to the [flows format](https://github.com/netobserv/network-observability-operator/blob/main/docs/flows-format.adoc) to see the possible fields.
Regexes are comma-separated, so you can use multiple at once, such as `--regexes=SrcK8S_Namespace~my-ns,SrcK8S_Name~my-app`. Refer to the [flows format](https://github.com/netobserv/netobserv-operator/blob/main/docs/flows-format.adoc) to see the possible fields.

## Unified Collector User Experience
All filtering capabilities are now supported for **packets** capture and displays enriched data while collecting. This improvement was made possible by introducing the [flowlogs-pipeline](https://github.com/netobserv/flowlogs-pipeline) component inside [eBPF agents](https://github.com/netobserv/netobserv-ebpf-agent), which parses packets and generates flows from them.
Expand Down
2 changes: 1 addition & 1 deletion content/posts/2025-02-13-perf-improvements/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@ The drawback of using per-CPU maps for network flows is that for the same flow,

Today, we change all of that by refactoring our data structures. Our solution is to avoid writing the flow map from the enrichment hooks. Instead of that, we are introducing a new map, dedicated to enriched data. Map keys are going to be duplicated across those two maps, but it's a lesser evil. So now, we can change the main map to be a shared one across CPUs, with a spinlock. We still have some reassembling work to do in the user space though, to merge the main map with the enrichment map, but it is more straightforward than merging entire flows together. We also have a couple of ideas to further improve this process, more on that later.

Splitting data between a main map and an enrichment map has another benefit: when no enrichment is needed (e.g. when none of the [agent features](https://github.com/netobserv/network-observability-operator/blob/main/docs/FlowCollector.md#flowcollectorspecagentebpf-1) are enabled), no memory is allocated for them, resulting — again — in a more efficient memory usage.
Splitting data between a main map and an enrichment map has another benefit: when no enrichment is needed (e.g. when none of the [agent features](https://github.com/netobserv/netobserv-operator/blob/main/docs/FlowCollector.md#flowcollectorspecagentebpf-1) are enabled), no memory is allocated for them, resulting — again — in a more efficient memory usage.

This is for a large part what triggered the memory improvement mentioned above:

Expand Down
2 changes: 1 addition & 1 deletion content/posts/2025-02-26-whats-new-1-8/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -315,4 +315,4 @@ This is another feature-rich release that works hand-in-hand with OVN-Kubernetes

Network Observability continues to provide flexibility in deciding what you want to observe so that you can minimize the resources used. This is in addition to the internal optimizations that we've made in this release.

While many of the features are in Developer Preview, it gives you a chance to try these out and give us some feedback before it becomes generally available. You can write comments and contact us on the [discussion board](https://github.com/netobserv/network-observability-operator/discussions).
While many of the features are in Developer Preview, it gives you a chance to try these out and give us some feedback before it becomes generally available. You can write comments and contact us on the [discussion board](https://github.com/netobserv/netobserv-operator/discussions).
2 changes: 1 addition & 1 deletion content/posts/2025-07-17-whats-new-1-9/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,6 @@ Figure 6: Wireshark - Packet capture file with comments

## Summary

This is another solid release from the Network Observability team. If you use IPsec, you can get insight into this type of traffic. A filter query was added in both flowlogs-pipeline and the Network Observability CLI. If you want to easily capture flows, metrics, and packets, Network Observability CLI is the tool for you! Write to us on the [discussion board](https://github.com/netobserv/network-observability-operator/discussions) if you have any feedback or suggestions for improvements.
This is another solid release from the Network Observability team. If you use IPsec, you can get insight into this type of traffic. A filter query was added in both flowlogs-pipeline and the Network Observability CLI. If you want to easily capture flows, metrics, and packets, Network Observability CLI is the tool for you! Write to us on the [discussion board](https://github.com/netobserv/netobserv-operator/discussions) if you have any feedback or suggestions for improvements.

_Special thanks to Julien Pinsonneau, Joel Takvorian, Mehul Modi, and Mohamed S. Mahmoud for reviewing._
Loading
Loading