[feat] Network Costs Test by Manas23601 · Pull Request #19 · opencost/opencost-integration-tests

Manas23601 · 2025-07-13T18:01:13Z

Description

Network Costs

Receive Bytes
Transfer Bytes

Prometheus Query

sum(increase(container_network_receive_bytes_total{pod!=""}[24h:5m])) by (pod, namespace)`

sum(increase(container_network_transmit_bytes_total{pod!="", %s}[%s:%dm])) by (pod_name, pod, namespace, %s)

Testing

Network Costs

md@Manas:/mnt/c/Manas/opencost-integration-tests$ go test -v ./test/integration/prometheus/network_costs_test.go
=== RUN   TestNetworkCosts
    network_costs_test.go:34: testCases: [{Yesterday 24h namespace false}]
=== RUN   TestNetworkCosts/Yesterday
    network_costs_test.go:180: Namespace argo
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 2113274703.39
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 3601434433.67
    network_costs_test.go:180: Namespace gpu-operator
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 3951830782.66
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 3303167969.23
    network_costs_test.go:180: Namespace cert-manager
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 51280593.62
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 35189496.75
    network_costs_test.go:180: Namespace ingress-nginx
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 77230121.14
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 69102378.82
    network_costs_test.go:180: Namespace load-generator
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 7790042667.26
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 20252712121.09
    network_costs_test.go:180: Namespace folding-at-home
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 389527427.79
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 591013869.49
    network_costs_test.go:180: Namespace sealed-secrets
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 9175452.21
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 8833045.30
    network_costs_test.go:180: Namespace prometheus-system
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 29482772937.37
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 48800928242.51
    network_costs_test.go:180: Namespace kube-system
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 104585511444.96
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 164773304552.70
    network_costs_test.go:180: Namespace opencost
    network_costs_test.go:185:   - NetworkTransferBytes[Pass]: ~ 174612718.16
    network_costs_test.go:190:   - NetworkReceiveBytes[Pass]: ~ 463782783.89
--- PASS: TestNetworkCosts (2.09s)
    --- PASS: TestNetworkCosts/Yesterday (2.09s)
PASS
ok      command-line-arguments  2.098s

Network Internet Costs

=== RUN   TestNetworkInternetCosts
    network_internet_costs_test.go:33: testCases: [{Yesterday 24h pod false}]
=== RUN   TestNetworkInternetCosts/Yesterday
    network_internet_costs_test.go:167: Pod test-install-speedtest-tracker-56944bbf5b-9lpd4
    network_internet_costs_test.go:172:   - NetworkInternetCost[Pass]: ~ 0.52375
--- PASS: TestNetworkInternetCosts (2.78s)
    --- PASS: TestNetworkInternetCosts/Yesterday (2.78s)
PASS
ok      command-line-arguments  2.785s

Note

Adds Prometheus vs Allocation integration tests for network bytes and egress cost (internet, zone, region), and wires them into test.bats.

Tests (Prometheus integration):
- Network bytes (namespace): Validates container_network_receive_bytes_total and container_network_transmit_bytes_total against Allocation /allocation by namespace in test/integration/prometheus/network_costs_test.go.
- Network egress costs (pod):
  - Internet egress: Compares kubecost_pod_network_egress_bytes_total{internet="true"} cost vs Allocation NetworkInternetCost in network_internet_costs_test.go.
  - Cross-zone egress: Compares kubecost_pod_network_egress_bytes_total{internet="false", same_zone="false", same_region="true"} cost vs Allocation NetworkCrossZoneCost in network_zone_costs_test.go.
  - Cross-region egress: Compares kubecost_pod_network_egress_bytes_total{internet="false", same_region="false"} cost vs Allocation NetworkCrossRegionCost in network_region_costs_test.go.
- Test runner: test/integration/prometheus/test.bats updated to execute the new network tests.

^{Written by Cursor Bugbot for commit 99d1524. This will update automatically on new commits. Configure here.}

mbolt35 · 2025-07-14T14:02:58Z

Just wanted to drop a quick note about network costing. When we calculate network costs, we do not use the NetworkTransferBytes or NetworkReceiveBytes -- It's not a bad test to run for comparison, but as it relates to cost, we do not use these statistics for costs because they include all sub-categories of traffic (ie: local traffic, which is free).

I'm not sure if the network-costs pods are installed on this cluster, so we should definitely ensure that gets done as soon as possible.

mbolt35 · 2025-07-14T14:05:44Z

The actual costs and usage data we use is here: https://github.com/opencost/opencost/blob/ef740759c88732487bffa644509480b070ca77c5/modules/prometheus-source/pkg/prom/metricsquerier.go#L907-L1021

But note that this requires a network-costs daemonset be running on the cluster (not sure if this is currently running on that cluster - if not, these metrics won't show up)

ameijer · 2025-07-28T14:30:42Z

test/integration/prometheus/network_internet_costs_test.go

 				networkCostsPod.AllocNetworkInternetGiB = allocationResponseItem.NetworkInternetCost
 			}

+			noNegligibleCosts := false


validCostsSeen := false

Manas23601 · 2025-07-28T17:30:07Z

@ameijer , if network zone and region tests look good, I'll go ahead anc comment those out in test.bats

ameijer · 2025-07-28T17:33:23Z

@Manas23601 looks like the tests for zone and region are failing? for those, it is ok if we don't see any valid values

ameijer · 2025-07-28T17:37:43Z

test/integration/prometheus/network_internet_costs_test.go

+				// 		AllocNetworkInternetGiB: 0.0,
+				// 	}
+				// 	continue
+				// }


why is this commented?

I wanted the option to drill down to a pod level if needed. do you think we need to go down further down?

Signed-off-by: Manas Sivakumar <manas23601@gmail.com>

Manas23601 · 2025-07-28T18:40:52Z

@Manas23601 looks like the tests for zone and region are failing? for those, it is ok if we don't see any valid values

Manas23601 · 2025-07-28T18:41:57Z

@Manas23601 looks like the tests for zone and region are failing? for those, it is ok if we don't see any valid values

These are meant to fail because there is no valid zone and region costs. We should either not fail the tests if there are no valid costs or not run this test at all

Signed-off-by: Manas Sivakumar <manas23601@gmail.com>

…ion-tests into networkcost

Manas23601 · 2025-11-18T18:18:03Z

bugbot run

cursor · 2025-11-18T18:18:05Z

Skipping Bugbot: Bugbot is disabled for this repository

Signed-off-by: Manas Sivakumar <manas23601@gmail.com>

ameijer · 2025-11-24T17:42:17Z

bugbot run

cursor

This PR is being reviewed by Cursor Bugbot

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

cursor · 2025-11-24T17:44:42Z

test/integration/prometheus/network_costs_test.go

+				networkCostsNamespace, ok := networkCostsNamespaceMap[namespace]
+				if !ok {
+					networkCostsNamespaceMap[namespace] = &NetworkCostsAggregate{
+						PromNetworkReceiveBytes: networkTransferBytesPod,


Bug: Wrong field assigned in network transfer initialization

When a namespace doesn't exist in the map during Network Transfer Bytes processing, PromNetworkReceiveBytes is incorrectly set to networkTransferBytesPod instead of 0.0. This causes the receive bytes value to be initialized with transfer bytes data, leading to incorrect aggregation and comparison results.

cursor · 2025-11-24T17:44:42Z

test/integration/prometheus/network_costs_test.go

+					t.Errorf("  - NetworkReceiveBytes[Fail]: DifferencePercent: %0.2f, Prometheus: %0.2f, /allocation: %0.2f", diff_percent, networkCostValues.PromNetworkReceiveBytes, networkCostValues.AllocNetworkReceiveBytes)
+				} else {
+					t.Logf("  - NetworkReceiveBytes[Pass]: ~ %0.2f", networkCostValues.PromNetworkReceiveBytes)
+				}


Bug: NetworkReceiveBytes validation uses wrong comparison result

The withinRange variable from the NetworkTransferBytes comparison on line 181 is reused for NetworkReceiveBytes validation without recalculating it. This means NetworkReceiveBytes always reports the same pass/fail status as NetworkTransferBytes, regardless of the actual NetworkReceiveBytes values. The code needs to call utils.AreWithinPercentage again for NetworkReceiveBytes before line 187.

cursor · 2025-11-24T17:44:42Z

test/integration/prometheus/network_internet_costs_test.go

+			// Network Internet Cost for all Pods
+			// --------------------------------
+
+			networkInternetCost := promNetworkInternetCostResponse.Data.Result[0].Value.Value


Bug: Potential panic from accessing empty result array

The code accesses Data.Result[0] without checking if the Result array is empty. If the Prometheus query returns no results, this causes a panic. This can happen when the network-costs daemonset isn't running or when there's no matching data for the query window.

Additional Locations (2)

test/integration/prometheus/network_region_costs_test.go#L100-L101

test/integration/prometheus/network_zone_costs_test.go#L100-L101

cursor · 2025-11-24T17:44:43Z

test/integration/prometheus/network_region_costs_test.go

+			promNetworkRegionInput.Filters = map[string]string{
+				"internet": "false",
+				"same_Region": "false",
+				"same_region": "false",


Bug: Duplicate filter key with different casing

The filters map contains both "same_Region": "false" and "same_region": "false". Since Go map keys are case-sensitive, this creates two separate filter entries. The capital-R version appears to be a typo and likely won't match any Prometheus label, potentially causing the query to return incorrect results or no results at all.

cursor · 2025-12-11T18:19:54Z

test/integration/prometheus/network_region_costs_test.go

+				"internet": "false",
+				"same_Region": "false",
+				"same_region": "false",
+			}


Bug: Missing same_zone filter uses wrong same_Region key

The filter map contains same_Region (capital R) instead of same_zone. According to the comment on line 56, the query should filter by same_zone="false" along with internet="false" and same_region="false". The same_Region key is a typo that won't match any Prometheus label, and the required same_zone filter is missing entirely, causing the query to return incorrect network cost data.

Manas23601 added 2 commits July 13, 2025 23:30

[feat] Network Costs

8644477

remove unclosed tilde

1a8cd56

Manas23601 added 2 commits July 14, 2025 21:35

run test in integration pipeline using bats

bdcb084

[Error] Fix Name of Test in bats

974807b

Manas23601 changed the title ~~NetWork Costs~~ [feat] Network Costs Jul 14, 2025

Manas23601 changed the title ~~[feat] Network Costs~~ [feat] Network Costs Test Jul 14, 2025

Manas23601 added 2 commits July 24, 2025 19:23

[feat] Network Zone Costs

b233444

Edge case where all costs are below 1 cent

39c1dc9

ameijer reviewed Jul 28, 2025

View reviewed changes

Manas23601 added 2 commits July 28, 2025 22:50

Changed Variable name to a one that makes sense

08b4800

Two Similar Tests for Zone and Region Network Costs

290c12e

ameijer reviewed Jul 28, 2025

View reviewed changes

Merge branch 'main' into networkcost

6ff0fe0

Signed-off-by: Manas Sivakumar <manas23601@gmail.com>

Manas23601 closed this Jul 28, 2025

Manas23601 reopened this Jul 28, 2025

Manas23601 added 3 commits July 29, 2025 00:34

Merge branch 'main' into networkcost

8e28d44

Signed-off-by: Manas Sivakumar <manas23601@gmail.com>

merge main branch

3e905b1

Merge branch 'networkcost' of github.com:Manas23601/opencost-integrat…

86ec5e2

…ion-tests into networkcost

Manas23601 self-assigned this Jul 31, 2025

Manas23601 requested a review from mbolt35 July 31, 2025 06:21

Merge branch 'main' into networkcost

eebf2a0

Signed-off-by: Manas Sivakumar <manas23601@gmail.com>

Manas23601 requested a review from a team as a code owner November 24, 2025 17:41

cursor bot reviewed Nov 24, 2025

View reviewed changes

Merge branch 'main' into networkcost

99d1524

cursor bot reviewed Dec 11, 2025

View reviewed changes

Comments

Conversation

Manas23601 commented Jul 13, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Prometheus Query

Testing

Network Costs

Network Internet Costs

Uh oh!

mbolt35 commented Jul 14, 2025

Uh oh!

mbolt35 commented Jul 14, 2025

Uh oh!

ameijer Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Manas23601 commented Jul 28, 2025

Uh oh!

ameijer commented Jul 28, 2025

Uh oh!

ameijer Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Manas23601 Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Manas23601 commented Jul 28, 2025

Uh oh!

Manas23601 commented Jul 28, 2025

Uh oh!

Manas23601 commented Nov 18, 2025

Uh oh!

cursor bot commented Nov 18, 2025

Uh oh!

ameijer commented Nov 24, 2025

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

This PR is being reviewed by Cursor Bugbot

Uh oh!

cursor bot Nov 24, 2025

Choose a reason for hiding this comment

Bug: Wrong field assigned in network transfer initialization

Uh oh!

cursor bot Nov 24, 2025

Choose a reason for hiding this comment

Bug: NetworkReceiveBytes validation uses wrong comparison result

Uh oh!

cursor bot Nov 24, 2025

Choose a reason for hiding this comment

Bug: Potential panic from accessing empty result array

Uh oh!

cursor bot Nov 24, 2025

Choose a reason for hiding this comment

Bug: Duplicate filter key with different casing

Uh oh!

cursor bot Dec 11, 2025

Choose a reason for hiding this comment

Bug: Missing same_zone filter uses wrong same_Region key

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Manas23601 commented Jul 13, 2025 •

edited by cursor bot

Loading