Update to handle multiDim histograms #207

aebid · 2026-01-16T13:35:05Z

Add 2D linearizing, multi-dim histograms, and fixed memory leak in histmerger

Copilot

Pull request overview

This pull request adds support for multi-dimensional (2D and 3D) histograms to the FLAF analysis framework and fixes a memory leak in the histogram merger. The changes enable linearization of 2D histograms into 1D representations with custom binning per y-bin range.

Changes:

Added C++ function rebinHistogramDict to rebin multi-dimensional histograms with custom binning per y-range
Extended Python histogram handling to support 1D, 2D, and 3D histograms throughout the analysis pipeline
Fixed memory leak in histogram merger by setting SetDirectory(0) on histograms during accumulation

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
include/HistHelper.h	Adds C++ `rebinHistogramDict` function for multi-dimensional histogram rebinning with custom y-range binning
Common/HistHelper.py	Extends `RebinHisto` and `GetModel` to support dict-based binning and multi-dimensional histograms; adds cache helper functions
Analysis/tasks.py	Adds variable flattening logic to extract individual variables from multi-dimensional variable definitions
Analysis/HistTupleProducer.py	Refactors to handle flattened variables and adds shifted tree processing for systematic uncertainties
Analysis/HistProducerFromNTuple.py	Updates bin counting logic to correctly handle 2D/3D histograms and adds dimension detection
Analysis/HistPlotter.py	Removes duplicate `RebinHisto` and `getNewBins` functions (now in HistHelper.py); adds 2D binning support
Analysis/HistMergerFromHists.py	Fixes memory leak by calling `SetDirectory(0)` when accumulating histograms and refactors file handling

Analysis/HistTupleProducer.py

include/HistHelper.h

Copilot · 2026-01-16T13:48:19Z

include/HistHelper.h

+        all_output_edges.erase(std::unique(all_output_edges.begin(), all_output_edges.end()), all_output_edges.end());
+
+        // Create output histogram with variable binning
+        TH1D* hist_output = new TH1D("rebinned", "rebinned", all_output_edges.size() - 1, all_output_edges.data());


The histogram name and title are hardcoded as 'rebinned'. Consider accepting these as parameters or deriving them from the input histogram to make the function more flexible and preserve histogram metadata.

Suggested change

TH1D* hist_output = new TH1D("rebinned", "rebinned", all_output_edges.size() - 1, all_output_edges.data());

TH1D* hist_output = new TH1D(hist_initial->GetName(), hist_initial->GetTitle(), all_output_edges.size() - 1, all_output_edges.data());

Analysis/HistProducerFromNTuple.py

Copilot · 2026-01-16T13:48:24Z

include/HistHelper.h

+
+    TH1D* rebinHistogramDict(TH1* hist_initial, int N_bins, 
+                                const std::vector<std::pair<float, float>>& y_bin_ranges,
+                                const std::vector<std::vector<float>>& output_bin_edges) {


The function creates a new histogram with new but there's no clear ownership or deletion mechanism documented. Consider documenting that the caller is responsible for deleting the returned histogram, or use ROOT's memory management (e.g., adding the histogram to a directory).

Suggested change

TH1D* rebinHistogramDict(TH1* hist_initial, int N_bins,

const std::vector<std::pair<float, float>>& y_bin_ranges,

const std::vector<std::vector<float>>& output_bin_edges) {

/**

* Rebin a 1D histogram using a dictionary of variable-width bin edges.

*

* The returned histogram is allocated with `new` and ownership is

* transferred to the caller. The caller is responsible for managing

* its lifetime, e.g. by:

* - explicitly deleting it with `delete`, or

* - adding it to a ROOT TDirectory/TFile so that ROOT manages

* its deletion.

*/

TH1D* rebinHistogramDict(TH1* hist_initial, int N_bins,

const std::vector<std::pair<float, float>>& y_bin_ranges,

const std::vector<std::vector<float>>& output_bin_edges) {

Common/HistHelper.py

aebid · 2026-01-16T13:55:52Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-16T13:56:13Z

pipeline#13793040 started

cms-flaf-bot · 2026-01-16T13:59:08Z

pipeline#13793040 failed

aebid · 2026-01-16T14:11:40Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-16T14:11:58Z

pipeline#13793221 started

aebid · 2026-01-16T14:28:53Z

I hate merge conflicts and apparently broke everything when I was trying to fix them 😢

cms-flaf-bot · 2026-01-16T14:33:25Z

pipeline#13793221 failed

aebid · 2026-01-16T14:56:33Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-16T14:56:49Z

pipeline#13793622 started

aebid · 2026-01-16T15:02:52Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-16T15:03:16Z

pipeline#13793706 started

cms-flaf-bot · 2026-01-16T15:25:08Z

pipeline#13793706 failed

aebid · 2026-01-16T15:28:12Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-16T15:28:26Z

pipeline#13793934 started

cms-flaf-bot · 2026-01-16T15:54:19Z

pipeline#13793934 failed

cms-flaf-bot · 2026-01-16T15:57:06Z

pipeline#13793622 failed

aebid · 2026-01-16T16:07:08Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-16T16:07:25Z

pipeline#13794271 started

cms-flaf-bot · 2026-01-16T16:32:57Z

pipeline#13794271 failed

aebid · 2026-01-16T19:28:06Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-16T19:28:27Z

pipeline#13795444 started

cms-flaf-bot · 2026-01-16T19:52:54Z

pipeline#13795444 passed

aebid · 2026-01-16T22:18:01Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-16T22:18:20Z

pipeline#13796018 started

cms-flaf-bot · 2026-01-16T22:38:17Z

pipeline#13796018 failed

aebid · 2026-01-17T00:02:55Z

Testing has to wait until cms-flaf/HH_bbWW#53 is merged

aebid · 2026-01-17T17:37:08Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54

cms-flaf-bot · 2026-01-17T17:37:22Z

pipeline#13799766 started

cms-flaf-bot · 2026-01-17T17:58:54Z

pipeline#13799766 failed

aebid · 2026-01-18T11:47:11Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54
HH_bbtautau_version=PR_44

cms-flaf-bot · 2026-01-18T11:47:32Z

pipeline#13803335 started

cms-flaf-bot · 2026-01-18T12:14:26Z

pipeline#13803335 passed

aebid · 2026-01-18T12:14:50Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54
HH_bbtautau_version=PR_44

cms-flaf-bot · 2026-01-18T12:15:03Z

pipeline#13803427 started

aebid · 2026-01-18T12:16:05Z

The formatting failure is dumb. The lxplus black puts the file this specific way. But the GitHub check wants it in a different way.

cms-flaf-bot · 2026-01-18T12:40:53Z

pipeline#13803427 passed

aebid · 2026-01-18T13:51:57Z

@cms-flaf-bot please test

HH_bbWW_version=PR_54
HH_bbtautau_version=PR_44

cms-flaf-bot · 2026-01-18T13:52:10Z

pipeline#13803678 started

cms-flaf-bot · 2026-01-18T14:19:56Z

pipeline#13803678 passed

kandrosov · 2026-01-19T15:27:38Z

Analysis/HistProducerFromNTuple.py

+    if dims == 1:
+        unit_hist = rdf.Filter(filter_to_apply).Histo1D(
+            unit_bin_model, *var_bin_list, weight_name
+        )
+    elif dims == 2:
+        unit_hist = rdf.Filter(filter_to_apply).Histo2D(
+            unit_bin_model, *var_bin_list, weight_name
+        )
+    elif dims == 3:
+        unit_hist = rdf.Filter(filter_to_apply).Histo3D(
+            unit_bin_model, *var_bin_list, weight_name
+        )


Suggested change

if dims == 1:

unit_hist = rdf.Filter(filter_to_apply).Histo1D(

unit_bin_model, *var_bin_list, weight_name

)

elif dims == 2:

unit_hist = rdf.Filter(filter_to_apply).Histo2D(

unit_bin_model, *var_bin_list, weight_name

)

elif dims == 3:

unit_hist = rdf.Filter(filter_to_apply).Histo3D(

unit_bin_model, *var_bin_list, weight_name

)

rdf_filtered = rdf.Filter(filter_to_apply)

if dims >= 1 and dims <= 3:

mkhist_fn = getattr(rdf_filtered, f'Histo{dims}D')

unit_hist = mkhist_fn(unit_bin_model, *var_bin_list, weight_name)

kandrosov · 2026-01-19T16:01:01Z

Analysis/HistProducerFromNTuple.py

+    # Yes I know this is ugly
+    # Axis needs +2 for under/overflow, but only if the return is not 1!!!
+    # Was having issue with Z axis in 2D. We don't want to multiply by 3 if it's 2D
+    N_xbins = unit_hist.GetNbinsX() + 2
+    N_ybins = unit_hist.GetNbinsY() if hasattr(unit_hist, "GetNbinsY") else 1
+    N_ybins = N_ybins + 2 if N_ybins > 1 else N_ybins
+    N_zbins = unit_hist.GetNbinsZ() if hasattr(unit_hist, "GetNbinsZ") else 1
+    N_zbins = N_zbins + 2 if N_zbins > 1 else N_zbins
+    N_bins = N_xbins * N_ybins * N_zbins
+    # If we use the THnD then we have 'GetNbins' function instead
+    N_bins = unit_hist.GetNbins() if hasattr(unit_hist, "GetNbins") else N_bins


Suggested change

# Yes I know this is ugly

# Axis needs +2 for under/overflow, but only if the return is not 1!!!

# Was having issue with Z axis in 2D. We don't want to multiply by 3 if it's 2D

N_xbins = unit_hist.GetNbinsX() + 2

N_ybins = unit_hist.GetNbinsY() if hasattr(unit_hist, "GetNbinsY") else 1

N_ybins = N_ybins + 2 if N_ybins > 1 else N_ybins

N_zbins = unit_hist.GetNbinsZ() if hasattr(unit_hist, "GetNbinsZ") else 1

N_zbins = N_zbins + 2 if N_zbins > 1 else N_zbins

N_bins = N_xbins * N_ybins * N_zbins

# If we use the THnD then we have 'GetNbins' function instead

N_bins = unit_hist.GetNbins() if hasattr(unit_hist, "GetNbins") else N_bins

N_bins = unit_hist.GetNbins() if hasattr(unit_hist, "GetNbins") else unit_hist.GetNcells()

kandrosov · 2026-01-19T16:01:16Z

Common/HistHelper.py

+        # Create ROOT vectors
+        # Call the C++ function which returns a new histogram
+        new_hist = ROOT.analysis.rebinHistogramDict(
+            hist_initial, N_bins, y_bin_ranges, output_bin_edges_vec


Suggested change

hist_initial, N_bins, y_bin_ranges, output_bin_edges_vec

hist_initial, y_bin_ranges, output_bin_edges_vec

kandrosov · 2026-01-19T16:01:55Z

Common/HistHelper.py

+        N_xbins = hist_initial.GetNbinsX() + 2
+        N_ybins = hist_initial.GetNbinsY() if hasattr(hist_initial, "GetNbinsY") else 1
+        N_ybins = N_ybins + 2 if N_ybins > 1 else N_ybins
+        N_zbins = hist_initial.GetNbinsZ() if hasattr(hist_initial, "GetNbinsZ") else 1
+        N_zbins = N_zbins + 2 if N_zbins > 1 else N_zbins
+        N_bins = N_xbins * N_ybins * N_zbins
+        # If we use the THnD then we have 'GetNbins' function instead
+        N_bins = (
+            hist_initial.GetNbins() if hasattr(hist_initial, "GetNbins") else N_bins
+        )


Suggested change

N_xbins = hist_initial.GetNbinsX() + 2

N_ybins = hist_initial.GetNbinsY() if hasattr(hist_initial, "GetNbinsY") else 1

N_ybins = N_ybins + 2 if N_ybins > 1 else N_ybins

N_zbins = hist_initial.GetNbinsZ() if hasattr(hist_initial, "GetNbinsZ") else 1

N_zbins = N_zbins + 2 if N_zbins > 1 else N_zbins

N_bins = N_xbins * N_ybins * N_zbins

# If we use the THnD then we have 'GetNbins' function instead

N_bins = (

hist_initial.GetNbins() if hasattr(hist_initial, "GetNbins") else N_bins

)

kandrosov · 2026-01-19T16:02:49Z

Common/HistHelper.py

-    unit_bin_model = ROOT.RDF.TH1DModel(
-        "", "", model.fNbinsX, -0.5, model.fNbinsX - 0.5
-    )
+        print("nD histogram not implemented yet")


Suggested change

print("nD histogram not implemented yet")

raise RuntimeError("nD histogram not implemented yet")

kandrosov · 2026-01-19T16:06:25Z

Common/HistHelper.py

+        unit_bin_Inputs = [model.fNbinsX, -0.5, model.fNbinsX - 0.5]
+        unit_bin_model = ROOT.RDF.TH1DModel("", "", *unit_bin_Inputs)
+
+    elif dims == 2:


Code for 2D and 3D can be the same.

kandrosov · 2026-01-19T16:50:43Z

include/HistHelper.h

    };
-
+
+    TH1D* rebinHistogramDict(TH1* hist_initial, int N_bins, 


Suggested change

TH1D* rebinHistogramDict(TH1* hist_initial, int N_bins,

TH1D rebinHistogramDict(const TH1& hist_initial, int N_bins,

kandrosov · 2026-01-19T16:52:11Z

Common/HistHelper.py

+                step = (out_max - out_min) / n_out_bins
+                for i in range(n_out_bins + 1):
+                    out_edges.push_back(out_min + i * step)
+            output_bin_edges_vec.push_back(out_edges)


raise an error if edges do not start from 0

kandrosov · 2026-01-19T16:56:16Z

include/HistHelper.h

+        std::vector<float> all_output_edges;
+        float last_edge = 0.0;
+        for (const auto& edges : output_bin_edges) {
+            for (float edge : edges) {
+                all_output_edges.push_back(edge + last_edge);
+            }
+            last_edge = all_output_edges.back();
+        }
+        // Sort and remove duplicates
+        std::sort(all_output_edges.begin(), all_output_edges.end());


switch to bin numbering

kandrosov · 2026-01-19T17:04:26Z

include/HistHelper.h

+            // Set bin content and error
+            if (global_bin >= 1 && global_bin <= (int)all_output_edges.size() - 1) {
+                hist_output->SetBinContent(global_bin, hist_output->GetBinContent(global_bin) + bin_content);
+                hist_output->SetBinError(global_bin, std::sqrt(std::pow(hist_output->GetBinError(global_bin), 2) + bin_error2));


Don't do sqrt & square for each iteration. first accumulate err2 and then set all at the end.

Update to handle multiDim histograms

3471e6c

kandrosov requested a review from Copilot January 16, 2026 13:42

Copilot started reviewing on behalf of kandrosov January 16, 2026 13:43 View session

Copilot AI reviewed Jan 16, 2026

View reviewed changes

Merge

0dddc08

aebid force-pushed the Add_MultiDim_Plots branch from 76b6877 to 0dddc08 Compare January 16, 2026 14:56

Update flaf env again after reset-merge

929d7cb

Fixed some issues with flattened vars

eaa9bbf

Formatting

711b98c

Bug fix

bda3535

Remove debug prints

ffc7872

Remove debug

129d233

Fix bug with 1D string binning

c979734

Merge with FLAF204

516c860

Add this var_entry thing and remove import star because I hate it

a195bc4

Formatting and fixed 1 more var_entry

71939ef

aebid added 2 commits January 18, 2026 14:50

Update flaf env to get latest black

0811e7d

Remove hardcode black version

e53f55a

kandrosov requested changes Jan 19, 2026

View reviewed changes

	TH1D* hist_output = new TH1D("rebinned", "rebinned", all_output_edges.size() - 1, all_output_edges.data());
	TH1D* hist_output = new TH1D(hist_initial->GetName(), hist_initial->GetTitle(), all_output_edges.size() - 1, all_output_edges.data());

-    TH1D* rebinHistogramDict(TH1* hist_initial, int N_bins,
-                                const std::vector<std::pair<float, float>>& y_bin_ranges,
-                                const std::vector<std::vector<float>>& output_bin_edges) {
+    /**
+     * Rebin a 1D histogram using a dictionary of variable-width bin edges.
+     *
+     * The returned histogram is allocated with `new` and ownership is
+     * transferred to the caller. The caller is responsible for managing
+     * its lifetime, e.g. by:
+     *   - explicitly deleting it with `delete`, or
+     *   - adding it to a ROOT TDirectory/TFile so that ROOT manages
+     *     its deletion.
+     */
+    TH1D* rebinHistogramDict(TH1* hist_initial, int N_bins,
+                             const std::vector<std::pair<float, float>>& y_bin_ranges,
+                             const std::vector<std::vector<float>>& output_bin_edges) {

	hist_initial, N_bins, y_bin_ranges, output_bin_edges_vec
	hist_initial, y_bin_ranges, output_bin_edges_vec

	print("nD histogram not implemented yet")
	raise RuntimeError("nD histogram not implemented yet")

	TH1D* rebinHistogramDict(TH1* hist_initial, int N_bins,
	TH1D rebinHistogramDict(const TH1& hist_initial, int N_bins,

Update to handle multiDim histograms #207

Are you sure you want to change the base?

Update to handle multiDim histograms #207

Uh oh!

Conversation

aebid commented Jan 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

cms-flaf-bot commented Jan 16, 2026

Uh oh!

aebid commented Jan 17, 2026

Uh oh!

aebid commented Jan 17, 2026

Uh oh!