should be None per the docs, no? (unless I'm misunderstanding what your goal is with this operation). You want one column for each unique value of "variable" for a given ".chain" and ".iteration".

https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.pivot.html

For the actual pyrenew inference data object, but not for the other examples I have made, using None and not first produces this error (I originally used None before making a notebook & tests with the pyrenew-hew InferenceData object given to me):

FAILED tests/test_idata_to_tidy.py::test_posterior_predictive_group - polars.exceptions.ComputeError: found multiple elements in the same group, please specify an aggregation function

That suggests something is not correct upstream. Perhaps the variable name regex?

I believe the source of this issue is the line in col: col.split(", ")[1].strip("')")

.rename(
    {
        col: col.split(", ")[1].strip("')")
        for col in idata_df.columns
        if col.startswith(f"('{group}',")
    }
)

I changed

col: col.split(", ")[1].strip("')")

to

col: re.search(r",\s*'?(.+?)'?\)", col).group(1)

which I got incorrect a handful of times but which I believe works now.

The target of this regex looks like, e.g., "('posterior', 'alpha')".

In the expression, the , matches comma; in \s*, the \s is spaces and * for zero or more spaces; in '?, the ' means single quote and ? means optional (0 or 1 times); in (.+?), the ( and ) capture whatever is in the parenthesis with . means non-newline character, + means one or more characters, and ? (which I am not sure is necessary to include here) tries to get this capture in as small a job as possible (this capture will get alpha in the example target); the '? is the same as before; the \) is the single closing parenthesis after the group (the \ is needed for escaping parenthesis; and .group(1) in re gets the first item captured by (.+?).

If I am missing anything or wrote something inaccurately, please, reader, let me know.

The aggregate function is not successfully set to None and not first.

Can you write a test to demonstrate the case where the previous version was failing and the new version succeeds? They both work correctly on the example provided in your comment: "('posterior', 'alpha')".

I do understand the subtle difference between the two approaches, but I do not understand why the split approach did not work in practice.

The split approach did not work on the InferenceData object from pyrenew-hew in tests when the aggregate function was set to None but did work when the aggregate function was set to first. Yes, both the split and re approaches work on "('posterior', 'alpha')".

Thank you for comment + I will write test.

The split approach did not work on the InferenceData object from pyrenew-hew in tests when the aggregate function was set to None but did work when the aggregate function was set to first. Yes, both the split and re approaches work on "('posterior', 'alpha')".

I should probably investigate why, exactly, aggregate=None w/ split doesn't work for the Pyrenew inference data.

This is nice, but I think it's important also to check the correct tidying of array-valued parameters. The example idata and associated test from forecasttools-R are nice and should be easily portable to Python

https://github.com/CDCgov/forecasttools/blob/main/data-raw/ex_inferencedata_dataframe.R
https://github.com/CDCgov/forecasttools/blob/main/tests/testthat/test_inferencedata_dataframe_to_tidydraws.R

I will save this task for 2025-03-14.

Feels like this would be better handled by providing a lambda rename mapping to .rename() rather than via dictionary comprehension. See https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.rename.html

Hadn't thought of this. In the back of mind there are reservations I've seen from others that I haven't verified regarding about lambda mappings but they're not sufficient to not act here. Polars rename seems apt. Thank you for the quick comment.

I changed the code to:

.rename(
    lambda col: re.search(r",\s*'?(.+?)'?\)", col).group(1)
    if col.startswith(f"('{group}',")
    else col
)

got

forecasttools/idata_to_tidy.py:66:40: B023 Function definition does not bind loop variable `group`

then changed to:

.rename(
    lambda col, group=group: re.search(
        r",\s*'?(.+?)'?\)", col
    ).group(1)
    if col.startswith(f"('{group}',")
    else col
)

which works w/ tests & linting.

Commenting above for my future self.

1. Readability concern: at least two different senses of "group" present here.

2. I'd make it a one argument lambda and exclude it from linting. Making "group" an argument that defaults to the value of group satisfies the linter but hurts readability imo

forecasttools/idata_to_tidy.py:66:40: B023 Function definition does not bind loop variable `group`

Re: #36 (comment)

Will do. I agree, group=group seems extraneous but my usual response is to defer to the linter.