`pandas` v3 Chained Assignment future proofing

[hsteptoe]

📰 Custom Issue

A note that iris.pandas will need to be updated to account for breaking changes expected in pandas v3 in the way it handles chained assignments.

Helpfully, tests currently run with the following warnings and suggested changes:

FutureWarning: ChainedAssignmentError: behaviour will change in pandas 3.0!
  You are setting values through chained assignment. Currently this works in certain cases, but when using Copy-on-Write (which will become the default behaviour in pandas 3.0) this will never work to update the original DataFrame or Series, because the intermediate object on which we are setting values will behave as a copy.
  A typical example is when you are setting values in a column of a DataFrame, like:
  
  df["col"][row_indexer] = value
  
  Use `df.loc[row_indexer, "col"] = values` instead, to perform the assignment in a single step and ensure this keeps updating the original `df`.

[trexfeathers]

Pandas v3 is here

[trexfeathers]

Our assumption here was that we would just need to adapt some lines of code. Unfortunately it is not as simple as that: iris.pandas relies heavily on the user having the choice for the Cube and DataFrame to share the same NumPy array - the copy parameter. This flies directly against the new Pandas philosophy, where such behaviour is considered misleading/dangerous¹.

iris/lib/iris/pandas.py

Lines 634 to 637 in fe7a4d7

	copy : bool, default=True
	Whether the :class:`pandas.DataFrame` is a copy of the the Cube
	:attr:`~iris.cube.Cube.data`. This option is provided to help with memory
	size concerns.

We believe strongly in the 'thin wrapper' concept, where we directly expose the behaviour of our dependencies as much as possible. So I think we are forced to refactor iris.pandas to no longer offer the switchable copy behaviour. I guess we raise an error or warning whenever copy=False. Thankfully True is already the default. The only choice is:

• Immediate refactor with announcement in Iris 3.15
• Pin pandas<3 for Iris 3.15, with a warning that things will change in Iris 3.16

@scitools/iris-devs thoughts?

Footnotes

1. pandas - Python Data Analysis Library ↩

[hsteptoe]

@trexfeathers In case this helps with a Peloton decision, I'm happy to action the necessary changes...

[trexfeathers]

From @SciTools/peloton: decision made: implement the fixes in Iris 3.15.

• Remove the temporary pandas<3 pin introduced in Pin pandas<3 and update lock files #6918
• Obviously fix any chained assignment
• Keep the copy parameter, force it to always be True, raise a Warning if the user ever says copy=False
• DeprecationWarning that the copy parameter is being removed IF anyone explicitly sets it

@hsteptoe are you good to implement this before 17th March?

[hsteptoe]

Yep, I should be able to work to that deadline 🫡