Skip to content

String dtype: keep select_dtypes(include=object) selecting string columns #62323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 21, 2025
Merged

String dtype: keep select_dtypes(include=object) selecting string columns #62323

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
866075d
String dtype: keep select_dtypes(include=object) selecting string col…
jorisvandenbossche Sep 12, 2025
535d121
Merge remote-tracking branch 'upstream/main' into string-dtype-select…
jorisvandenbossche Sep 21, 2025
46c0731
add whatsnew
jorisvandenbossche Sep 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions doc/source/whatsnew/v2.3.3.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,16 @@ Most changes in this release are related to :class:`StringDtype` which will
become the default string dtype in pandas 3.0. See
:ref:`whatsnew_230.upcoming_changes` for more details.

.. _whatsnew_233.string_fixes.improvements:

Improvements
^^^^^^^^^^^^
- Update :meth:`DataFrame.select_dtypes` to keep selecting ``str`` columns when
specifying ``include=["object"]`` for backwards compatibility. In a future
release, this will be deprecated and code for pandas 3+ should be updated to
do ``include=["str"]`` (:issue:`61916`)


.. _whatsnew_233.string_fixes.bugs:

Bug fixes
Expand Down
4 changes: 3 additions & 1 deletion pandas/core/dtypes/cast.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -865,7 +865,9 @@ def invalidate_string_dtypes(dtype_set: set[DtypeObj]) -> None:
np.dtype("<U").type, # type: ignore[arg-type]
}
if non_string_dtypes != dtype_set:
raise TypeError("string dtypes are not allowed, use 'object' instead")
raise TypeError(
"numpy string dtypes are not allowed, use 'str' or 'object' instead"
)


def coerce_indexer_dtype(indexer, categories) -> np.ndarray:
Expand Down
12 changes: 8 additions & 4 deletions pandas/core/frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5150,10 +5150,14 @@ def check_int_infer_dtype(dtypes):
def dtype_predicate(dtype: DtypeObj, dtypes_set) -> bool:
# GH 46870: BooleanDtype._is_numeric == True but should be excluded
dtype = dtype if not isinstance(dtype, ArrowDtype) else dtype.numpy_dtype
return issubclass(dtype.type, tuple(dtypes_set)) or (
np.number in dtypes_set
and getattr(dtype, "_is_numeric", False)
and not is_bool_dtype(dtype)
return (
issubclass(dtype.type, tuple(dtypes_set))
or (
np.number in dtypes_set
and getattr(dtype, "_is_numeric", False)
and not is_bool_dtype(dtype)
)
or (dtype.type is str and np.object_ in dtypes_set)
)

def predicate(arr: ArrayLike) -> bool:
Expand Down
18 changes: 10 additions & 8 deletions pandas/tests/frame/methods/test_select_dtypes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,10 @@ def test_select_dtypes_include_using_list_like(self, using_infer_string):
ri = df.select_dtypes(include=[str])
tm.assert_frame_equal(ri, ei)

ri = df.select_dtypes(include=["object"])
ei = df[["a"]]
tm.assert_frame_equal(ri, ei)

def test_select_dtypes_exclude_using_list_like(self):
df = DataFrame(
{
Expand Down Expand Up @@ -309,17 +313,15 @@ def test_select_dtypes_not_an_attr_but_still_valid_dtype(self, using_infer_strin
df["g"] = df.f.diff()
assert not hasattr(np, "u8")
r = df.select_dtypes(include=["i8", "O"], exclude=["timedelta"])
if using_infer_string:
e = df[["b"]]
else:
e = df[["a", "b"]]
# if using_infer_string:
# TODO warn
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the issue you said And in any case, we should probably still add a warning to pandas 2.3 about this when the string mode is enabled (for if we do a 2.3.2 release)

so this TODO should this be part of this PR?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR just restores the old behaviour, and in that case we don't need to warn, I think. We should add a warning that "object" will stop selecting string columns at some point, but in any case I want to only do that in a later PR because that is a lot more complicated.

It was for the case that we decided to keep the current-main behaviour of object not selecting string columns that we should have definitely added a warning to warn users that they are not getting the result they are expecting.

e = df[["a", "b"]]
tm.assert_frame_equal(r, e)

r = df.select_dtypes(include=["i8", "O", "timedelta64[ns]"])
if using_infer_string:
e = df[["b", "g"]]
else:
e = df[["a", "b", "g"]]
# if using_infer_string:
# TODO warn
e = df[["a", "b", "g"]]
tm.assert_frame_equal(r, e)

def test_select_dtypes_empty(self):
Expand Down
Loading