indexing with a `Categorical` of `Interval`s is inefficient

[flying-sheep]

This line converts the IntervalIndex into a numpy object array:

pandas/pandas/core/dtypes/dtypes.py

Line 520 in faf3bbb

cat_array = hash_array(np.asarray(categories), categorize=False)

then in this block, a TypeError is raised and causes that object array to be converted into strings:

TypeError: (-0.00872, 0.439] of type <class 'pandas._libs.interval.Interval'> is not a valid type for hashing, must be string or null

pandas/pandas/core/util/hashing.py

Lines 333 to 339 in faf3bbb

	try:
	vals = hash_object_array(vals, hash_key, encoding)
	except TypeError:
	# we have mixed types
	vals = hash_object_array(
	vals.astype(str).astype(object), hash_key, encoding
	)

[jbrockmendel]

Does hash array get called in indexing?

[flying-sheep]

Yeah, when the indexed data frame’s .index is unique:

import pandas as pd

df = pd.DataFrame(dict(a=range(3)), pd.cut(range(3), 3))
assert df.index.is_unique  # bug only triggers if this is the case

df.loc[df.index.categories[:2]]

set a breakpoint in the except TypeError branch in _hash_ndarray and execute the above in a debugger, and the breakpoint will be hit.

I discovered this because in some older versions of pandas or numpy, the vals.astype(str).astype(object) raises a RuntimeWarning about “invalid values encountered in cast”. This no longer happens, but I think the casting should probably not happen here.

[jbrockmendel]

Looks like in a .equals check we go through categories_match_up_to_permutation, which checks the hash of each dtype, which goes through path in the OP.