Expose PyUnicode_ToLower and friends

[lysnikolaou]

Hey folks! 👋

This is about python/cpython#76535 (Unclear intention of deprecating Py_UNICODE_TOLOWER / Py_UNICODE_TOUPPER). As far as I understand, these were deprecated in the versions 3.3 - 3.12, but were undeprecated in 3.13. However, they're still pretty unusable, because of the fact that changing the case of a Unicode character might lead to two or more characters.

There's alternative APIs for this that do the right thing, _PyUnicode_ToLowerFull and friends. I'm suggesting to rename these to PyUnicode_ToLower etc. and expose them as public API. An implementation can be found on python/cpython#136176.

Some context: I'm working on making NumPy string array operations faster. Up until now, these were working by converting the array item (a UCS4 buffer) to a PyObject, calling the corresponding str method and then converting the result back to a UCS4 buffer to put back into the array. Instead of doing this, we're writing fast string ufuncs that operate on the buffer without having to do the back-and-forth with PyObjects. We've been able to do this with almost all of the string operations by traversing the array, then traversing the UCS4 buffer and doing the operation codepoint by codepoint. The only ones missing right now are lower and friends, because we don't have a good way of doing that same thing.

[encukou]

Works for me, but IMO

• the names should advertise that the functions don't work on str objects. Perhaps PyUnicode_CodepointToLower or even PyUCS4_ToLower.

• Should we should also expose the buffer size needed to do any of these operations on any character (i.e. 3 for Unicode 16.0)? This can't be part of the limited API, but might be quite helpful elsewhere.

[lysnikolaou]

Thanks @encukou for the quick response on this!

• the names should advertise that the functions don't work on str objects. Perhaps PyUnicode_CodepointToLower or even PyUCS4_ToLower.

I like PyUCS4_ToLower. If everyone agrees, I can rename the new APIs to that.

• Should we should also expose the buffer size needed to do any of these operations on any character (i.e. 3 for Unicode 16.0)? This can't be part of the limited API, but might be quite helpful elsewhere.

@serhiy-storchaka proposed this as well if I remember correctly. Should we add macros that define the buffer size?

[encukou]

IMO, a single #define PyUCS4_CASE_CONVERSION_BUFFER_SIZE 3 would do. There's not much to be gained in making it specific to each function.

[malemburg]

I like PyUCS4_ToLower. If everyone agrees, I can rename the new APIs to that.

Same here.

[lysnikolaou]

Addressed all feedback on the PR.

[encukou]

Add PyUCS4_ToLower, PyUCS4_ToUpper, PyUCS4_ToTitle, PyUCS4_ToFolded, PyUCS4_CASE_CONVERSION_BUFFER_SIZE

• @encukou
• @erlend-aasland
• @mdboom
• @serhiy-storchaka
• @vstinner
• @zooba

[ngoldbaum]

With my NumPy maintainer hat on: it'd be really nice to be able to add fast implementations for np.strings.upper() and np.strings.lower(). What we currently do is very suboptimal.

[serhiy-storchaka]

These functions will be called in a tight loop. Passing an extra argument, testing it in the function, and checking the result after each function call will add unnecessary overhead. Changing the result type from int to Py_ssize_t can also affect this.

And since a these functions are called in a tight loop in the implementations of the string methods, this will affect each program.

There is no need to do all these checks for each character in a loop. The maximal size of the buffer is small, just use the buffer of the maximal capacity. Check it's size outside of the loop. To make it future proof, we should provide the maximal buffer size as a macro and and as a function. You can create a buffer of static size and then check that its size matches the runtime size.

In summary, I propose to return to the original interface of private functions and provide a macro and a function for the maximal buffer size.

[vstinner]

What is the API of PyUCS4_ToLower()?

[encukou]

From the PR:

Py_ssize_t PyUCS4_ToLower(Py_UCS4 ch, Py_UCS4 *buffer, Py_ssize_t size)
Py_ssize_t PyUCS4_ToUpper(Py_UCS4 ch, Py_UCS4 *buffer, Py_ssize_t size)
Py_ssize_t PyUCS4_ToTitle(Py_UCS4 ch, Py_UCS4 *buffer, Py_ssize_t size)
Py_ssize_t PyUCS4_ToFolded(Py_UCS4 ch, Py_UCS4 *buffer, Py_ssize_t size)
#define PyUCS4_CASE_CONVERSION_BUFFER_SIZE 3

@lysnikolaou, would the speedup from not checking on each call be noticeable?

[vstinner]

Passing an extra argument, testing it in the function, and checking the result after each function call will add unnecessary overhead.

I don't think that it's a good API if we don't check the size (ex: Py_ssize_t PyUCS4_ToLower(Py_UCS4 ch, Py_UCS4 *buffer)). It can lead to buffer overflow, no? I prefer to offer a safe API.

[encukou]

@ngoldbaum: Are you able to check if the size check adds significant overhead?

[vstinner]

I ran benchmarks: see python/cpython#136176 (comment)

[ngoldbaum]

Are you able to check if the size check adds significant overhead?

Not easily - I'd need to implement ufuncs in NumPy that depend on a patched version of CPython.

But I also don't think size checks would be a big deal. To give you an idea of how slow our current implementation is:

In [4]: chars = list(string.ascii_letters + string.digits)
   ...: chars = np.array(chars, dtype="U1")
   ...: ret = np.random.choice(chars, size=100 * 10000, replace=True)
   ...: random_string_list = ret.view("U100")

In [5]: random_string_list.shape
Out[5]: (10000,)

In [6]: %timeit np.strings.rstrip(random_string_list)
193 μs ± 683 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [7]: %timeit np.strings.upper(random_string_list)
2.01 ms ± 6.97 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The current implementation needs to individually convert each array item to a PyStr, call str.upper() on that string, then convert the result back to a numpy unicode string.

Comparing with rstrip, which is implemented as a ufunc, you can see that all that copying adds a lot of unnecessary overhead.

We could of course add our own unicode data to NumPy and do operations like capitalization using that, but then we would have to be very careful to make sure we're doing exactly the same thing as CPython (if we care about that). If we get an API in CPython we can use like the other string operations, we get that "for free".

We did actually copy the string searching utilities from CPython, converted them to C++, and added them to NumPy. But I'd prefer not to do that more of that than we already did.