Not checking for None return from getlocale() in python/tmdb3/tmdb3/lookup.py

[mcampin]

• Linux 6.14.0-33-generic Have mythtranscode report aac_latm #33~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 17:02:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux:

• MythTV 34:

• mythtv-34.0+fixes.20240210.e3e165a1:

• tmdb3/Python:

What steps will reproduce the bug?

I was able to reproduce this failure outside of mythtv using the following:
unset LANG
export LC_ALL=C
/usr/share/mythtv/metadata/Movie/tmdb3.py -l en -a US -D 575265

Note: export LC_ALL=C.utf-8 works.

How often does it reproduce? Is there a required condition?

It happens for retrieving details for movies. When this happens the code falls through to series, so retrieving details for series/TV works.

What is the expected behaviour?

Works when locale is "C".

What do you see instead?

Additional information

[mcampin]

Darn it. A bunch of information is missing.

$ /usr/share/mythtv/metadata/Movie/tmdb3.py -l en -a US -D 575265
Traceback (most recent call last):
File "/usr/share/mythtv/metadata/Movie/tmdb3.py", line 150, in
sys.exit(main("movie",'tmdb3.py'))
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/mythtv/metadata/Movie/tmdb3.py", line 127, in main
xml = buildSingle(args[0], opts)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/MythTV/tmdb3/lookup.py", line 156, in buildSingle
if "_" in s_locale[0]:
^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 228, in partial_apport_excepthook
return apport_excepthook(binary, exc_type, exc_obj, exc_tb)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 144, in apport_excepthook
os.open(pr_filename, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), "wb"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/var/crash/_usr_share_mythtv_metadata_Movie_tmdb3.py.6663.crash'

Original exception was:
Traceback (most recent call last):
File "/usr/share/mythtv/metadata/Movie/tmdb3.py", line 150, in
sys.exit(main("movie",'tmdb3.py'))
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/share/mythtv/metadata/Movie/tmdb3.py", line 127, in main
xml = buildSingle(args[0], opts)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/MythTV/tmdb3/lookup.py", line 156, in buildSingle
if "_" in s_locale[0]:
^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

[mcampin]

To debug this, I commented out the code in mythsystemunix.cpp that redirected stderr to /dev/null in the fork function. It's much easier to see the error using the steps to reproduce above.

I'm still investigating why the environment is different when it's being called from mythfrontend. tmdb3.py runs fine in a terminal window unless I explicitly modify LC_ALL.

[mcampin]

I backported this section from git. It works if I add a test for None:

if s_locale[0] is not None and "_" in s_locale[0]:
    system_language = s_locale[0].split("_")[0]
    system_country = s_locale[0].split("_")[1]
else:
    system_language = 'en'
    system_country = 'US'

[mcampin]

Ubuntu 24.04 LTS
python3 3.12.3-0ubuntu2
libmyth-python 2:34.0+fixes.20240210.e3e165a1-0ubuntu6

[mcampin]

According to https://docs.python.org/3/library/locale.html:

locale.getlocale(category=LC_CTYPE)
    ... The “C” locale is represented as (None, None).

[rcrdnalor]

The traceback from your previous comment

Original exception was: Traceback (most recent call last): File "/usr/share/mythtv/metadata/Movie/tmdb3.py", line 150, in sys.exit(main("movie",'tmdb3.py')) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/share/mythtv/metadata/Movie/tmdb3.py", line 127, in main xml = buildSingle(args[0], opts) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/MythTV/tmdb3/lookup.py", line 156, in buildSingle if "_" in s_locale[0]: ^^^^^^^^^^^^^^^^^^ TypeError: argument of type 'NoneType' is not iterable

is not from MythTV/fixes/34. The code in question was updated in MythTV v35 by Commit 33d6644.

[mcampin]

Open your commit 33d6644 link and select lookup.py. The statement 'if "_" in s_locale[0]:' on line 153 fails if s_locale = (None, None). This is the expected value when LC_ALL = 'C'.

As I previously mentioned, it works if LC_ALL = 'C.UTF-8'.

The line number in my pasted error message is slightly different because I backported it from commit ea3399d to my Ubuntu 24.04 LTS package. In either case, the code appear to be the same.

[rcrdnalor]

Mythcorecontext greets you with a big fat warning if you run a mythtv application without utf-8 context:

mythtv/mythtv/libs/libmythbase/mythcorecontext.cpp

Line 272 in ea3399d

LOG(VB_GENERAL, LOG_WARNING, QString("This application expects to "

Do you see this warning in the logs?
Please post the output of
locale and locale -a

[mcampin]

I'd like to focus on the original issue.

It can be reproduced by export LC_ALL=C and running a query like "tmdb3.py -l en -a US -D 575265", and the code doesn't check the return from getlocale()/getdefaultlocale() for None resulting in a crash. This can easily be fixed by adding something like "s_locale[0] is not None and" to the if-statement.

Code shouldn't crash because it doesn't check a return value. This crash was difficult to debug because the fork() redirects stderr to /dev/null resulting in no crash message.

There is a locale warning in mythfrontend.log. However, it's unlikely someone would associate a failure to "Retrieve Details" for movies-only with it. If the policy is to not handle non-utf8 locales then this should be an error instead of a warning.

As to why I set LC_All=C, I had issues with another program in the past (which is beyond the scope of this issue).