Incosistent data types and OverflowError

[konstantinos2018]

Relevant commit

Commit: a7e40d0

Dependencies

OS: Ubuntu 24.04
Python version: Python 3.13.1
Conda version: conda 24.9.2
mamba version: 2.0.5

Issue Description

I've encountered the following two Overflow errors and made some workarounds which made polymer to run properly, although I haven't tested whether the atmospherically corrected products are as they should be:

Error 1

Traceback (most recent call last):
  File "/home/kostageo/Projects/polymer/my_example.py", line 14, in <module>
    run_atm_corr(input_obj, output_obj, multiprocessing=0)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kostageo/Projects/polymer/polymer/main.py", line 522, in run_atm_corr
    for block in block_iter:
                 ^^^^^^^^^^
  File "/home/kostageo/Projects/polymer/polymer/main.py", line 438, in blockiterator
    for block in level1.blocks(params.bands_read()):
                 ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/home/kostageo/Projects/polymer/polymer/level1_msi.py", line 496, in blocks
    yield self.read_block(size, offset, bands_read)
          ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kostageo/Projects/polymer/polymer/level1_msi.py", line 397, in read_block
    Rtoa = (raw_data + self.radio_offset_list[iband])/self.quantif
            ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OverflowError: Python integer -1000 out of bounds for uint16

Workaround

Change "/home/kostageo/Projects/polymer/polymer/level1_msi.py", line 397 to
Rtoa = (raw_data.astype('float32') + self.radio_offset_list[iband])/self.quantif

Error 2

Traceback (most recent call last):
  File "/home/kostageo/Projects/polymer/my_example.py", line 14, in <module>
    run_atm_corr(input_obj, output_obj, multiprocessing=0)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kostageo/Projects/polymer/polymer/main.py", line 524, in run_atm_corr
    l2.write(block)
    ~~~~~~~~^^^^^^^
  File "/home/kostageo/Projects/polymer/polymer/level2.py", line 122, in write
    self.write_block(d, block[d], S,
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
                     block.attributes.get(d, {}))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kostageo/Projects/polymer/polymer/level2_nc.py", line 117, in write_block
    data[np.isnan(data)] = fill_value
    ~~~~^^^^^^^^^^^^^^^^
OverflowError: Python integer -32767 out of bounds for uint16

Workaround

Add before "/home/kostageo/Projects/polymer/polymer/level2_nc.py", line 117 the following
data = data.astype('float32')

[fsteinmetz]

Indeed, this seems to be a remaining incompatibility with numpy2.
As a workaround, you can also install an env with numpy<2.
I will fix this, thanks for the feedback.

[Novskilol]

also with product :
S2B_MSIL1C_20240415T165849_N0510_R069_T15UYS_20240415T203738.SAFE
I have a similar problem:
with these lines on variable "bitmask" inside level2_nc.py:

if self.format == 'NETCDF4_CLASSIC':
# data type conversion
if data.dtype == np.uint16:
typ = np.int16
elif data.dtype == np.uint32:
typ = np.int32
else:
typ = data.dtype
else:
typ = data.dtype

dtype is uint16, so typ = np.int16, fill_value becomes -32767

and this : data[np.isnan(data)] = fill_value tries to write -32767 in a uint16 dataset :

File "/usr/local/clemero/polymer/polymer/level2.py", line 122, in write
self.write_block(d, block[d], S,
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
block.attributes.get(d, {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/clemero/polymer/polymer/level2_nc.py", line 117, in write_block
data[np.isnan(data)] = fill_value
~~~~^^^^^^^^^^^^^^^^
OverflowError: Python integer -32767 out of bounds for uint16

[Novskilol]

"fixed" with
data = data.astype(typ)
data[np.isnan(data)] = fill_value

[fsteinmetz]

I have pushed a commit 87802ef which fixes Error 2.
The suggested workaround would convert bitmask to float, which should be avoided.