query_generation_per_plant() no longer working properly

[henrfj]

Seemingly due to a change in the entsoe web api https://web-api.tp.entsoe.eu/api - query_generation_per_plant() no longer seems to work as intended - for EntsoePandasClient: and the parser might be the problem.

I have tested the method using these control areas | BZs:

Area.NO, Area.NO_1, Area.NO_2, Area.NO_3, Area.NO_4, Area.NO_5
Area.SE, Area.SE_1, Area.SE_2, Area.SE_3, Area.SE_4
Area.DK, Area.DK_1, Area.DK_2

Testing different datetimes (1 day intervals) this year, last year, last month... as well as parameters psr_type, include_eic, eic_code (specified or left open) I always get:

• For control areas: ValueError: All objects passed were None
• For BZ: entsoe.exceptions.NoMatchingDataError

I then proceeded to test directly towards the entsoe web api, and found the possible reason.

• Calling the api with a BZ instead of a control area doesn't work; returning an empty xml file (empty as in no data, still some metadata). This might always have been the case, just wasn't very clear.

• Also, all data returned from the web api call will have the control-area as inBiddingZone_Domain.mRID (in the XML file returned), despite stating that its a BZ, which it isnt.

• The reason why query_generation_per_plant() still doesnt work - pandasClient, even when passing a control area, like Area.NO, might be due to a new format of the XML file. After the 15 min update xml files has gotten more generalized, with a period, resolution and interval, so that each datapoint doesn't come with its own timestamp. This seems to cause the parser to fail:

entsoe-py/entsoe/entsoe.py, line 2400-2438

...
df = parse_generation(text, per_plant=True, include_eic=include_eic)
...

A parser needs to look for period, start and resolution and use these to add timestamps to each point in a timeseries.

[henrfj]

Here is the parser I use, with some custom logic of course:

ns = {'ns': 'urn:iec62325.351:tc57wg16:451-6:generationloaddocument:3:0'}
root = ET.fromstring(xml_text)
records = []

for ts in root.findall('ns:TimeSeries', ns):
    plant_name = ts.findtext('ns:MktPSRType/ns:PowerSystemResources/ns:name', default='', namespaces=ns)
    in_bidding_zone = ts.findtext('ns:inBiddingZone_Domain.mRID', default='', namespaces=ns)
    registered_resource = ts.findtext('ns:registeredResource.mRID', default='', namespaces=ns)
    unit = ts.findtext('ns:quantity_Measure_Unit.name', default='', namespaces=ns)
    psr_type = ts.findtext('ns:MktPSRType/ns:psrType', default='', namespaces=ns)

    period = ts.find('ns:Period', ns)
    if period is None:
        continue
    start = period.findtext('ns:timeInterval/ns:start', default='', namespaces=ns)
    resolution = period.findtext('ns:resolution', default='', namespaces=ns)
    if not start or not resolution:
        continue

    # Parse start time
    start_time = pd.to_datetime(start)
    # Parse resolution (e.g., PT15M)
    if resolution.startswith('PT') and resolution.endswith('M'):
        minutes = int(resolution[2:-1])
        delta = timedelta(minutes=minutes)
    else:
        # Add more resolution parsing as needed
        delta = timedelta(minutes=15)

    for point in period.findall('ns:Point', ns):
        #pos = int(point.findtext('ns:position', default='1', namespaces=ns))
        pos_str = point.findtext('ns:position', default='1', namespaces=ns)
        pos = int(pos_str.replace(",", ""))
        quantity = float(point.findtext('ns:quantity', default='nan', namespaces=ns))
        timestamp = start_time + delta * (pos - 1)
        records.append({
            'plant_name': plant_name,
            'inBiddingZone_Domain.mRID': in_bidding_zone,
            'registeredResource.mRID': registered_resource,
            'quantity_Measure_Unit.name': unit,
            'psrType': psr_type,
            'timestamp': timestamp,
            'quantity': quantity
        })

df = pd.DataFrame(records)
return df

[emvollmer]

Hello,

I just started working with entso-e and this package yesterday, but also encountered error-related issues with query_generation_per_plant, so I thought I'd respond with my observations in case they are of help.

To test things, I looped through all codes in the Area class (results are at the bottom). For me, the call either works or it fails with a entsoe.exceptions.NoMatchingDataError or HTTPError error. The HTTPError is always along the lines of:

requests.exceptions.HTTPError: 400 Client Error:  for url: https://web-api.tp.entsoe.eu/api?documentType=A73&processType=A16&in_Domain=10YPL-AREA-----S&securityToken=<my-token>&periodStart=202407312200&periodEnd=202408012200

The ValueError you describe doesn't appear for me unless I try to use the EntsoePandasClient. It seems that that client fails to parse the results properly, because when I use the EntsoeRawClient to get the raw XML, convert it to a dict and build a df out of it, I get the desired content. Judging by your parsing logic though, it looks like you're also using the raw client, so I don't know why you're getting a ValueError when using f.e. country_code=Area.NO.

Since you specifically mention Norway as an example, I tested it and found that the following all work fine for me:

client = EntsoeRawClient(api_key=<my_api_key>)
xml_text = client.query_generation_per_plant('NO', start=pd.Timestamp('2024-08-01'), end=pd.Timestamp('2024-08-02'), psr_type=None)
xml_text = client.query_generation_per_plant(Area.NO, start=pd.Timestamp('2024-08-01'), end=pd.Timestamp('2024-08-02'), psr_type=None)
xml_text = client.query_generation_per_plant(Area.NO, start=pd.Timestamp('2024-08-01'), end=pd.Timestamp('2024-08-02'))

Using your logic, I get a df with the contents as expected:

            plant_name  ... quantity
0     Alta Krvg2 Hydro  ...   105.97
1     Alta Krvg2 Hydro  ...   106.00
...

It even works if I try to get the most current data from today (i.e. end=pd.Timestamp('2025-11-27')), so I can't seem to reproduce your ValueError.

However, I did observe some strange behavior when changing the year in the start and end definitions:

• <= 2019 -> raises a NoMatchingDataError
• 2020 -> raises a HTTPError
• >= 2021 -> works

The NoMatchingDataError is obviously due to there being no data available for Norway before 2020, but I don't know what's causing the HTTPError. When I check the Transparency Platform, there is in fact data present on the date in question in 2020. This affects various countries seemingly at random. F.e. for Poland, accessing data from 2024-08-01 raises the error (s. below), but 2023-08-01 works fine. It would be great to know what is causing this!

Here are the results from checking all country codes (for the time span '2024-08-01' to '2024-08-02'):

{
  "AL": "valid",
  "AT": "valid",
  "BA": "valid",
  "BE": "valid",
  "BG": "valid",
  "BY": "NoMatchingDataError",
  "CH": "valid",
  "CWE": "NoMatchingDataError",
  "CY": "NoMatchingDataError",
  "CZ": "valid",
  "CZ_DE_SK": "NoMatchingDataError",
  "DE": "NoMatchingDataError",
  "DE_50HZ": "valid",
  "DE_AMPRION": "valid",
  "DE_AMP_LU": "NoMatchingDataError",
  "DE_AT_LU": "NoMatchingDataError",
  "DE_LU": "NoMatchingDataError",
  "DE_TENNET": "valid",
  "DE_TRANSNET": "valid",
  "DK": "NoMatchingDataError",
  "DK_1": "NoMatchingDataError",
  "DK_1_NO_1": "NoMatchingDataError",
  "DK_2": "NoMatchingDataError",
  "DK_CA": "valid",
  "EE": "valid",
  "ES": "HTTPError",
  "FI": "valid",
  "FR": "HTTPError",
  "GB": "NoMatchingDataError",
  "GB_ELECLINK": "NoMatchingDataError",
  "GB_IFA": "NoMatchingDataError",
  "GB_IFA2": "NoMatchingDataError",
  "GB_NIR": "valid",
  "GE": "valid",
  "GR": "valid",
  "HR": "NoMatchingDataError",
  "HU": "valid",
  "IE": "valid",
  "IE_SEM": "NoMatchingDataError",
  "IS": "NoMatchingDataError",
  "IT": "valid",
  "IT_BRNN": "NoMatchingDataError",
  "IT_CALA": "NoMatchingDataError",
  "IT_CNOR": "NoMatchingDataError",
  "IT_CSUD": "NoMatchingDataError",
  "IT_FOGN": "NoMatchingDataError",
  "IT_GR": "NoMatchingDataError",
  "IT_MACRO_NORTH": "NoMatchingDataError",
  "IT_MACRO_SOUTH": "NoMatchingDataError",
  "IT_MALTA": "NoMatchingDataError",
  "IT_NORD": "NoMatchingDataError",
  "IT_NORD_AT": "NoMatchingDataError",
  "IT_NORD_CH": "NoMatchingDataError",
  "IT_NORD_FR": "NoMatchingDataError",
  "IT_NORD_SI": "NoMatchingDataError",
  "IT_PRGP": "NoMatchingDataError",
  "IT_ROSN": "NoMatchingDataError",
  "IT_SACO_AC": "NoMatchingDataError",
  "IT_SACO_DC": "NoMatchingDataError",
  "IT_SARD": "NoMatchingDataError",
  "IT_SICI": "NoMatchingDataError",
  "IT_SUD": "NoMatchingDataError",
  "LT": "valid",
  "LU": "NoMatchingDataError",
  "LV": "valid",
  "MD": "valid",
  "ME": "valid",
  "MK": "valid",
  "MT": "NoMatchingDataError",
  "NL": "valid",
  "NO": "valid",
  "NO_1": "NoMatchingDataError",
  "NO_1A": "NoMatchingDataError",
  "NO_2": "NoMatchingDataError",
  "NO_2A": "NoMatchingDataError",
  "NO_2_NSL": "NoMatchingDataError",
  "NO_3": "NoMatchingDataError",
  "NO_4": "NoMatchingDataError",
  "NO_5": "NoMatchingDataError",
  "PL": "HTTPError",
  "PL_CZ": "NoMatchingDataError",
  "PT": "valid",
  "RO": "valid",
  "RS": "valid",
  "RU": "NoMatchingDataError",
  "RU_KGD": "NoMatchingDataError",
  "SE": "valid",
  "SE_1": "NoMatchingDataError",
  "SE_2": "NoMatchingDataError",
  "SE_3": "NoMatchingDataError",
  "SE_4": "NoMatchingDataError",
  "SI": "valid",
  "SK": "valid",
  "TR": "NoMatchingDataError",
  "UA": "NoMatchingDataError",
  "UA_BEI": "NoMatchingDataError",
  "UA_DOBTPP": "NoMatchingDataError",
  "UA_IPS": "NoMatchingDataError",
  "UK": "NoMatchingDataError",
  "XK": "valid"
}

[henrfj]

Thanks for the in-depth comment - and yes, I was using the EntsoePandasClient for all my testing - and thats where I got the ValueError - and I agree, it seems to be from the parser used parse_generation - that doesnt handle the format of the xml response.

(will specify in my original text that I use EntsoePandasClient )

So, I also use the rawClient | or even just a direct api call to the web api, with the parser code I showed you above. So yes, I found a workaround - thanks.

When it comes to your HTTPError I have only experienced a 503 from time to time; do you get the 400 when using the raw client?

[emvollmer]

You're welcome :) Ah, I didn't realise the parser was your workaround - that explains it of course!

Yes, I always get the 400 HTTPError under the described circumstances (using the raw client) - so f.e. for the following method calls:

client = EntsoeRawClient(api_key=<my_api_key>)
client.query_generation_per_plant(Area.NO, start=pd.Timestamp('2020-08-01'), end=pd.Timestamp('2020-08-02'))
client.query_generation_per_plant(Area.NO, start=pd.Timestamp('2020-09-01'), end=pd.Timestamp('2020-09-02'))
...

I iterated through the previously posted code list for the 1st of August of the years 2020 to 2024. The following constellations raised that same error:

"ES": [2020, 2021, 2022, 2023, 2024],
"FR": [2020, 2021, 2022, 2023, 2024],
"GB": [2020],
"IT": [2020, 2021, 2022],
"NO": [2020],
"PL": [2024]

It doesn't seem to matter when the data is accessed - the error always appears for these. I don't know why it happens, seeing as the data is available on the Transparency Platform for those instances I've checked (i.e. France, Spain, Norway).

Did you by any chance try out one of these calls? Does the same thing happen for you?

For reference, the whole error is:

Traceback (most recent call last):
  File "/myprofile/myproject/.venv/lib/python3.9/site-packages/entsoe/entsoe.py", line 446, in query_generation_per_plant
    response = self._base_request(params=params, start=start, end=end)
  File "/myprofile/myproject/.venv/lib/python3.9/site-packages/entsoe/decorators.py", line 24, in retry_wrapper
    result = func(*args, **kwargs)
  File "/myprofile/myproject/.venv/lib/python3.9/site-packages/entsoe/entsoe.py", line 135, in _base_request
    raise e
  File "/myprofile/myproject/.venv/lib/python3.9/site-packages/entsoe/entsoe.py", line 109, in _base_request
    response.raise_for_status()
  File "/myprofile/myproject/.venv/lib/python3.9/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error:  for url: https://web-api.tp.entsoe.eu/api?documentType=A73&processType=A16&in_Domain=10YFR-RTE------C&securityToken=<my-token>&periodStart=202508010000&periodEnd=202508020000