Meta issue for JSONDecode errors we see often lately

[JOJ0]

Several users have reported similar randomly happening JSONDecode errors when retrieving arbitrary stuff via python3-discogs-client. I experienced it myself and post my experience soon. I'd like to summarize discussion and improvement suggestions in this issue here.

In my opinion it looks like api.discogs.com responds with unexpected data but only sometimes. I would guess it is a problem on their end for whatever reason, and they might be aware of and working on it already, or not..... In the end we have to live with it and find a workaround.

A second reason is that our backoff mechanism could be faulty or not compatible anymore since something might have changed at Discogs. It worked very well for us so far and it's in place since ...about 4 years already....#34

Related reports so far:

• JSON.decode errors #153
• Problems with main release #162
• empty json request when accessing information #26

Did any of you solve the issue simply with try/except catchin jsondecode errors? If yes, share you experience, maybe a simple fix like that can be included in python3-discogs-client already.

Can this problem be reproduced when using api.discogs.com simply with Python requests module only?

Discogs reports outages publicly here: https://status.discogs.com/, do they maybe correlate with issues like that?

[JOJ0]

This is what I do to catch such errors in an application using discogs_client. For example this loops through my sales inventory and imports into an sqlite database. Instead of crashing I'd rather have missing entries. This also includes an error counter, since I currently want to learn how often these errors happen.

            for listing in self.collection.me.inventory:
                try:
                    self.collection.create_sales_entry({
                        "d_sales_release_id": listing.release.id,
                        "d_sales_listing_id": listing.id,
                        "d_sales_release_url": listing.release.url,
                    })
                except JSONDecodeError as e:
                    log.error("Catched a JSONDecodeError. Not retrying! %s", e)
                    decode_err += 1
                except Exception as e:
                    log.error("Catched an Exception. Not retrying! %s", e)
                    other_err += 1

        print(f"Discogs JSONDecode errors : {decode_err}.")
        print(f"Other errors : {other_err}.")

I'm not 100% sure since this error is hard to reproduce but I'm pretty sure this must catch JSONDecodeErrors, no matter where they happen!

[JOJ0]

@AnssiAhola do you think we could include the catching of the JSONDecodeError (only! not other exceptions!) into the core of the client?

I'm not sure if simply wrapping the "main fetch wrapper":

discogs_client/discogs_client/fetchers.py

Lines 57 to 61 in 98059c0

	return request(
	method=method, url=url, data=data,
	headers=headers, params=params,
	timeout=(self.connect_timeout, self.read_timeout)
	)

in a try except is the best idea or an approach like the backoff decorator is a better solution:

discogs_client/discogs_client/utils.py

Lines 44 to 48 in 98059c0

	def backoff(f):
	"""
	Wraps the request method of the Fetcher class to provide
	exponential backoff if rate limit is hit.
	"""

or is all this a bad idea anyway? And of course we should add tests for it!

[AnssiAhola]

Sorry, completely missed this!

I think we would need to figure out how this would impact the client as a whole, for example if we catch this error and return None or empty object, would it "break" something elsewhere or would it result in data that is in invalid state, like an Release instance with all properties set to None?

Documenting that this exception could be thrown, in the doc string, would also help a lot even if it doesn't fix the issue.

But overall I'm not against this at all.

[JOJ0]

@AnssiAhola this kind of got lower priority at least in my personal opinion and personal experience. It seems like these were temporary issues with api.discogs.com. I don't see these errors at all anymore.

But nevertheless if this pops up again and people report it here, we should further work on it. For now, I'm waiting for reports or occurences happening in my own Discogs API use-cases.

Thanks for your opinion and ideas so far!!!

[JOJ0]

Back in 2021 we had an issue like that already: #26

[JOJ0]

I just discovered that Discogs actually has a very detailed "outages" website. It would be interesting to see if JSONDecode errors correlate with reported "downs" of parts of their service: https://status.discogs.com/

[romainhild]

I encountered this issue yesterday and the day before. According to the status page, there was no outage on discogs.
I ended up adding a retry when catching this error, and it seems to be enough for now, since I haven't seen this again.