403 Error - The request could not be satisfied

[duncanjbain]

Hi there,

Just got this up and running, but I immediately get a 403 error stating the request could not be satisfied, from Cloudflare. Any ideas? Perhaps Twickets can somehow detect it's not an actual browser?

Any help appreciated!

Cheers,

Duncan

[duncanjbain]

On second thoughts, is this normal, and I'm only seeing the occasional request that's being blocked rather than them all being blocked?

[duncanjbain]

Had a look around the twigot repo, and it looks like it might be due to having count=100 in:

https://www.twickets.live/services/catalogue?q=countryCode=GB&count=100&api_key=83d6......

as this url string gives the same error in my local browser, but when this is reduced to 10 it seems to work fine.

[xxgfxx]

Had a look around the twigot repo, and it looks like it might be due to having count=100 in:

https://www.twickets.live/services/catalogue?q=countryCode=GB&count=100&api_key=83d6......

as this url string gives the same error in my local browser, but when this is reduced to 10 it seems to work fine.

In which file did you find this string of code? I have the same error.

Although I notice if I change the URL in the browser from 100 to 10, it does load but not properly. So not sure if it's a fix?

[duncanjbain]

Had a look around the twigot repo, and it looks like it might be due to having count=100 in:
https://www.twickets.live/services/catalogue?q=countryCode=GB&count=100&api_key=83d6......
as this url string gives the same error in my local browser, but when this is reduced to 10 it seems to work fine.

In which file did you find this string of code? I have the same error.

Although I notice if I change the URL in the browser from 100 to 10, it does load but not properly. So not sure if it's a fix?

It's actually just an example in a comment, if you look in main.go in this repo there's the following:

const (
	maxNumTickets = 250
......
)

Seems to work changing this to 10, and then just running the app outside of docker for now. I spent a few hours trying different things earlier this afternoon, so can't remember exactly what worked. I've got the Docker image running currently though on a VPS as well, which I don't think I've made any changes to do, so maybe it's fixed itself.

[ahobsonsayers]

Hey, sorry for the delay getting back to you! Thanks for looking into this!

This is interesting as we should never be fetching more that 10 tickets at a time for each request, so the url query param should indeed be 10, not 100. I thought this was the case (it definitely used to be), but this might be a bug I introduced recently...

Let me look into this! I really appreciate your patience.

[ahobsonsayers]

Hey, ok, I've had a look.

We are indeed still only getting 10 tickets at a time, so the query parameter will be 10 in the request URL we use.

The maximum ticket number of 250 (in the variable you showed) is the maximum number of tickets we can theoretically get, given the number retrieved per request.

So, for example, if we get 10 tickets per request, and we have a maximum number of 250, we will make a maximum of 25 successive requests.

Of course, if there are fewer than 250 tickets since the last ticket we saw, it will only retrieve up to the number that have been listed, no more.

Usually, a maximum of around 20/30 tickets get listed between polls, so typically, we will only need to make 1-3 requests.

It could be that the second/third request it makes is causing the error you see - we dont wait long enough between calls, and we hit a rate limit.

Regardless, you do not want to set the maximum number to 10 instead of 250. If you do this, you will absolutely miss tickets if more than 10 are listed between polls - which is often the case.

We have two possible solutions, I guess:

• Reduce polling time (or make it configurable) so we never need to make two very quick consecutive requests. Maybe every 30 seconds? I don't think this will help though, as twickets seem to batch add tickets to the feed all at once, so I think our poll time is irrelevant.
• Add a slight delay between the successive requests when fetching all tickets listed since last fetch. Maybe half a second?

As an aside - I don't see this issue on my side - I am wondering how our setups differ

[duncanjbain]

Hey, no worries and thanks for the response!

Ah that makes sense re the max ticket variable, I better change it back or just re-pull the Docker image and run that instead then! It is weird why mine wasn't working in the morning, I saw the link to the Telegram channel for Oasis tickets and joined it roughly when I was having issues, not sure if it was just a coincidence but noticed there were a lack of Oasis ticket alerts roughly at the same time I was struggling with the 403 errors, wonder if it was a Twickets issue that cleared up?

I wasn't getting any ntfy or telegram notifications either from my instance so maybe I'll give up and just stick with your channel for now, although even after clicking the link less than 10 seconds after the notification I'm still not having any luck! :(

Edit: Although at 20:45 UK time I seem to be getting the 403's again.

[ahobsonsayers]

I'll do some testing tomorrow and see if I can make some changes that might be able to help reduce the 403's (by introducing a very small sleep between requests)

As for not having any luck with Oasis tickets - I am very sorry 😔 Sadly I have also noticed this and have come to the conclusion there are others out there who have built something similar to me, but have 1. done it in private and 2. been unethical with it and created bots to auto buy. I'm not willing to be part of the problem on this front though (damn those pesky morals) so I guess we are doomed to try clicking the notification fast enough to get in before the bots. Sorry about that!

Edit: as of 11.30pm I have checked my instance and am also getting blocked by cloudfront. Will see what I can do about it. They might have upped their security.

[ahobsonsayers]

Not sure why this was closed sorry - i think i have solved this but will wait for testing by people in this issue before we close

[ahobsonsayers]

Should be fixed in v0.8.0 https://github.com/ahobsonsayers/twitchets/releases/tag/v0.8.0

🤞