fix: number of google pages results, limited perms, improvements from WCC

[MQ37]

closes #80

⚠️ MERGE ONLY AFTER #82 IS MERGED ⚠️

Fixes the google pages number of results scraping. Instead of using the ?num param that is now ignored by the google it uses the ?start for offset pagination to gather the google result pages urls. Implementation based on https://github.com/apify-store/google-search

[jirispilka]

if possible I would use reference implementation, I think it is better to have one source of truth

These are LLM complains:
The reference implementation is more robust because it:

• Validates pagination links exist in HTML
• Tracks page numbers to avoid non-existent pages
• Uses multiple fallback conditions
• Handles Google's inconsistent pagination behavior

[matyascimbulka]

I like moving the validation of the results to the collection also the move to fix pagination looks good. But I have found some duplicate code and some opportunities to make the code cleaner.

[MQ37]

I like moving the validation of the results to the collection also the move to fix pagination looks good. But I have found some duplicate code and some opportunities to make the code cleaner.

Thank you for the review, I think I addressed your comments 👍

[MQ37]

if possible I would use reference implementation, I think it is better to have one source of truth

These are LLM complains: The reference implementation is more robust because it:

* Validates pagination links exist in HTML

* Tracks page numbers to avoid non-existent pages

* Uses multiple fallback conditions

* Handles Google's inconsistent pagination behavior

Regarding the "reference" implementation Ondra mentioned that the page tracking logic is legacy and they just left it there, so our simpler implementation should be fine.

[matyascimbulka]

Looks better but still found some issues that would most likely cause issues down the line. It would be good to see some test runs of the entire PR stack.

[matyascimbulka]

I just realize that my comments about it failing are wrong. Still it be nice to see some test runs.

[jirispilka]

Regarding the "reference" implementation Ondra mentioned that the page tracking logic is legacy and they just left it there, so our simpler implementation should be fine.

Ok, thanks for the clarification. IMHO it is always better to ask than miss some edge cases

[jirispilka]

@matyascimbulka thanks for the comments. I appreciate it. I'll leave this to you two :)

[MQ37]

I just realize that my comments about it failing are wrong. Still it be nice to see some test runs.

Here is few of my test runs:

• basic run Cheerio - https://console.apify.com/view/runs/Z50RSxXrERLUGulIC
• basic run Playwright - https://console.apify.com/view/runs/EuFSPFMcLoQghzRMN
• run to test the google serp pages crawling with obscure search term - https://console.apify.com/view/runs/FwSzRAAsyEhBE8RCJ

Then also tested some edge cases and I haven't encountered any issues with the google serp crawling - feel free to test.

[MQ37]

I just realize that my comments about it failing are wrong. Still it be nice to see some test runs.

Here is few of my test runs:

* basic run Cheerio - https://console.apify.com/view/runs/Z50RSxXrERLUGulIC

* basic run Playwright - https://console.apify.com/view/runs/EuFSPFMcLoQghzRMN

* run to test the google serp pages crawling with obscure search term - https://console.apify.com/view/runs/FwSzRAAsyEhBE8RCJ

Then also tested some edge cases and I haven't encountered any issues with the google serp crawling - feel free to test.

@matyascimbulka it is now broken because of the PR I merged here, working on a fix sorry

[MQ37]

@matyascimbulka fixed and should be ready now

[matyascimbulka]

Looks good, just found small nit.