Chrome no headless option

[PierreBrisorgueil]

it might be nice to have the possibility for the build to choose a non-headless for some usecases

[ziflex]

Non-headless is not supported by Docker.
If you want to use Worker with open Chrome, you need to run the browser locally with open debugging port and let the Worker to connect to it.

[PierreBrisorgueil]

um, i thought something was playable with xvfb

okok 🤔

[ziflex]

ok, I'm taking my words back. Yes, it's possible to use xvfb for that.

[ziflex]

Can you add it to the base Image?

[PierreBrisorgueil]

Headfull

Hey ! Sadly not for the moment, I'm stuck on this :

• remote-debugging siomiz/chrome#26
• https://forums.docker.com/t/docker-chrome-headfull-with-xvfb-and-remote-debugging-active-dbus/108202

Everything start but chrome api couldn't be called 🤔

I've stoped my research on this last answer :

• https://stackoverflow.com/questions/49285658/how-to-solve-docker-issue-failed-to-connect-to-bus-no-such-file-or-directory

Maybe i'm missing something 🤷‍♂️

If you want to test i probably have my test branch somewhere based on https://github.com/siomiz/chrome , ping me on discord

Headless

For these reasons I began to study concealing a headless chrome based on https://github.com/paulirish/headless-cat-n-mouse, with an MITM docker before page explorer interpretation. This allow us JS injection to bypass client side headless detection. The cat n mouse race to hide headless is not ended, but we should share it. I've added you to my private repo example, we could integrate it as an option in worker or side by side (DockerCompose) or in ferret (like a cdp option).

Actually my example success on this test : https://intoli.com/blog/not-possible-to-block-chrome-headless/chrome-headless-test.html
More informations : https://intoli.com/blog/not-possible-to-block-chrome-headless/

(I've added an fql example on discord to test chrome based on intoli test)

[jvitor83]

Take a look at: https://github.com/jvitor83/chrome-debug-node-vnc

It is a built-in chrome with debug capabilities and VNC for remote viewer and node to use puppeteer. In some testing it isn't detected as a bot because it is headfull (but inside docker container).

[ziflex]

@PierreBrisorgueil hey, have you managed to create a Docker image with non-headless Chromium?

[PierreBrisorgueil]

@ziflex Nop, I set up a MITM to hide the headless with multiple JS tricks, but I still haven't managed to make the headfull in docker.

I also wanted to watch how they handled the head full on the crawlee side

[PierreBrisorgueil]

@ziflex have you deep dive the subject ?

[1369521908]

Take a look at: https://github.com/jvitor83/chrome-debug-node-vnc

It is a built-in chrome with debug capabilities and VNC for remote viewer and node to use puppeteer. In some testing it isn't detected as a bot because it is headfull (but inside docker container).

works, thx