Disable overlays

[mgifford]

This is a much bigger push that it probably should be but it is working code I wanted to share.

I wanted to be able to disable the overlays as part of a scan. This seems to work. It needs review but with an increasing number of sites using overlays, this is an important option for site scanning tools.

[younglim]

Hi @mgifford ,

Thanks for sharing! A couple questions to clarify/help me understand:

1. I roughly understand sites can have page load overlays (e.g. divs) that prevent proper scanning of the actual page content

2. Is there a site I can scan with this code, to understand what the overlays blocked challenge?

3. There's added compatibility code so the crawler doesn't get blocked. There were 403 errors you encountered when you tried scanning certain sites, hence added those code? After adding those new lines of code, the scan runs smoothly?

[mgifford]

@younglim Think of this as a proof of concept. There are definitely going to be better ways to accomplish this.

1. Yes. This is just about disabling the 3rd party Javascript that the site is running. It is possible that it could do more.

2. I was looking for a list of sites with overlays. Mighty be easiest to just go to their websites. I'll try to get you a better list, but until then: https://userway.org/

3. Yes, I ran into some problems with the crawler getting blocked. I wasn't sure if that was tied to the overlay or the CDN.

Feel free to close this PR. I wanted to share the code and the idea primarily.