Content-Type application/xhtml+xml not filtered

[WRFan]

If you remove the user-agent from the request headers or use some user-agent that google doesn't recognize, it sends some weird mobile page to the browser, which Proxydomo fails to filter. If I enable the Web Filter Debug in the Log window and load the page, Proxydomo just displays some binary (gzipped ?) output. Could you please look into this issue?

https://github.com/amate/Proxydomo/issues/new

Request:

Request sent to website
GET /search?hl=en&nfpr=1&prmd=u&q=a HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, /
Accept-Language: en-GB,en-US,en,de-DE,ru-RU
Accept-Encoding: gzip, deflate
User-Agent: AdsBot-Google
Host: www.google.de
DNT: 1
Connection: Keep-Alive

Response:

Response sent to browser
HTTP/1.1 200 OK
Content-Type: application/xhtml+xml; charset=ISO-8859-1
Date: Sat, 30 Nov 2019 23:41:58 GMT
Content-Encoding: gzip
Transfer-Encoding: chunked
Access-Control-Allow-Origin: *

[nhantrn]

What are you trying to do after removing the UA? Leaving the ua blank probably tripped some Google rule and they switched to their minimal mobile page.

It still got filtered fine on my end with this:

[Patterns]
Name = "google test"
Version = ""
Author = ""
Comment = ""
Active = TRUE
Multi = FALSE
URL = "www.google.com/search*"
Bounds = ""
Limit = 2048
Match = "<header*</header>"
Replace = "<h1>TEST</h1>"

[WRFan]

The filter you tested with is a web page filter, I'm talking about the user agent request header - "outgoing header" in proxydomo:

[HTTP headers]
Key = "User-Agent: User-Agent Debug (Out)"
In = FALSE
Out = TRUE
Version = ""
Author = ""
Comment = ""
Active = TRUE
Multi = FALSE
URL = ""
Bounds = ""
Limit = 256
Match = "$URL(http(s|)://(.|)google./search?)"
Replace = "\0"

It's not about google, the question is if it's the only page on the internet that causes this problem. If it is I can live with it, but there may be more pages like this.

Btw, it's interesting which user-agent strings google expects to send the standard non-mobile page. I tested it a little and found out the google servers expect one of the following user-agents:

(MSIE 6; trident/6)

Trident/7
"7" matters

Firefox/7
"gecko/" before the string is ok

(windows) applewebkit/ Edge
(windows) applewebkit/ Chrome/5 Safari

Applewebkit/537 Version/09 Safari/
Applewebkit/600

------------------------------------------------------------------------------- google Images
(MSIE 1
Applewebkit/1

Anything else, or if the user-agent is not there at all (as in the filter above), and google sends the mobile page that's ignored by Proxydomo