Merge pull request #1489 from seleniumbase/undetected-chromedriver

Add option "--uc" for using an undetectable chromedriver

Author: mdmintz

README.md

@@ -496,6 +496,7 @@ The code above will leave your browser window open in case there's a failure. (i
 --enable-ws  # (Enable Web Security on Chromium-based browsers.)
 --enable-sync  # (Enable "Chrome Sync".)
 --use-auto-ext  # (Use Chrome's automation extension.)
+--undetected  # (Use an undetectable chromedriver to evade bot-detection.)
 --remote-debug  # (Enable Chrome's Remote Debugger on http://localhost:9222)
 --final-debug  # (Enter Debug Mode after each test ends. Don't use with CI!)
 --dashboard  # (Enable the SeleniumBase Dashboard. Saved at: dashboard.html)

examples/ReadMe.md

@@ -1,4 +1,4 @@
-## [<img src="https://seleniumbase.io/img/logo6.png" title="SeleniumBase" width="32">](https://github.com/seleniumbase/SeleniumBase/) Running Example Tests:
+## [<img src="https://seleniumbase.io/img/logo6.png" title="SeleniumBase" width="32">](https://github.com/seleniumbase/SeleniumBase/) Example Tests:
 
 <p align="left"><a align="center" href="https://github.com/seleniumbase/SeleniumBase/blob/master/examples/test_demo_site.py"><img align="center" src="https://seleniumbase.io/cdn/img/sb_demo_page.png" alt="SeleniumBase Demo Page" width="420" /></a></p>
 
@@ -7,7 +7,7 @@
 * Tests are structured using [20 unique syntax formats](https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/syntax_formats.md).
 * Logs from test failures are saved to ``./latest_logs/``.
 * Tests can be run with [multiple command-line options](https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/customizing_test_runs.md).
-* Example tests are found in: **[SeleniumBase/examples/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples)**.
+* Examples can be found in: **[SeleniumBase/examples/](https://github.com/seleniumbase/SeleniumBase/tree/master/examples)**.
 
 (NOTE: Some example tests fail on purpose to demonstrate [logging features](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/example_logs/ReadMe.md).)
 

examples/raw_parameter_script.py

@@ -63,6 +63,7 @@
     sb.enable_ws = False
     sb.enable_sync = False
     sb.use_auto_ext = False
+    sb.undetectable = False
     sb.no_sandbox = False
     sb.disable_gpu = False
     sb._multithreaded = False

examples/tour_examples/ReadMe.md

@@ -8,7 +8,7 @@
 
 > **[IntroJS](https://introjs.com/)**, **[Bootstrap Tour](http://bootstraptour.com/)**, **[DriverJS](https://kamranahmed.info/driver.js/)**, **[Shepherd](https://shepherdjs.dev/)**, and **[Hopscotch](https://linkedinattic.github.io/hopscotch/)**.
 
-<b>Example tour: (with autoplay)</b>
+<b>A tour demo: (with autoplay)</b>
 
 <img src="https://seleniumbase.io/cdn/gif/introjs_tour.gif" title="SeleniumBase Tour of Google"><br>
 
@@ -19,7 +19,7 @@ cd examples/tour_examples
 pytest maps_introjs_tour.py --interval=1
 ```
 
-<b>Here's a longer example:</b>
+<b>Here's a longer version:</b>
 
 <img src="https://seleniumbase.io/cdn/gif/google_tour_4.gif" title="SeleniumBase Tour of Google"><br>
 
@@ -97,7 +97,7 @@ You can play a tour by calling:
 
 All methods have the optional ``name`` argument, which is only needed if you're creating multiple tours at once. Then, when you're adding a step or playing a tour, SeleniumBase knows which tour you're referring too. You can avoid using the ``name`` arg for multiple tours if you play a tour before creating a new one.
 
-### Here's an example of using SeleniumBase Tours:
+### Here's how the code looks:
 
 ```python
 from seleniumbase import BaseCase
@@ -122,7 +122,7 @@ class MyTourClass(BaseCase):
         self.play_tour()
 ```
 
-#### This example is from [google_tour.py](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/tour_examples/google_tour.py), which you can run from the ``examples/tour_examples`` folder with the following command:
+#### That code is from [google_tour.py](https://github.com/seleniumbase/SeleniumBase/blob/master/examples/tour_examples/google_tour.py), which you can run from the ``tour_examples/`` folder with the following command:
 
 ```bash
 pytest google_tour.py

examples/verify_undetected.py

@@ -0,0 +1,17 @@
+"""
+Determine if your browser is detectable by anti-bot services.
+Some sites use scripts to detect Selenium and then block you.
+To evade detection, add --uc as a pytest command-line option.
+"""
+from seleniumbase import BaseCase
+
+
+class UndetectedTest(BaseCase):
+    def test_browser_is_undetected(self):
+        self.open("https://nowsecure.nl")
+        try:
+            self.assert_text("OH YEAH, you passed!", "h1", timeout=6.6)
+            self.post_message("Browser wasn't detected!", duration=1.6)
+            self._print("\n Success! Website did not detect Selenium!")
+        except Exception:
+            self.fail('Browser was detected! Try using: "pytest --uc"')

help_docs/customizing_test_runs.md

@@ -162,6 +162,7 @@ pytest my_first_test.py --settings-file=custom_settings.py
 --enable-ws  # (Enable Web Security on Chromium-based browsers.)
 --enable-sync  # (Enable "Chrome Sync".)
 --use-auto-ext  # (Use Chrome's automation extension.)
+--undetected  # (Use an undetectable chromedriver to evade bot-detection.)
 --remote-debug  # (Enable Chrome's Remote Debugger on http://localhost:9222)
 --final-debug  # (Enter Debug Mode after each test ends. Don't use with CI!)
 --dashboard  # (Enable the SeleniumBase Dashboard. Saved at: dashboard.html)

help_docs/method_summary.md

@@ -245,6 +245,7 @@ self.get_new_driver(
     port=None,
     proxy=None,
     proxy_bypass_list=None,
+    proxy_pac_url=None,
     agent=None,
     switch_to=True,
     cap_file=None,
@@ -254,6 +255,7 @@ self.get_new_driver(
     enable_ws=None,
     enable_sync=None,
     use_auto_ext=None,
+    undetectable=None,
     no_sandbox=None,
     disable_gpu=None,
     incognito=None,
@@ -263,12 +265,14 @@ self.get_new_driver(
     swiftshader=None,
     ad_block_on=None,
     block_images=None,
+    do_not_track=None,
     chromium_arg=None,
     firefox_arg=None,
     firefox_pref=None,
     user_data_dir=None,
     extension_zip=None,
     extension_dir=None,
+    page_load_strategy=None,
     external_pdf=None,
     is_mobile=None,
     d_width=None,

help_docs/webdriver_installation.md

@@ -10,13 +10,13 @@ sbase get geckodriver
 sbase get edgedriver
 ```
 
-* If you have the latest version of Chrome installed, get the latest chromedriver (<i>otherwise it defaults to chromedriver 72.0.3626.69 for compatibility reasons</i>):
+* ``sbase get chromedriver`` automatically tries to detect the version you need. If it can't, it defaults to ``chromedriver 72.0.3626.69`` for compatibility reasons. To force getting the latest version, use:
 
 ```bash
 sbase get chromedriver latest
 ```
 
-* You can also install a specific version of chromedriver for a specific version of Chrome:
+* You can also get a specific version of chromedriver for a specific version of Chrome:
 
 ```bash
 sbase get chromedriver 102.0.5005.61
@@ -31,11 +31,11 @@ wget https://raw.githubusercontent.com/seleniumbase/SeleniumBase/master/examples
 pytest upgrade_chromedriver.py -s
 ```
 
-* If you run a test without the correct webdriver installed, the driver will be downloaded automatically.
+* If you run a test without the correct webdriver available, the driver will be downloaded automatically.
 
 If you plan on using the [Selenium Grid integration](https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/utilities/selenium_grid/ReadMe.md) (which allows for remote webdriver), you'll need to put the drivers on your System PATH. On macOS and Linux, ``/usr/local/bin`` is a good PATH spot. On Windows, you may need to set the System PATH under Environment Variables to include the location where you placed the driver files. As a shortcut, you could place the driver files into your Python ``Scripts/`` folder in the location where you have Python installed, which should already be on your System PATH.
 
-Here's where you can go to manually install web drivers from the source:
+Here's where you can go to manually get web drivers from the source:
 
 * For Chrome, get [Chromedriver](https://sites.google.com/a/chromium.org/chromedriver/downloads) on your System PATH.
 
@@ -69,7 +69,7 @@ brew upgrade geckodriver
 
 **Linux shortcuts**:
 
-If you still need the web drivers, here are some scripts to help you install chromedriver and geckodriver on a Linux machine:
+If you still need the web drivers, here are some scripts to help you get chromedriver and geckodriver on a Linux machine:
 
 ```bash
 wget https://chromedriver.storage.googleapis.com/72.0.3626.69/chromedriver_linux64.zip

integrations/google_cloud/ReadMe.md

@@ -1,6 +1,6 @@
 ### Building a browser-based test automation server on the [Google Cloud Platform](https://cloud.google.com/) by using [SeleniumBase](https://github.com/seleniumbase/SeleniumBase)
 
-(This tutorial, [from a previous Google Cloud Meetup](https://www.meetup.com/Boston-Google-Cloud-Meetup/events/230839686/?showDescription=true), will teach you how to setup a Linux server for running automated browser tests. The cost of running this server is about [$13.61/month on Google Cloud](https://console.cloud.google.com/launcher/details/bitnami-launchpad/jenkins) (enough to handle **5 parallel tests**). This is less expensive than using other platforms.)
+(This tutorial, [from a previous Google Cloud Meetup](https://www.meetup.com/Boston-Google-Cloud-Meetup/events/230839686/?showDescription=true), will teach you how to setup a Linux server for running automated browser tests. The cost of running this server is about [$13.60/month on Google Cloud](https://console.cloud.google.com/launcher/details/bitnami-launchpad/jenkins) (enough to handle **5 parallel tests**). This is less expensive than using other platforms.)
 
 <!-- YouTube View --><a href="https://www.youtube.com/watch?v=n-sno20R9P0"><img src="https://seleniumbase.io/other/gcp_video_thumb.png" title="SeleniumBase on YouTube" width="285" /></a>
 <!-- GitHub Only --><p>(<b><a href="https://www.youtube.com/watch?v=n-sno20R9P0">SeleniumBase Google Cloud Video</a></b>)</p>

mkdocs.yml

@@ -71,6 +71,7 @@ plugins:
         - CONTRIBUTING.md
         - SECURITY.md
         - help_docs/chinese.md
+        - help_docs/mysql_installation.md
         - integrations/katalon/ReadMe.md
         - seleniumbase/utilities/selenium_ide/ReadMe.md
         - seleniumbase/examples/chart_maker/ReadMe.md
@@ -82,7 +83,7 @@ plugins:
 # Page tree
 nav:
   - SeleniumBase ReadMe: README.md
-  - Features List: help_docs/features_list.md
+  - List of Features: help_docs/features_list.md
   - Running Example Tests: examples/ReadMe.md
   - Command Line Options: help_docs/customizing_test_runs.md
   - Dashboard / Reports: examples/example_logs/ReadMe.md