DOC: Add code samples to introduction and refactor howto guides. (#287)

* DOC: Add code samples to introduction and refactor howto guides.

This change adds a proper introduction article to introduce `read_gbq`
and `to_gbq` with code samples. To reduce the amount of duplicated
content, code samples have been extracted into sample files. To give a
better next steps experience, the content in the authentication,
reading, and writing guides has been rearranged to have the most
relevant next steps (after the introduction) as the first section.

* Move private_key_contents fixture to conftest

* Autouse credentials, so that samples tests are skipped when credentials are not available.

* Run system tests on Circle CI by checking for a file at `ci/service_account.json`.

Author: tswast

conftest.py

@@ -0,0 +1,73 @@
+"""Shared pytest fixtures for system tests."""
+
+import os
+import os.path
+import uuid
+
+import google.oauth2.service_account
+import pytest
+
+
+@pytest.fixture(scope="session")
+def project_id():
+    return os.environ.get("GBQ_PROJECT_ID") or os.environ.get(
+        "GOOGLE_CLOUD_PROJECT"
+    )  # noqa
+
+
+@pytest.fixture(scope="session")
+def private_key_path():
+    path = os.path.join(
+        "ci", "service_account.json"
+    )  # Written by the 'ci/config_auth.sh' script.
+    if "GBQ_GOOGLE_APPLICATION_CREDENTIALS" in os.environ:
+        path = os.environ["GBQ_GOOGLE_APPLICATION_CREDENTIALS"]
+    elif "GOOGLE_APPLICATION_CREDENTIALS" in os.environ:
+        path = os.environ["GOOGLE_APPLICATION_CREDENTIALS"]
+
+    if not os.path.isfile(path):
+        pytest.skip(
+            "Cannot run integration tests when there is "
+            "no file at the private key json file path"
+        )
+        return None
+
+    return path
+
+
+@pytest.fixture(scope="session")
+def private_key_contents(private_key_path):
+    if private_key_path is None:
+        return None
+
+    with open(private_key_path) as f:
+        return f.read()
+
+
+@pytest.fixture(scope="module")
+def bigquery_client(project_id, private_key_path):
+    from google.cloud import bigquery
+
+    return bigquery.Client.from_service_account_json(
+        private_key_path, project=project_id
+    )
+
+
+@pytest.fixture()
+def random_dataset_id(bigquery_client):
+    import google.api_core.exceptions
+
+    dataset_id = "".join(["pandas_gbq_", str(uuid.uuid4()).replace("-", "_")])
+    dataset_ref = bigquery_client.dataset(dataset_id)
+    yield dataset_id
+    try:
+        bigquery_client.delete_dataset(dataset_ref, delete_contents=True)
+    except google.api_core.exceptions.NotFound:
+        pass  # Not all tests actually create a dataset
+
+
+@pytest.fixture()
+def credentials(private_key_path):
+    return google.oauth2.service_account.Credentials.from_service_account_file(
+        private_key_path
+    )

docs/source/changelog.rst

@@ -11,6 +11,12 @@ Changelog
   ``max_results`` to 0 to ignore query outputs, such as for DML or DDL
   queries. (:issue:`102`)
 
+Documentation
+~~~~~~~~~~~~~
+
+- Add code samples to introduction and refactor howto guides. (:issue:`239`)
+
+
 .. _changelog-0.11.0:
 
 0.11.0 / 2019-07-29
@@ -44,6 +50,10 @@ Internal changes
 0.10.0 / 2019-04-05
 -------------------
 
+- **Breaking Change:** Default SQL dialect is now ``standard``. Use
+  :attr:`pandas_gbq.context.dialect` to override the default value.
+  (:issue:`195`, :issue:`245`)
+
 Documentation
 ~~~~~~~~~~~~~
 

docs/source/howto/authentication.rst

@@ -1,11 +1,68 @@
 Authentication
 ==============
 
+Before you begin, you must create a Google Cloud Platform project. Use the
+`BigQuery sandbox <https://cloud.google.com/bigquery/docs/sandbox>`__ to try
+the service for free.
+
 pandas-gbq `authenticates with the Google BigQuery service
-<https://cloud.google.com/bigquery/docs/authentication/>`_ via OAuth 2.0.
+<https://cloud.google.com/bigquery/docs/authentication/>`_ via OAuth 2.0. Use
+the ``credentials`` argument to explicitly pass in Google
+:class:`~google.auth.credentials.Credentials`.
 
 .. _authentication:
 
+Default Authentication Methods
+------------------------------
+
+If the ``credentials`` parameter is not set, pandas-gbq tries the following
+authentication methods:
+
+1. In-memory, cached credentials at ``pandas_gbq.context.credentials``. See
+   :attr:`pandas_gbq.Context.credentials` for details.
+
+   .. code:: python
+
+       import pandas_gbq
+
+       credentials = ...  # From google-auth or pydata-google-auth library.
+
+       # Update the in-memory credentials cache (added in pandas-gbq 0.7.0).
+       pandas_gbq.context.credentials = credentials
+       pandas_gbq.context.project = "your-project-id"
+
+       # The credentials and project_id arguments can be omitted.
+       df = pandas_gbq.read_gbq("SELECT my_col FROM `my_dataset.my_table`")
+
+2. Application Default Credentials via the :func:`google.auth.default`
+   function.
+
+   .. note::
+
+       If pandas-gbq can obtain default credentials but those credentials
+       cannot be used to query BigQuery, pandas-gbq will also try obtaining
+       user account credentials.
+
+       A common problem with default credentials when running on Google
+       Compute Engine is that the VM does not have sufficient scopes to query
+       BigQuery.
+
+3. User account credentials.
+
+   pandas-gbq loads cached credentials from a hidden user folder on the
+   operating system.
+
+   Windows
+       ``%APPDATA%\pandas_gbq\bigquery_credentials.dat``
+
+   Linux/Mac/Unix
+       ``~/.config/pandas_gbq/bigquery_credentials.dat``
+
+   If pandas-gbq does not find cached credentials, it prompts you to open a
+   web browser, where you can grant pandas-gbq permissions to access your
+   cloud resources. These credentials are only used locally. See the
+   :doc:`privacy policy <privacy>` for details.
+
 
 Authenticating with a Service Account
 --------------------------------------
@@ -131,55 +188,3 @@ credentials are not found.
 Additional information on the user credentials authentication mechanism
 can be found in the `Google Cloud authentication guide
 <https://cloud.google.com/docs/authentication/end-user>`__.
-
-
-Default Authentication Methods
-------------------------------
-
-If the ``credentials`` parameter (or the deprecated ``private_key``
-parameter) is ``None``, pandas-gbq tries the following authentication
-methods:
-
-1. In-memory, cached credentials at ``pandas_gbq.context.credentials``. See
-   :attr:`pandas_gbq.Context.credentials` for details.
-
-   .. code:: python
-
-       import pandas_gbq
-
-       credentials = ...  # From google-auth or pydata-google-auth library.
-
-       # Update the in-memory credentials cache (added in pandas-gbq 0.7.0).
-       pandas_gbq.context.credentials = credentials
-       pandas_gbq.context.project = "your-project-id"
-
-       # The credentials and project_id arguments can be omitted.
-       df = pandas_gbq.read_gbq("SELECT my_col FROM `my_dataset.my_table`")
-
-2. Application Default Credentials via the :func:`google.auth.default`
-   function.
-
-   .. note::
-
-       If pandas-gbq can obtain default credentials but those credentials
-       cannot be used to query BigQuery, pandas-gbq will also try obtaining
-       user account credentials.
-
-       A common problem with default credentials when running on Google
-       Compute Engine is that the VM does not have sufficient scopes to query
-       BigQuery.
-
-3. User account credentials.
-
-   pandas-gbq loads cached credentials from a hidden user folder on the
-   operating system.
-
-   Windows
-       ``%APPDATA%\pandas_gbq\bigquery_credentials.dat``
-
-   Linux/Mac/Unix
-       ``~/.config/pandas_gbq/bigquery_credentials.dat``
-
-   If pandas-gbq does not find cached credentials, it opens a browser window
-   asking for you to authenticate to your BigQuery account using the product
-   name ``pandas GBQ``.

docs/source/index.rst

@@ -13,7 +13,7 @@ with a shape and data types derived from the source table. Additionally,
 DataFrames can be inserted into new BigQuery tables or appended to existing
 tables.
 
-.. warning::
+.. note::
 
    To use this module, you will need a valid BigQuery account. Use the
    `BigQuery sandbox <https://cloud.google.com/bigquery/docs/sandbox>`__ to

docs/source/install.rst

@@ -40,15 +40,16 @@ This module requires following additional dependencies:
 - `pydata-google-auth <https://github.com/pydata/pydata-google-auth>`__: Helpers for authentication to Google's API
 - `google-auth <https://github.com/GoogleCloudPlatform/google-auth-library-python>`__: authentication and authorization for Google's API
 - `google-auth-oauthlib <https://github.com/GoogleCloudPlatform/google-auth-library-python-oauthlib>`__: integration with `oauthlib <https://github.com/idan/oauthlib>`__ for end-user authentication
-- `google-cloud-bigquery <http://github.com/GoogleCloudPlatform/google-cloud-python>`__: Google Cloud client library for BigQuery
+- `google-cloud-bigquery <https://googleapis.dev/python/bigquery/latest/index.html>`__: Google Cloud client library for BigQuery
+- `google-cloud-bigquery-storage <https://googleapis.dev/python/bigquerystorage/latest/index.html>`__: Google Cloud client library for BigQuery Storage API
 
 .. note::
 
-   The dependency on `google-cloud-bigquery <http://github.com/GoogleCloudPlatform/google-cloud-python>`__ is new in version 0.3.0 of ``pandas-gbq``.
+   The dependency on `google-cloud-bigquery <https://googleapis.dev/python/bigquery/latest/index.html>`__ is new in version 0.3.0 of ``pandas-gbq``.
    Versions less than 0.3.0 required the following dependencies:
 
    - `httplib2 <https://github.com/httplib2/httplib2>`__: HTTP client (no longer required)
-   - `google-api-python-client <http://github.com/google/google-api-python-client>`__: Google's API client (no longer required, replaced by `google-cloud-bigquery <http://github.com/GoogleCloudPlatform/google-cloud-python>`__:)
+   - `google-api-python-client <http://github.com/google/google-api-python-client>`__: Google's API client (no longer required, replaced by `google-cloud-bigquery <hhttps://googleapis.dev/python/bigquery/latest/index.html>`__:)
    - `google-auth <https://github.com/GoogleCloudPlatform/google-auth-library-python>`__: authentication and authorization for Google's API
    - `google-auth-oauthlib <https://github.com/GoogleCloudPlatform/google-auth-library-python-oauthlib>`__: integration with `oauthlib <https://github.com/idan/oauthlib>`__ for end-user authentication
    - `google-auth-httplib2 <https://github.com/GoogleCloudPlatform/google-auth-library-python-httplib2>`__: adapter to use ``httplib2`` HTTP client with ``google-auth`` (no longer required)

docs/source/intro.rst

@@ -1,16 +1,43 @@
 Introduction
 ============
 
-Supported Data Types
-++++++++++++++++++++
+The pandas-gbq package reads data from `Google BigQuery
+<https://cloud.google.com/bigquery/docs/>`__ to a :class:`pandas.DataFrame`
+object and also writes :class:`pandas.DataFrame` objects to BigQuery tables.
 
-Pandas supports all these `BigQuery data types <https://cloud.google.com/bigquery/data-types>`__:
-``STRING``, ``INTEGER`` (64bit), ``FLOAT`` (64 bit), ``BOOLEAN`` and
-``TIMESTAMP`` (microsecond precision). Data types ``BYTES`` and ``RECORD``
-are not supported.
+Authenticating to BigQuery
+--------------------------
 
-Logging
-+++++++
+Before you begin, you must create a Google Cloud Platform project. Use the
+`BigQuery sandbox <https://cloud.google.com/bigquery/docs/sandbox>`__ to try
+the service for free.
+
+If you do not provide any credentials, this module attempts to load
+credentials from the environment. If no credentials are found, pandas-gbq
+prompts you to open a web browser, where you can grant it permissions to
+access your cloud resources. These credentials are only used locally. See the
+:doc:`privacy policy <privacy>` for details.
+
+Learn about authentication methods in the :doc:`authentication guide
+<howto/authentication>`.
+
+Reading data from BigQuery
+--------------------------
+
+Use the :func:`pandas_gbq.read_gbq` function to run a BigQuery query and
+download the results as a :class:`pandas.DataFrame` object.
+
+.. literalinclude:: samples/read_gbq_simple.py
+   :language: python
+   :dedent: 4
+   :start-after: [START bigquery_pandas_gbq_read_gbq_simple]
+   :end-before: [END bigquery_pandas_gbq_read_gbq_simple]
+
+By default, queries use standard SQL syntax. Visit the :doc:`reading tables
+guide <reading>` to learn about the available options.
+
+Adjusting log vebosity
+^^^^^^^^^^^^^^^^^^^^^^
 
 Because some requests take some time, this library will log its progress of
 longer queries. IPython & Jupyter by default attach a handler to the logger.
@@ -23,3 +50,19 @@ more verbose logs, you can do something like:
    logger = logging.getLogger('pandas_gbq')
    logger.setLevel(logging.DEBUG)
    logger.addHandler(logging.StreamHandler())
+
+Writing data to BigQuery
+------------------------
+
+Use the :func:`pandas_gbq.to_gbq` function to write a
+:class:`pandas.DataFrame` object to a BigQuery table.
+
+.. literalinclude:: samples/to_gbq_simple.py
+   :language: python
+   :dedent: 4
+   :start-after: [START bigquery_pandas_gbq_to_gbq_simple]
+   :end-before: [END bigquery_pandas_gbq_to_gbq_simple]
+
+The destination table and destination dataset will automatically be created.
+By default, writes to BigQuery fail if the table already exists. Visit the
+:doc:`writing tables guide <writing>` to learn about the available options.