diff --git a/docs/connect/index.md b/docs/connect/index.md index a36f3b4a..f982bdb9 100644 --- a/docs/connect/index.md +++ b/docs/connect/index.md @@ -179,7 +179,7 @@ application java javascript php -python +python/index ruby natural All drivers diff --git a/docs/connect/python.md b/docs/connect/python.md deleted file mode 100644 index b7509f5a..00000000 --- a/docs/connect/python.md +++ /dev/null @@ -1,220 +0,0 @@ -(connect-python)= -# Python - -:::{div} sd-text-muted -Connect to CrateDB and CrateDB Cloud using different kinds of Python drivers. -::: - -Individual drivers offer specific features for specific -needs of your application, so consider reading this enumeration carefully. - -(python-drivers-official)= -## Official drivers - -(crate-python)= -### crate-python - -The `crate` Python package offers a database client implementation compatible -with the Python Database API 2.0 specification, and also includes the CrateDB -SQLAlchemy dialect. See the full documentation {ref}`here `. -The package can be installed using `pip install crate`. - -```python -from crate import client - -conn = client.connect("https://.cratedb.net:4200", username="admin", password="", verify_ssl_cert=True) - -with conn: - cursor = conn.cursor() - cursor.execute("SELECT * FROM sys.summits") - result = cursor.fetchone() - print(result) -``` - -(sqlalchemy-cratedb)= -### sqlalchemy-cratedb - -The [SQLAlchemy] dialect for CrateDB, based on the HTTP-based DBAPI client -library [crate-python]. -See the full documentation {ref}`here `. -The package can be installed using `pip install sqlalchemy-cratedb`. - -```python -import sqlalchemy as sa - -engine = sa.create_engine("crate://localhost:4200", echo=True) -connection = engine.connect() - -result = connection.execute(sa.text("SELECT * FROM sys.summits;")) -for record in result.all(): - print(record) -``` - -[SQLAlchemy] is the Python SQL toolkit and Object Relational Mapper that -gives application developers the full power and flexibility of SQL. - -Python-based {ref}`dataframe` -and {ref}`ML ` frameworks, and a few {ref}`ETL ` -frameworks, are using SQLAlchemy as database adapter library when connecting to -[RDBMS]. - -- [The CrateDB SQLAlchemy Dialect] -- [Working with SQLAlchemy and CrateDB] -- [SQLAlchemy Code Examples] - - -(python-drivers-community)= -## Community drivers - -(cratedb-async)= - -### cratedb-async - -Asynchronous Python driver for CrateDB based on [HTTPX]. -See the full documentation at . -The package can be installed using `pip install cratedb-async`. - -```python -import asyncio -from cratedb_async.client import CrateClient - -async def main(): - crate = CrateClient("https://.cratedb.net:4200") - response = await crate.query("SELECT * FROM sys.summits") - print(response.as_table()) - -asyncio.run(main()) -``` - -(psycopg2)= - -### psycopg2 - -Psycopg is a popular PostgreSQL database adapter for Python. Its main features -are the complete implementation of the Python DB API 2.0 specification and the -thread safety (several threads can share the same connection). -For more information, see the [psycopg documentation]. - -```python -import psycopg2 - -conn = psycopg2.connect(host=".cratedb.net", port=5432, user="admin", password="", sslmode="require") - -with conn: - with conn.cursor() as cursor: - cursor.execute("SELECT * FROM sys.summits") - result = cursor.fetchone() - print(result) -``` - -(psycopg3)= - -### psycopg3 - -[Psycopg 3] is a newly designed PostgreSQL database adapter for the Python -programming language. Psycopg 3 presents a familiar interface for everyone who -has used Psycopg 2 or any other DB-API 2.0 database adapter, but allows to use -more modern PostgreSQL and Python features, such as: - -- Asynchronous support -- COPY support from Python objects -- A redesigned connection pool -- Support for static typing -- Server-side parameters binding -- Prepared statements -- Statements pipeline -- Binary communication -- Direct access to the libpq functionalities - -```python -import psycopg - -with psycopg.connect("postgres://crate@localhost:5432/doc") as conn: - with conn.cursor() as cursor: - cursor.execute("SELECT * FROM sys.summits") - for record in cursor: - print(record) -``` - -(aiopg)= - -### aiopg - -aiopg is a python library for accessing a PostgreSQL database from the asyncio -(PEP-3156/tulip) framework. It wraps asynchronous features of the Psycopg -database driver. -For more information, see the [aiopg documentation]. - -```python -import asyncio -import aiopg - -async def run(): - async with aiopg.create_pool(host=".cratedb.net", port=5432, user="admin", password="", sslmode="require") as pool: - async with pool.acquire() as conn: - async with conn.cursor() as cursor: - await cursor.execute("SELECT * FROM sys.summits") - result = await cursor.fetchone() - print(result) - -loop = asyncio.get_event_loop() -loop.run_until_complete(run()) -``` - -(asyncpg)= - -### asyncpg - -asyncpg is a database interface library designed specifically for PostgreSQL -and Python/asyncio. asyncpg is an efficient, clean implementation of the -PostgreSQL server binary protocol for use with Python's asyncio framework. -For more information, see the [asyncpg documentation]. - -```python -import asyncio -import asyncpg - -async def run(): - conn = await asyncpg.connect(host=".cratedb.net", port=5432, user="admin", password="", ssl=True) - try: - result = await conn.fetch("SELECT * FROM sys.summits") - finally: - await conn.close() - print(result) - -loop = asyncio.get_event_loop() -loop.run_until_complete(run()) -``` - - -(python-dataframe)= -(df)= -(dataframe)= -(dataframes)= -(dataframe-examples)= -## Dataframe libraries - -How to use CrateDB together with popular open-source DataFrame libraries. - -### Dask -- {ref}`dask` - -### pandas -- {ref}`pandas` - -### Polars -- {ref}`polars` - - - -[aiopg documentation]: https://aiopg.readthedocs.io/ -[asyncpg documentation]: https://magicstack.github.io/asyncpg/current/ -[httpx]: https://www.python-httpx.org/ -[psycopg 3]: https://www.psycopg.org/psycopg3/docs/ -[psycopg documentation]: https://www.psycopg.org/docs/ - -[RDBMS]: https://en.wikipedia.org/wiki/RDBMS -[SQLAlchemy]: https://www.sqlalchemy.org/ -[SQLAlchemy Code Examples]: https://github.com/crate/cratedb-examples/tree/main/by-language/python-sqlalchemy -[The CrateDB SQLAlchemy Dialect]: inv:sqlalchemy-cratedb:*:label#index -[Working with SQLAlchemy and CrateDB]: inv:sqlalchemy-cratedb:*:label#by-example diff --git a/docs/connect/python/aiopg.md b/docs/connect/python/aiopg.md new file mode 100644 index 00000000..106fd46c --- /dev/null +++ b/docs/connect/python/aiopg.md @@ -0,0 +1,27 @@ +(aiopg)= + +# aiopg + +aiopg is a python library for accessing a PostgreSQL database from the asyncio +(PEP-3156/tulip) framework. It wraps asynchronous features of the Psycopg +database driver. +For more information, see the [aiopg documentation]. + +```python +import asyncio +import aiopg + +async def run(): + async with aiopg.create_pool(host=".cratedb.net", port=5432, user="admin", password="", sslmode="require") as pool: + async with pool.acquire() as conn: + async with conn.cursor() as cursor: + await cursor.execute("SELECT * FROM sys.summits") + result = await cursor.fetchone() + print(result) + +loop = asyncio.get_event_loop() +loop.run_until_complete(run()) +``` + + +[aiopg documentation]: https://aiopg.readthedocs.io/ diff --git a/docs/connect/python/asyncpg.md b/docs/connect/python/asyncpg.md new file mode 100644 index 00000000..e2baec52 --- /dev/null +++ b/docs/connect/python/asyncpg.md @@ -0,0 +1,27 @@ +(asyncpg)= + +# asyncpg + +asyncpg is a database interface library designed specifically for PostgreSQL +and Python/asyncio. asyncpg is an efficient, clean implementation of the +PostgreSQL server binary protocol for use with Python's asyncio framework. +For more information, see the [asyncpg documentation]. + +```python +import asyncio +import asyncpg + +async def run(): + conn = await asyncpg.connect(host=".cratedb.net", port=5432, user="admin", password="", ssl=True) + try: + result = await conn.fetch("SELECT * FROM sys.summits") + finally: + await conn.close() + print(result) + +loop = asyncio.get_event_loop() +loop.run_until_complete(run()) +``` + + +[asyncpg documentation]: https://magicstack.github.io/asyncpg/current/ diff --git a/docs/connect/python/conecta.md b/docs/connect/python/conecta.md new file mode 100644 index 00000000..da0b8666 --- /dev/null +++ b/docs/connect/python/conecta.md @@ -0,0 +1,34 @@ +(conecta-intro)= + +# Conecta + +:::{div} .float-right .text-right +[![conecta-core CI](https://github.com/surister/conecta/actions/workflows/test_core.yml/badge.svg)](https://github.com/surister/conecta/actions/workflows/test_core.yml) +[![conecta-python CI](https://github.com/surister/conecta/actions/workflows/test_python.yml/badge.svg)](https://github.com/surister/conecta/actions/workflows/test_python.yml) +::: +:::{div} .clearfix +::: + +{ref}`conecta` is a library designed to load data from SQL databases into +Arrow with maximum speed and memory efficiency by leveraging zero-copy and +true concurrency in Python. + +```python +from pprint import pprint +from conecta import read_sql + +table = read_sql( + "postgres://crate:crate@localhost:5432/doc", + queries=["SELECT country, region, mountain, height, latitude(coordinates), longitude(coordinates) FROM sys.summits ORDER BY height DESC LIMIT 3"], +) + +# Display in Python format. +pprint(table.to_pylist()) + +# Optionally convert to pandas dataframe. +print(table.to_pandas()) + +# Optionally convert to Polars dataframe. +import polars as pl +print(pl.from_arrow(table)) +``` diff --git a/docs/connect/python/connectorx.md b/docs/connect/python/connectorx.md new file mode 100644 index 00000000..4e3b1a2b --- /dev/null +++ b/docs/connect/python/connectorx.md @@ -0,0 +1,29 @@ +(connectorx)= + +# ConnectorX + +:::{div} .float-right .text-right +[![ConnectorX CI](https://github.com/crate/cratedb-examples/actions/workflows/lang-python-connectorx.yml/badge.svg)](https://github.com/crate/cratedb-examples/actions/workflows/lang-python-connectorx.yml) +::: +:::{div} .clearfix +::: + +[ConnectorX] enables you to load data from databases into Python in the +fastest and most memory-efficient way. + +```python +import connectorx as cx + +cx.read_sql( + "postgresql://username:password@server:port/database", + "SELECT * FROM lineitem", + partition_on="l_orderkey", + partition_num=10, +) +``` + +- [Connect to CrateDB using ConnectorX] + + +[ConnectorX]: https://sfu-db.github.io/connector-x/ +[Connect to CrateDB using ConnectorX]: https://github.com/crate/cratedb-examples/tree/main/by-language/python-connectorx diff --git a/docs/connect/python/crate-python.md b/docs/connect/python/crate-python.md new file mode 100644 index 00000000..94847f12 --- /dev/null +++ b/docs/connect/python/crate-python.md @@ -0,0 +1,30 @@ +(crate-python)= +# crate-python + +:::{div} .float-right .text-right +[![Python DB API CI](https://github.com/crate/cratedb-examples/actions/workflows/lang-python-dbapi.yml/badge.svg)](https://github.com/crate/cratedb-examples/actions/workflows/lang-python-dbapi.yml) +::: +:::{div} .clearfix +::: + +The `crate` Python package offers a database client implementation compatible +with the Python Database API 2.0 specification, and also includes the CrateDB +SQLAlchemy dialect. See the full documentation {ref}`here `. +The package can be installed using `pip install crate`. + +```python +from crate import client + +conn = client.connect("https://.cratedb.net:4200", username="admin", password="", verify_ssl_cert=True) + +with conn: + cursor = conn.cursor() + cursor.execute("SELECT * FROM sys.summits") + result = cursor.fetchone() + print(result) +``` + +- [Connect to CrateDB using the Python DB API] + + +[Connect to CrateDB using the Python DB API]: https://github.com/crate/cratedb-examples/tree/main/by-language/python-dbapi diff --git a/docs/connect/python/cratedb-async.md b/docs/connect/python/cratedb-async.md new file mode 100644 index 00000000..1119c8b7 --- /dev/null +++ b/docs/connect/python/cratedb-async.md @@ -0,0 +1,22 @@ +(cratedb-async)= + +# cratedb-async + +Asynchronous Python driver for CrateDB based on [HTTPX]. +See the full documentation at . +The package can be installed using `pip install cratedb-async`. + +```python +import asyncio +from cratedb_async.client import CrateClient + +async def main(): + crate = CrateClient("https://.cratedb.net:4200") + response = await crate.query("SELECT * FROM sys.summits") + print(response.as_table()) + +asyncio.run(main()) +``` + + +[HTTPX]: https://www.python-httpx.org/ diff --git a/docs/connect/python/index.md b/docs/connect/python/index.md new file mode 100644 index 00000000..d4cc3405 --- /dev/null +++ b/docs/connect/python/index.md @@ -0,0 +1,119 @@ +(connect-python)= +# Python + +:::{div} sd-text-muted +Connect to CrateDB and CrateDB Cloud from Python. +::: + +Individual adapters, clients, and drivers offer specific features for specific +needs of your application, so please read this enumeration carefully, to +select the optimal adapter client or driver for your purposes. + +As a general recommendation, using the SQLAlchemy adapter for many reasons +isn't a bad choice, because many other frameworks will use it anyway. +A good example is the most high-level Python package `records`. + +Other than this, we can recommend `psycopg3` and `asyncpg` if you are +looking at using the PostgreSQL protocol. + +(python-client-official)= +## Official clients + +:::{rubric} Standard +::: +Standard clients implementing the Python DB API and the CrateDB dialect for SQLAlchemy. +`crate-python` uses `urllib3`. `sqlalchemy-cratedb` uses `crate-python`. +:::{toctree} +crate-python +sqlalchemy-cratedb +::: + +(python-client-special)= +:::{rubric} Special purpose +::: +Adapter clients and drivers for special requirements on the environment or +on performance details. +`conecta` uses Rust and Apache Arrow. +`cratedb-async` uses `httpx`. +`micropython-cratedb` uses MicroPython's standard `httplib`. +:::{toctree} +conecta +cratedb-async +micropython-cratedb +::: + +(python-client-community)= +## Tertiary clients + +Clients that either use the PostgreSQL wire protocol, or standard +interfaces like SQLAlchemy. +`psycopg2` and `psycopg3` use `libpq`. `aiopg` uses `psycopg2`. +`connectorx` uses Rust. `records` uses `sqlalchemy`, so it also +uses `sqlalchemy-cratedb`. `turbodbc` uses PostgreSQL ODBC. + +:::{toctree} +psycopg2 +psycopg3 +aiopg +asyncpg +connectorx +records +turbodbc +::: + +(python-dataframe)= +(df)= +(dataframe)= +(dataframes)= +(dataframe-examples)= +## Dataframe libraries + +Use CrateDB together with popular open-source dataframe libraries. +Each of them is using the CrateDB SQLAlchemy dialect for database +communications. + +:::::{grid} 2 3 4 4 +:gutter: 3 3 4 5 +:padding: 0 + +::::{grid-item-card} Dask +:link: dask +:link-type: ref +:link-alt: Connect to CrateDB using Dask +:text-align: center +:class-card: sd-pt-3 +:class-body: sd-fs-1 +:class-title: sd-fs-6 +![Dask logo](https://github.com/crate/crate-clients-tools/assets/453543/99bd2234-c501-479b-ade7-bcc2bfc1f288){height=40px} +:::: + +::::{grid-item-card} pandas +:link: pandas +:link-type: ref +:link-alt: Connect to CrateDB using pandas +:text-align: center +:class-card: sd-pt-3 +:class-body: sd-fs-1 +:class-title: sd-fs-6 +![pandas logo](https://pandas.pydata.org/static/img/pandas.svg){height=40px} +:::: + +::::{grid-item-card} Polars +:link: polars +:link-type: ref +:link-alt: Connect to CrateDB using Polars +:text-align: center +:class-card: sd-pt-3 +:class-body: sd-fs-1 +:class-title: sd-fs-6 +![Polars logo](https://github.com/pola-rs/polars-static/raw/master/logos/polars-logo-dark.svg){height=40px} +:::: + +::::: + +:::{tip} +The `insert_bulk` utility function from `sqlalchemy-cratedb` unlocks +a performance path, see also {ref}`sqlalchemy-cratedb:support-insert-bulk`. +About optimally configuring pandas and Dask for efficient insert operations, +see also {ref}`sqlalchemy-cratedb:dataframe`. +::: diff --git a/docs/connect/python/micropython-cratedb.md b/docs/connect/python/micropython-cratedb.md new file mode 100644 index 00000000..dc35fe87 --- /dev/null +++ b/docs/connect/python/micropython-cratedb.md @@ -0,0 +1,31 @@ +(micropython-cratedb)= + +# micropython-cratedb + +:::{div} .float-right .text-right +[![MicroPython CI](https://github.com/crate/micropython-cratedb/actions/workflows/tests.yml/badge.svg)](https://github.com/crate/micropython-cratedb/actions/workflows/tests.yml) +::: +:::{div} .clearfix +::: + +A MicroPython library connecting to the CrateDB HTTP API. +See the full documentation at . +The package can be installed using `mpremote mip install github:crate/micropython-cratedb`. + +```python +import cratedb + +crate = cratedb.CrateDB( + host="localhost", + port=4200, + user="crate", + password="crate", + use_ssl=False +) + +response = crate.execute( + "SELECT * FROM sys.summits ORDER BY height DESC LIMIT 3" +) + +print(response) +``` diff --git a/docs/connect/python/psycopg2.md b/docs/connect/python/psycopg2.md new file mode 100644 index 00000000..b61c1949 --- /dev/null +++ b/docs/connect/python/psycopg2.md @@ -0,0 +1,23 @@ +(psycopg2)= + +# psycopg2 + +Psycopg is a popular PostgreSQL database adapter for Python. Its main features +are the complete implementation of the Python DB API 2.0 specification and the +thread safety (several threads can share the same connection). +For more information, see the [psycopg2 documentation]. + +```python +import psycopg2 + +conn = psycopg2.connect(host=".cratedb.net", port=5432, user="admin", password="", sslmode="require") + +with conn: + with conn.cursor() as cursor: + cursor.execute("SELECT * FROM sys.summits") + result = cursor.fetchone() + print(result) +``` + + +[psycopg2 documentation]: https://www.psycopg.org/docs/ diff --git a/docs/connect/python/psycopg3.md b/docs/connect/python/psycopg3.md new file mode 100644 index 00000000..56880224 --- /dev/null +++ b/docs/connect/python/psycopg3.md @@ -0,0 +1,31 @@ +(psycopg3)= + +# psycopg3 + +[Psycopg 3] is a newly designed PostgreSQL database adapter for the Python +programming language. Psycopg 3 presents a familiar interface for everyone who +has used Psycopg 2 or any other DB-API 2.0 database adapter, but allows to use +more modern PostgreSQL and Python features, such as: + +- Asynchronous support +- COPY support from Python objects +- A redesigned connection pool +- Support for static typing +- Server-side parameters binding +- Prepared statements +- Statements pipeline +- Binary communication +- Direct access to the libpq functionalities + +```python +import psycopg + +with psycopg.connect("postgres://crate@localhost:5432/doc") as conn: + with conn.cursor() as cursor: + cursor.execute("SELECT * FROM sys.summits") + for record in cursor: + print(record) +``` + + +[psycopg 3]: https://www.psycopg.org/psycopg3/docs/ diff --git a/docs/connect/python/records.md b/docs/connect/python/records.md new file mode 100644 index 00000000..6392b918 --- /dev/null +++ b/docs/connect/python/records.md @@ -0,0 +1,31 @@ +(records)= + +# Records + +:::{div} .float-right .text-right +[![records (framework)](https://github.com/crate/cratedb-examples/actions/workflows/framework-records.yml/badge.svg)](https://github.com/crate/cratedb-examples/actions/workflows/framework-records.yml) +::: +:::{div} .clearfix +::: + +[Records] is a deceptively simple but powerful library for making raw SQL +queries to most relational databases. Powered by SQLAlchemy and Tablib, +it covers many database types and allows you to export your results to +CSV, XLS, JSON, HTML Tables, YAML, or pandas dataframes with a single +line of code. + +```shell +pip install --upgrade records sqlalchemy-cratedb +``` +```python +import records + +db = records.Database("crate://", echo=True) +rows = db.query("SELECT region, mountain, height FROM sys.summits ORDER BY height DESC LIMIT 3") +data = rows.all() + +print(rows.export("json")) +``` + + +[Records]: https://github.com/kennethreitz/records diff --git a/docs/connect/python/sqlalchemy-cratedb.md b/docs/connect/python/sqlalchemy-cratedb.md new file mode 100644 index 00000000..e5b8320d --- /dev/null +++ b/docs/connect/python/sqlalchemy-cratedb.md @@ -0,0 +1,43 @@ +(sqlalchemy-cratedb)= +# sqlalchemy-cratedb + +:::{div} .float-right .text-right +[![CrateDB SQLAlchemy CI](https://github.com/crate/cratedb-examples/actions/workflows/lang-python-sqlalchemy.yml/badge.svg)](https://github.com/crate/cratedb-examples/actions/workflows/lang-python-sqlalchemy.yml) +::: +:::{div} .clearfix +::: + +The [SQLAlchemy] dialect for CrateDB, based on the HTTP-based DB API client +library {ref}`crate-python:index`. +See the full documentation {ref}`here `. +The package can be installed using `pip install sqlalchemy-cratedb`. + +```python +import sqlalchemy as sa + +engine = sa.create_engine("crate://localhost:4200", echo=True) +connection = engine.connect() + +result = connection.execute(sa.text("SELECT * FROM sys.summits;")) +for record in result.all(): + print(record) +``` + +[SQLAlchemy] is the Python SQL toolkit and Object Relational Mapper that +gives application developers the full power and flexibility of SQL. + +Python-based {ref}`dataframe` +and {ref}`ML ` frameworks, and a few {ref}`ETL ` +frameworks, are using SQLAlchemy as database adapter library when connecting to +[RDBMS]. + +- [The CrateDB SQLAlchemy Dialect] +- [Working with SQLAlchemy and CrateDB] +- [SQLAlchemy Code Examples] + + +[RDBMS]: https://en.wikipedia.org/wiki/RDBMS +[SQLAlchemy]: https://www.sqlalchemy.org/ +[SQLAlchemy Code Examples]: https://github.com/crate/cratedb-examples/tree/main/by-language/python-sqlalchemy +[The CrateDB SQLAlchemy Dialect]: inv:sqlalchemy-cratedb:*:label#index +[Working with SQLAlchemy and CrateDB]: inv:sqlalchemy-cratedb:*:label#by-example diff --git a/docs/connect/python/turbodbc.md b/docs/connect/python/turbodbc.md new file mode 100644 index 00000000..6e0afccd --- /dev/null +++ b/docs/connect/python/turbodbc.md @@ -0,0 +1,24 @@ +(turbodbc)= + +# turbodbc + +:::{div} .float-right .text-right +[![turbodbc CI](https://github.com/crate/cratedb-examples/actions/workflows/lang-python-turbodbc.yml/badge.svg)](https://github.com/crate/cratedb-examples/actions/workflows/lang-python-turbodbc.yml) +::: +:::{div} .clearfix +::: + +[Turbodbc] is a Python module to access relational databases via the Open +Database Connectivity (ODBC) interface. Its primary target audience are +data scientists that use databases for which no efficient native Python +drivers are available. + +For maximum performance, turbodbc offers built-in NumPy and Apache Arrow +support and internally relies on batched data transfer instead of +single-record communication as other popular ODBC modules do. + +- [Using CrateDB with turbodbc] + + +[turbodbc]: https://turbodbc.readthedocs.io/ +[Using CrateDB with turbodbc]: https://github.com/crate/cratedb-examples/tree/main/by-language/python-turbodbc