Connect: Migrate pages from crate-clients-tools, to be phased out soon

Author: amotl

docs/connect/cli.md

@@ -0,0 +1,183 @@
+(cli)=
+(connect-cli)=
+# Using command-line programs with CrateDB
+
+This section provides a quick overview about a few CLI programs, and how to
+use them for connecting to CrateDB clusters. We recommend to use crash,
+psql, http ([HTTPie]), or curl.
+
+You can use them to quickly validate HTTP and PostgreSQL connectivity to your
+database cluster, or to conduct basic scripting.
+
+Before running the command-line snippets outlined below, please use the correct
+settings instead of the placeholder tokens `<hostname>`, `<username>` and
+`<password>`.
+
+When using CrateDB Cloud, `<hostname>` will be something like
+`<clustername>.{aks1,eks1}.region.{azure,aws}.cratedb.net`.
+
+
+(crash)=
+## crash
+
+```{div}
+:style: "float: right"
+![image](https://cratedb.com/docs/crate/crash/en/latest/_images/query.png){w=240px}
+```
+
+The **CrateDB Shell** is an interactive command-line interface (CLI) tool for
+working with CrateDB. For more information, see the documentation about [crash].
+
+```{div}
+:style: "clear: both"
+```
+
+::::{tab-set}
+
+:::{tab-item} CrateDB and CrateDB Cloud
+:sync: server
+
+```{code-block} shell
+CRATEPW=<password> \
+    crash --hosts 'https://<hostname>:4200' --username '<username>' \
+    --command "SELECT 42.42;"
+```
+:::
+
+:::{tab-item} CrateDB on localhost
+:sync: localhost
+
+```{code-block} shell
+# No authentication. 
+crash --command "SELECT 42.42;"
+ 
+```
+:::
+
+::::
+
+
+(psql)=
+## psql
+
+```{div}
+:style: "float: right"
+![image](https://github.com/crate/crate-clients-tools/assets/453543/8f0a0e06-87f6-467f-be2d-b38121afbafa){w=240px}
+```
+
+**psql** is a terminal-based front-end to PostgreSQL. It enables you to type in
+queries interactively, issue them to PostgreSQL, and see the query results.
+For more information, see the documentation about [psql].
+
+```{div}
+:style: "clear: both"
+```
+
+::::{tab-set}
+
+:::{tab-item} CrateDB and CrateDB Cloud
+:sync: server
+
+```{code-block} shell
+PGUSER=<username> PGPASSWORD=<password> \
+    psql postgresql://<hostname>/crate --command "SELECT 42.42;"
+```
+:::
+
+:::{tab-item} CrateDB on localhost
+:sync: localhost
+
+```{code-block} shell
+# No authentication.
+psql postgresql://crate@localhost:5432/crate --command "SELECT 42.42;"
+```
+:::
+
+::::
+
+
+(httpie)=
+## HTTPie
+
+```{div}
+:style: "float: right"
+![image](https://github.com/crate/crate-clients-tools/assets/453543/f5a2916d-3730-4901-99cf-b88b9af03329){w=240px}
+```
+
+The **HTTPie CLI** is a modern, user-friendly command-line HTTP client with
+JSON support, colors, sessions, downloads, plugins & more. 
+For more information, see the documentation about [HTTPie].
+
+```{div}
+:style: "clear: both"
+```
+
+::::{tab-set}
+
+:::{tab-item} CrateDB and CrateDB Cloud
+:sync: server
+
+```{code-block} shell
+http "https://<username>:<password>@<hostname>:4200/_sql?pretty" \
+    stmt="SELECT 42.42;"
+```
+:::
+
+:::{tab-item} CrateDB on localhost
+:sync: localhost
+
+```{code-block} shell
+http "localhost:4200/_sql?pretty" \
+    stmt="SELECT 42.42;"
+```
+:::
+
+::::
+
+
+(curl)=
+## curl
+
+```{div}
+:style: "float: right"
+![image](https://github.com/crate/crate-clients-tools/assets/453543/318b0819-a0d4-4112-a320-23852263362c){w=240px}
+```
+
+The venerable **curl** is the ubiquitous command line tool and library for transferring
+data with URLs. For more information, see the documentation about [curl].
+
+This example combines it with [jq], a lightweight and flexible command-line JSON processor.
+
+```{div}
+:style: "clear: both"
+```
+
+::::{tab-set}
+
+:::{tab-item} CrateDB and CrateDB Cloud
+:sync: server
+
+```{code-block} shell
+echo '{"stmt": "SELECT 42.42;"}' \
+    | curl "https://<username>:<password>@<hostname>:4200/_sql?pretty" --silent --data @- | jq
+```
+:::
+
+:::{tab-item} CrateDB on localhost
+:sync: localhost
+
+```{code-block} shell
+echo '{"stmt": "SELECT 42.42;"}' \
+    | curl "localhost:4200/_sql?pretty" --silent --data @- | jq
+```
+:::
+
+::::
+
+
+
+[curl]: https://curl.se/
+[crash]: inv:crate-crash:*:label#index
+[HTTPie]: https://httpie.io/
+[jq]: https://jqlang.github.io/jq/
+[psql]: https://www.postgresql.org/docs/current/app-psql.html

docs/connect/df.md

@@ -0,0 +1,184 @@
+(df)=
+(dataframes)=
+# CrateDB and DataFrame libraries
+
+Data frame libraries and frameworks which can
+be used together with CrateDB.
+
+
+:::::{grid} 1 2 2 2
+:margin: 4 4 0 0
+:padding: 0
+:gutter: 2
+
+::::{grid-item-card} {material-outlined}`lightbulb;2em` Tutorials
+:link: guide:dataframes
+:link-type: ref
+Learn how to use CrateDB together with popular open-source data frame
+libraries, on behalf of hands-on tutorials and code examples.
++++
+{tag-info}`Dask` {tag-info}`pandas` {tag-info}`Polars`
+::::
+
+::::{grid-item-card} {material-outlined}`read_more;2em` SQLAlchemy
+CrateDB's SQLAlchemy dialect implementation provides fundamental infrastructure
+to integrations with Dask, pandas, and Polars.
++++
+[ORM Guides](inv:guide#orm) •
+{ref}`ORM Catalog <orm>`
+::::
+
+:::::
+
+
+(dask)=
+## Dask
+
+[Dask] is a parallel computing library for analytics with task scheduling.
+It is built on top of the Python programming language, making it easy to scale
+the Python libraries that you know and love, like NumPy, pandas, and scikit-learn.
+
+```{div}
+:style: "float: right"
+[![](https://github.com/crate/crate-clients-tools/assets/453543/99bd2234-c501-479b-ade7-bcc2bfc1f288){w=180px}](https://www.dask.org/)
+```
+
+- [Dask DataFrames] help you process large tabular data by parallelizing pandas,
+  either on your laptop for larger-than-memory computing, or on a distributed
+  cluster of computers.
+
+- [Dask Futures], implementing a real-time task framework, allow you to scale
+  generic Python workflows across a Dask cluster with minimal code changes,
+  by extending Python's `concurrent.futures` interface.
+
+```{div}
+:style: "clear: both"
+```
+
+
+(pandas)=
+## pandas
+
+```{div}
+:style: "float: right"
+[![](https://pandas.pydata.org/static/img/pandas.svg){w=180px}](https://pandas.pydata.org/)
+```
+
+[pandas] is a fast, powerful, flexible, and easy to use open source data analysis
+and manipulation tool, built on top of the Python programming language. 
+
+Pandas (stylized as pandas) is a software library written for the Python programming
+language for data manipulation and analysis. In particular, it offers data structures
+and operations for manipulating numerical tables and time series.
+
+:::{rubric} Data Model
+:::
+- Pandas is built around data structures called Series and DataFrames. Data for these
+  collections can be imported from various file formats such as comma-separated values,
+  JSON, Parquet, SQL database tables or queries, and Microsoft Excel.
+- A Series is a 1-dimensional data structure built on top of NumPy's array.
+- Pandas includes support for time series, such as the ability to interpolate values
+  and filter using a range of timestamps.
+- By default, a Pandas index is a series of integers ascending from 0, similar to the
+  indices of Python arrays. However, indices can use any NumPy data type, including
+  floating point, timestamps, or strings.
+- Pandas supports hierarchical indices with multiple values per data point. An index
+  with this structure, called a "MultiIndex", allows a single DataFrame to represent
+  multiple dimensions, similar to a pivot table in Microsoft Excel. Each level of a
+  MultiIndex can be given a unique name.
+
+```{div}
+:style: "clear: both"
+```
+
+
+(polars)=
+## Polars
+
+```{div}
+:style: "float: right; margin-left: 0.5em"
+[![](https://github.com/pola-rs/polars-static/raw/master/logos/polars-logo-dark.svg){w=180px}](https://pola.rs/)
+```
+
+[Polars] is a blazingly fast DataFrames library with language bindings for
+Rust, Python, Node.js, R, and SQL. Polars is powered by a multithreaded,
+vectorized query engine, it is open source, and written in Rust.
+
+- **Fast:** Written from scratch in Rust and with performance in mind,
+  designed close to the machine, and without external dependencies.
+
+- **I/O:** First class support for all common data storage layers: local,
+  cloud storage & databases.
+
+- **Intuitive API:** Write your queries the way they were intended. Polars,
+  internally, will determine the most efficient way to execute using its query
+  optimizer. Polars' expressions are intuitive and empower you to write
+  readable and performant code at the same time.
+
+- **Out of Core:** The streaming API allows you to process your results without
+  requiring all your data to be in memory at the same time.
+
+- **Parallel:** Polars' multi-threaded query engine utilises the power of your
+  machine by dividing the workload among the available CPU cores without any
+  additional configuration.
+
+- **Vectorized Query Engine:** Uses [Apache Arrow], a columnar data format, to
+  process your queries in a vectorized manner and SIMD to optimize CPU usage.
+  This enables cache-coherent algorithms and high performance on modern processors. 
+
+- **Open Source:** Polars is and always will be open source. Driven by an active
+  community of developers. Everyone is encouraged to add new features and contribute.
+  It is free to use under the MIT license.
+
+:::{rubric} Data formats
+:::
+
+Polars supports reading and writing to many common data formats.
+This allows you to easily integrate Polars into your existing data stack.
+ 
+- Text: CSV & JSON
+- Binary: Parquet, Delta Lake, AVRO & Excel
+- IPC: Feather, Arrow
+- Databases: MySQL, Postgres, SQL Server, Sqlite, Redshift & Oracle
+- Cloud Storage: S3, Azure Blob & Azure File
+
+```{div}
+:style: "clear: both"
+```
+
+
+## Examples
+
+How to use CrateDB together with popular open-source dataframe libraries.
+
+## Dask
+- [Guide to efficient data ingestion to CrateDB with pandas and Dask]
+- [Efficient batch/bulk INSERT operations with pandas, Dask, and SQLAlchemy]
+- [Import weather data using Dask]
+- [Dask code examples]
+
+## pandas
+- [Guide to efficient data ingestion to CrateDB with pandas]
+- [Importing Parquet files into CrateDB using Apache Arrow and SQLAlchemy]
+- [pandas code examples]
+ 
+## Polars
+- [Polars code examples]
+
+
+
+[Apache Arrow]: https://arrow.apache.org/
+[Dask]: https://www.dask.org/
+[Dask DataFrames]: https://docs.dask.org/en/latest/dataframe.html
+[Dask Futures]: https://docs.dask.org/en/latest/futures.html
+[pandas]: https://pandas.pydata.org/
+[Polars]: https://pola.rs/
+
+[Dask code examples]: https://github.com/crate/cratedb-examples/tree/main/by-dataframe/dask
+[Efficient batch/bulk INSERT operations with pandas, Dask, and SQLAlchemy]: https://cratedb.com/docs/python/en/latest/by-example/sqlalchemy/dataframe.html
+[Guide to efficient data ingestion to CrateDB with pandas]: https://community.cratedb.com/t/guide-to-efficient-data-ingestion-to-cratedb-with-pandas/1541
+[Guide to efficient data ingestion to CrateDB with pandas and Dask]: https://community.cratedb.com/t/guide-to-efficient-data-ingestion-to-cratedb-with-pandas-and-dask/1482
+[Import weather data using Dask]: https://github.com/crate/cratedb-examples/blob/main/topic/timeseries/dask-weather-data-import.ipynb
+[Importing Parquet files into CrateDB using Apache Arrow and SQLAlchemy]: https://community.cratedb.com/t/importing-parquet-files-into-cratedb-using-apache-arrow-and-sqlalchemy/1161
+[pandas code examples]: https://github.com/crate/cratedb-examples/tree/main/by-dataframe/pandas
+[Polars code examples]: https://github.com/crate/cratedb-examples/tree/main/by-dataframe/polars