|
1 | | -# Why Timeplus? |
| 1 | +# What is Timeplus |
| 2 | + |
| 3 | +Timeplus is a unified real-time data processing platform built for developers who need to move, transform, and act on data fast. At the heart of Timeplus is an incremental processing engine that uses modern vectorization (SIMD), just-in-time (JIT) compilation, and advanced database internals to ingest, transform, store, and serve data with low latency and high throughput. |
| 4 | + |
| 5 | +It plugs right into the tools you already use — stream sources like Kafka, Redpanda, and Pulsar, and sinks like ClickHouse, Apache Iceberg, S3, Splunk, Elasticsearch, and MongoDB. You can easily build pipelines that consume events, run streaming ETL, joins, aggregations, filtering, and other transformations, then push the results wherever they need to go — fast. |
| 6 | + |
| 7 | +Timeplus isn’t just for streaming. It also supports scheduled batch jobs, so you can mix real-time and periodic workloads in one place. Pair it with Timeplus Alert to trigger actions on live data and manage the full lifecycle of your data applications—without duct-taping multiple tools together. |
| 8 | + |
| 9 | +## Why Timeplus |
2 | 10 |
|
3 | 11 | Timeplus simplifies stateful stream processing and analytics with a fast, single-binary engine. Using SQL as a domain-specific language and both row and column-based state stores, it enables developers to build real-time applications, data pipelines, and analytical dashboards at the edge or in the cloud, reducing the cost, time, and complexity of multi-component stacks. |
4 | 12 |
|
5 | | -## Architecture: The Best of Both Worlds {#architecture} |
| 13 | +### Architecture: The Best of Both Worlds {#architecture} |
| 14 | + |
6 | 15 |  |
7 | 16 |
|
8 | | -## Unified streaming and historical data processing {#unified} |
| 17 | +### Unified Streaming and Historical Data Processing {#unified} |
9 | 18 |
|
10 | | -Timeplus streams offer high performance, resiliency, and seamless querying by using an internal Write Ahead Log (WAL) and Historical Store. The WAL ensures ultra-fast inserts and updates, while the Historical Store, optimized for various query types, handles efficient historical queries. |
| 19 | +Timeplus streams deliver high performance, resiliency, and smooth querying through an internal Write-Ahead Log (WAL, called NativeLog) and a Historical Store. The WAL enables ultra-fast data ingestion, while the Historical Store — stored in either columnar or row format and enhanced with compaction and indexing — supports efficient historical range and point queries. |
11 | 20 |
|
12 | 21 | This architecture transparently serves data to users based on query type from both, often eliminating the need for Apache Kafka as a commit log or a separate downstream database, streamlining your data infrastructure. |
13 | 22 |
|
14 | | -## Append and Mutable Streams {#streams} |
| 23 | +### Append and Mutable Streams {#streams} |
15 | 24 |
|
16 | 25 | Configure types of streams to optimize performance. |
17 | 26 |
|
18 | 27 | * [Append streams:](/append-stream) |
19 | 28 | Excel at complex aggregations, storing data in a columnar format for faster access and processing. |
20 | | -* [Mutable streams:](/mutable-stream) Support UPSERTs and DELETEs, ideal for applications like Materialized Caches or GDPR compliance, using a row-based store optimized for fast data retrieval and query consistency. |
21 | 29 |
|
22 | | -## Single Binary {#binary} |
23 | | -Timeplus is a fast, powerful, and efficient SQL stream processing platform with no dependencies, JVM, or ZK. It runs in bare-metal or Kubernetes environments, from edge to cloud, using a single binary (~150MB). |
| 30 | +* [Mutable streams:](/mutable-stream) Support UPSERTs and DELETEs, ideal for applications which require high frequent and high cardinality data mutations, using a row-based store optimized for fast data retrieval and query consistency. |
| 31 | + |
| 32 | +### Single Binary {#binary} |
| 33 | + |
| 34 | +Timeplus is a fast, powerful, and efficient SQL stream processing platform with no dependencies, JVM, or ZK. It runs in bare-metal or Kubernetes environments, from edge to cloud, using a single binary. |
24 | 35 |
|
25 | 36 | Timeplus scales easily from edge devices to multi-node clusters, and with its Append-Only data structures and historical stores, some use cases may not need Kafka or a separate database at all. |
26 | 37 |
|
27 | | -## Multi-JOINs and ASOF JOINs {#join} |
| 38 | +### Multi-JOINs and ASOF JOINs {#join} |
28 | 39 |
|
29 | 40 | Stream processing involves combining multiple data sources, and [MULTI-JOINs](/joins) are essential for enriching and correlating events in streaming queries. Timeplus allows you to run ad-hoc historical queries on the same data, reducing the need for denormalization in downstream data warehouses. |
30 | 41 |
|
31 | 42 | In many cases, Business Intelligence and analytical queries can be executed directly in Timeplus, eliminating the need for a separate data warehouse. [ASOF JOINs](/joins) enable approximate time-based lookups for comparing recent versus historical data. |
32 | 43 |
|
33 | | -## Python and JavaScript UDF {#udf} |
| 44 | +### Python and JavaScript UDF {#udf} |
| 45 | + |
34 | 46 | We understand that SQL may not be able to express all business logic for streaming or querying. [JavaScript](/js-udf) and [Python](/py-udf) User Defined Functions (UDFs) and User Defined Aggregate Functions (UDAFs) can be used to extend Timeplus to encapsulate custom logic for both stateless and stateful queries. |
35 | 47 |
|
36 | 48 | With Python UDFs, this opens up the possibility to bring in pre-existing and popular libraries, including data science and machine learning libraries! |
37 | 49 |
|
38 | | -## External Stream, External Table {#external} |
| 50 | +### External Stream, External Table {#external} |
| 51 | + |
39 | 52 | We want to simplify the experience of joining data from Apache Kafka and writing results out to data warehouses such as Clickhouse, or another Timeplus instance. Timeplus implements native integration to these systems in timeplusd via EXTERNAL STREAM (with [Kafka](/proton-kafka) and [Timeplus](/timeplus-external-stream)) and [EXTERNAL TABLE (with ClickHouse)](/proton-clickhouse-external-table). No need for deploying yet another Connector component. |
40 | 53 |
|
41 | 54 | We understand that we cannot do this for all systems and for that, we have Timeplus Connector which can be configured to integrate with hundreds of other systems if needed. |
42 | 55 |
|
43 | | -## Collection {#collection} |
44 | | -With built-in External Streams and External Tables, Timeplus can natively collect real-time data from, or send data to, Kafka, Redpanda, ClickHouse, or another Timeplus instance, without any data duplication. |
| 56 | +### Collection {#collection} |
| 57 | + |
| 58 | +With built-in External Streams and External Tables, Timeplus can natively collect real-time data from, or send data to, Kafka, Redpanda, ClickHouse, or another Timeplus instance, without duplicating data in yet another place. |
45 | 59 |
|
46 | 60 | Timeplus also supports a wide range of data sources through sink/source connectors. Users can push data from files (CSV/TSV), via native SDKs in Java, Go, or Python, JDBC/ODBC, Websockets, or REST APIs. |
47 | 61 |
|
48 | | -## Transformation {#transformation} |
| 62 | +### Transformation {#transformation} |
| 63 | + |
49 | 64 | With a powerful streaming SQL console, users can leverage their preferred query language to create Streams, Views, and incremental Materialized Views. This enables them to transform, roll up, join, correlate, enrich, aggregate, and downsample real-time data, generating meaningful outputs for real-time alerting, analytics, or any downstream systems. |
50 | 65 |
|
51 | | -## Routing {#routing} |
| 66 | +### Routing {#routing} |
| 67 | + |
52 | 68 | Timeplus allows data to be routed to different sinks based on SQL-based criteria and provides a data lineage view of all derived streams in its console. A single data result can generate multiple outputs for various scenarios and systems, such as analytics, alerting, compliance, etc., without any vendor lock-in. |
53 | 69 |
|
54 | | -## Analytics and Alerting {#alerts} |
| 70 | +### Analytics and Alerting {#alerts} |
| 71 | + |
55 | 72 | Powered by SSE (Server-Sent Events), Timeplus supports push-based, low-latency dashboards to visualize real-time insights through data pipelines or ad-hoc queries. Additionally, users can easily build observability dashboards using Grafana plugins. |
56 | 73 |
|
57 | 74 | SQL-based rules can be used to trigger or resolve alerts in systems such as PagerDuty, Slack, and other downstream platforms. |
| 75 | + |
| 76 | +### Scalability and Elasticity |
| 77 | + |
| 78 | +Timeplus supports both MPP architecture for pure on-prem deployments—ideal when ultra-low latency is critical and storage/compute separation for elastic, cloud-native setups. In the latter mode, S3 is used for the NativeLog, Historical Store, and Query State Checkpoints. Combined with Kubernetes HPA or AWS Auto Scaling Groups, this enables highly concurrent continuous queries on clusters that scale automatically with demand. |
0 commit comments