Skip to content

Commit 88c1e45

Browse files
arnaud-lacuriearnaud-lacurie
committed
Rewrite README to emphasize SQL/relational API over protobuf API
1 parent 93fc505 commit 88c1e45

File tree

1 file changed

+197
-33
lines changed

1 file changed

+197
-33
lines changed

‎README.md‎

Lines changed: 197 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -4,43 +4,207 @@ FoundationDB is a distributed database designed to handle large volumes of struc
44

55
To learn more about FoundationDB, visit [foundationdb.org](https://www.foundationdb.org/)
66

7-
# FoundationDB Record Layer
8-
9-
The Record Layer is a Java API providing a record-oriented store on top of FoundationDB,
10-
(very) roughly equivalent to a simple relational database, featuring:
11-
12-
* **Structured types** - Records are defined and stored in terms of
13-
[protobuf](https://developers.google.com/protocol-buffers/) messages.
14-
* **Indexes** - The Record Layer supports a variety of different index
15-
types including value indexes (the kind provided by most databases),
16-
rank indexes, and aggregate indexes. Indexes and primary keys can
17-
be defined either via protobuf options or programmatically.
18-
* **Complex types** - Support for complex types, such as lists and
19-
nested records, including the ability to define indexes against
20-
such nested structures.
21-
* **Queries** - The Record Layer does not provide a query language, however
22-
it provides query APIs with the ability to scan, filter, and sort
23-
across one or more record types, and a query planner capable of
24-
automatic selection of indexes.
25-
* **Many record stores, shared schema** - The Record Layer provides the
26-
ability to support many discrete record store instances, all with
27-
a shared (and evolving) schema. For example, rather than modeling a
28-
single database in which to store all users' data, each user can be
29-
given their own record store, perhaps sharded across different FDB
30-
cluster instances.
31-
* **Very light weight** - The Record layer is designed to be used in a
32-
large, distributed, stateless environment. The time between opening
33-
a store and the first query is intended to be measured in milliseconds.
34-
* **Extensible** - New index types and custom index key expressions
35-
may be dynamically incorporated into a record store.
36-
37-
The Record Layer may be used directly or provides an excellent foundational
38-
layer on which more complex systems can be constructed.
7+
# FoundationDB Record Layer (FRL)
8+
9+
FRL provides a **relational database with SQL support** built on top of FoundationDB, featuring:
10+
11+
* **SQL Database** - SQL support with JDBC connectivity for defining schemas,
12+
querying data, and managing tables using familiar SQL syntax. The SQL API is under
13+
active development with frequent enhancements.
14+
* **Advanced Data Types** - Beyond standard SQL types (STRING, INTEGER, FLOAT, BOOLEAN),
15+
FRL supports:
16+
* **Nested Structures** - User-defined struct types that can be nested arbitrarily deep
17+
* **Arrays** - Collections of primitives or complex types
18+
* **Vectors** - Fixed-dimension numerical vectors for ML embeddings and similarity search
19+
* **Schema Templates** - Reusable schema definitions that enable multi-tenant architectures
20+
where each tenant gets their own database instance with a shared, evolvable schema.
21+
* **Intelligent Query Planning** - Automatic index selection and query optimization with
22+
support for JOINs, aggregations (COUNT, SUM, etc.), GROUP BY, and ORDER BY.
23+
Queries are efficiently executed using index-backed operations without in-memory sorting.
24+
* **Indexes** - Rich indexing capabilities including value indexes, rank indexes, aggregate
25+
indexes, and indexes on nested fields. Indexes are materialized views that update incrementally.
26+
* **Scalable Architecture** - Designed for distributed, stateless environments with
27+
millisecond-level store initialization. Perfect for applications managing thousands
28+
of discrete database instances.
29+
* **ACID Transactions** - Full transactional semantics inherited from FoundationDB,
30+
with support for continuations for efficiently paging through large result sets.
31+
32+
## Quick Start with SQL
33+
34+
```java
35+
// Connect via JDBC
36+
String url = "jdbc:embed:/__SYS?schema=CATALOG";
37+
Connection conn = DriverManager.getConnection(url);
38+
39+
// Define a schema template with tables and indexes
40+
conn.createStatement().execute("""
41+
CREATE SCHEMA TEMPLATE my_template
42+
CREATE TABLE customers (
43+
customer_id BIGINT,
44+
name STRING,
45+
email STRING,
46+
PRIMARY KEY(customer_id)
47+
)
48+
CREATE INDEX email_idx AS
49+
SELECT email FROM customers ORDER BY email
50+
""");
51+
52+
// Create a database and schema
53+
conn.createStatement().execute(
54+
"CREATE DATABASE /my_app/production");
55+
conn.createStatement().execute(
56+
"CREATE SCHEMA /my_app/production/main WITH TEMPLATE my_template");
57+
58+
// Insert and query data
59+
PreparedStatement insert = conn.prepareStatement(
60+
"INSERT INTO customers VALUES (?, ?, ?)");
61+
insert.setLong(1, 1);
62+
insert.setString(2, "Alice");
63+
insert.setString(3, "alice@example.com");
64+
insert.executeUpdate();
65+
66+
ResultSet rs = conn.createStatement().executeQuery(
67+
"SELECT * FROM customers WHERE email = 'alice@example.com'");
68+
```
69+
70+
## Key Features
71+
72+
### Multi-Tenant Schema Templates
73+
74+
Schema templates enable efficient multi-tenant architectures:
75+
76+
```sql
77+
-- Define the template once
78+
CREATE SCHEMA TEMPLATE user_data_template
79+
CREATE TABLE documents (id BIGINT, content STRING, PRIMARY KEY(id))
80+
CREATE INDEX content_idx AS SELECT content FROM documents ORDER BY content;
81+
82+
-- Create separate database instances for each tenant
83+
CREATE DATABASE /tenant/user_1;
84+
CREATE SCHEMA /tenant/user_1/data WITH TEMPLATE user_data_template;
85+
86+
CREATE DATABASE /tenant/user_2;
87+
CREATE SCHEMA /tenant/user_2/data WITH TEMPLATE user_data_template;
88+
```
89+
90+
Each tenant's data is completely isolated with its own database, yet all share
91+
the same schema definition for easy management and evolution.
92+
93+
### Advanced Type System
94+
95+
Define complex, nested data structures:
96+
97+
```sql
98+
-- Define custom struct types
99+
CREATE TYPE AS STRUCT address (
100+
street STRING,
101+
city STRING,
102+
postal_code STRING
103+
)
104+
105+
CREATE TYPE AS STRUCT contact_info (
106+
email STRING,
107+
phone STRING,
108+
mailing_address address
109+
)
110+
111+
-- Use in tables with arrays and nesting
112+
CREATE TABLE users (
113+
user_id BIGINT,
114+
name STRING,
115+
contacts contact_info ARRAY,
116+
PRIMARY KEY(user_id)
117+
)
118+
```
119+
120+
### Vector Support for ML Applications
121+
122+
Store and query high-dimensional vectors for embeddings and similarity search:
123+
124+
```sql
125+
CREATE TABLE embeddings (
126+
doc_id BIGINT,
127+
content STRING,
128+
embedding_half VECTOR(128, HALF), -- 16-bit precision
129+
embedding_float VECTOR(768, FLOAT), -- 32-bit precision
130+
embedding_double VECTOR(1024, DOUBLE), -- 64-bit precision
131+
PRIMARY KEY(doc_id)
132+
)
133+
```
134+
135+
Vectors are inserted via JDBC PreparedStatements and can be efficiently stored
136+
and retrieved using the FoundationDB backend.
137+
138+
### Index-Backed Query Execution
139+
140+
FRL's query planner intelligently selects indexes to execute
141+
queries efficiently:
142+
143+
```sql
144+
-- Queries use indexes automatically
145+
SELECT name FROM customers WHERE email = 'alice@example.com';
146+
-- Uses email_idx if available
147+
148+
-- JOINs using comma-separated FROM clause
149+
SELECT c.name, o.order_id
150+
FROM customers c, orders o
151+
WHERE c.customer_id = o.customer_id;
152+
153+
-- Aggregations backed by indexes
154+
SELECT category, COUNT(*)
155+
FROM products
156+
GROUP BY category;
157+
-- Requires ordered index or primary key on category for streaming aggregate, or a aggregate index for direct retrieval
158+
159+
-- ORDER BY requires index or primary key order
160+
SELECT * FROM customers ORDER BY email;
161+
-- Requires index on email (like email_idx above)
162+
```
163+
164+
**Important**: FRL does not perform in-memory sorting or aggregation. Operations like ORDER BY, GROUP BY,
165+
and aggregates require underlying indexes to provide the required ordering.
166+
167+
## Architecture Notes
168+
169+
FRL is designed for:
170+
* **Horizontal scalability** - Thousands of independent database instances
171+
* **Low latency** - Millisecond-level initialization and query execution
172+
* **Stateless services** - No server-side state; all data in FoundationDB
173+
* **Schema evolution** - Templates can evolve over time (template evolution features
174+
coming to relational layer; currently available via advanced Record Layer API)
175+
176+
## Advanced: Direct Record Layer API
177+
178+
For applications requiring fine-grained control over storage layout, index
179+
maintenance, or features not yet available in the SQL Relational layer, the Record Layer
180+
provides a low-level Java API using Protocol Buffers.
181+
182+
**Note**: This API is maintained for advanced use cases but is being positioned
183+
as a lower-level alternative to the SQL interface. Features available only
184+
through this API will migrate to the SQL layer over time. Long-term support of this lower-level API is not guaranteed once equivalent features are available at the Relational Layer.
185+
186+
Key Record Layer API features:
187+
* **Protobuf-based schema definition** - Define records using `.proto` files
188+
* **Programmatic index management** - `IndexMaintainer` extension points
189+
* **Custom query components** - Extend the query planner
190+
* **Schema evolution** - `MetaDataEvolutionValidator` for safe schema changes
191+
* **Low-level control** - Direct access to FoundationDB operations
192+
193+
See [Record Layer Documentation](https://foundationdb.github.io/fdb-record-layer/Overview.html) for details.
39194

40195
## Documentation
41196

42-
* [Documentation Home](https://foundationdb.github.io/fdb-record-layer/)
197+
* **Documentation Home** - [Documentation](https://foundationdb.github.io/fdb-record-layer/)
198+
* **Getting Started** - [SQL Quick Start](https://foundationdb.github.io/fdb-record-layer/GettingStarted.html)
199+
* **SQL Reference** - [SQL Commands and Data Types](https://foundationdb.github.io/fdb-record-layer/SQL_Reference.html)
200+
* **Schema Templates** - [Databases, Schemas, and Templates](https://foundationdb.github.io/fdb-record-layer/reference/Databases_Schemas_SchemaTemplates.html)
201+
* **Advanced: Record Layer API** - [Record Layer Overview](https://foundationdb.github.io/fdb-record-layer/Overview.html)
43202
* [Contributing](CONTRIBUTING.md)
44203
* [Code of Conduct](CODE_OF_CONDUCT.md)
45204
* [License](LICENSE)
46205

206+
## Getting Help
207+
208+
* **Documentation Issues**: [Submit Documentation Feedback](https://github.com/FoundationDB/fdb-record-layer/issues)
209+
* **Bugs & Feature Requests**: [GitHub Issues](https://github.com/FoundationDB/fdb-record-layer/issues)
210+
* **Community**: [FoundationDB Community Forums](https://forums.foundationdb.org/)

0 commit comments

Comments
 (0)