Skip to content

docs(README): Redirect documentation to the Trunk Data Plateform website #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SteBaum merged 1 commit into from
Jan 30, 2026
Merged

docs(README): Redirect documentation to the Trunk Data Plateform website #106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@

TDP is an open source project hosted on [GitHub](https://github.com/TOSIT-IO/TDP) administered by the [TOSIT association](https://tosit.fr/). It is released under the [Apache License 2.0](https://github.com/TOSIT-IO/TDP/blob/main/LICENSE).

Learn more about how to [contribute to TDP ](https://www.trunkdataplatform.io/en/contribute/project/contributing) on our website.
Learn more about how to [contribute to TDP](https://www.trunkdataplatform.io/en/contribute/project/contributing) on our website.
161 changes: 37 additions & 124 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,146 +2,59 @@

![](static/tdp_logo.png)

Trunk Data Platform is an Open Source, free, Hadoop distribution.
[Trunk Data Platform](https://www.trunkdataplatform.io) is an Open Source, free, Hadoop distribution built from Apache projects source code.

This distribution is built by EDF (French electricity provider) & DGFIP (Tax Office by the French Ministry of Finance), through an association called TOSIT (The Open source I Trust).
## Authors

TDP is built from Apache projects source code.
This distribution is built by EDF (French electricity provider) & DGFIP (Tax Office by the French Ministry of Finance), through an association called [TOSIT](https://tosit.fr/) (The Open source I Trust).

## TDP repositories
## Local build environment for components

The TDP project is composed of multiple repositories:
- [tdp-collection](https://github.com/TOSIT-IO/tdp-collection): main Ansible collection to deploy TDP core components.
- [tdp-collection-extras](https://github.com/TOSIT-IO/tdp-collection-extras): Ansible collection to deploy extra components that are not part of TDP core.
- [tdp-collection-prerequisites](https://github.com/TOSIT-IO/tdp-collection-prerequisites): Ansible collection to deploy prerequisite components to a TDP installation (i.e.: KDC, PostgreSQL, etc.).
- [tdp-lib](https://github.com/TOSIT-IO/tdp-lib): Python library to configure, manage and deploy TDP.
- [tdp-server](https://github.com/TOSIT-IO/tdp-server): REST API for tdp-lib orchestration.
- [tdp-ui](https://github.com/TOSIT-IO/tdp-ui): Web UI for TDP clusters deployment and configuration, uses tdp-server.
- [tdp-getting-started](https://github.com/TOSIT-IO/tdp-getting-started): A ready to deploy TDP virtual environment based of Vagrant showcasing how to use every component of TDP.
In order to build the TDP components, two distinct images have been made:

Each component of TDP also has its own repository.
- [tdp-builder]((build-env/README.md)) containing Maven for Java compilation of the Apache Hadoop environment components.
- [tdp-builder-python](build-env-python/README.md) containing a manylinux2014 image with different Python versions for the packaging of JupyterHub, Jupyterlab, Sparkmagic and Hue.

## Trunk Data Platform Release TDP-1.0 components version
### tdp-builder

### TDP Core
Run the following script to build the image, open the container and be ready to compile the components with Maven:

The following table shows the core components of TDP as well as the Apache branch they were based on and the TDP branch which serves as base for our releases.
```sh
./bin/start-build-env.sh
```

| Component | Version | Apache Git branch | TDP Git Branch | TDP commits |
| --------------------------- | ---------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| Apache ZooKeeper | 3.4.6 | release-3.4.6 | XXX | X.X.X |
| Apache Hadoop | 3.1.1-0.0 | [rel/release-3.1.1](https://github.com/apache/hadoop/commits/branch-3.1.1) | [branch-3.1.1-TDP](https://github.com/TOSIT-IO/hadoop/commits/branch-3.1.1-TDP) | [compare](https://github.com/TOSIT-IO/hadoop/compare/branch-3.1.1...branch-3.1.1-TDP) |
| Apache Hive | 3.1.3-1.0 | [branch-3.1](https://github.com/apache/hive/commits/branch-3.1) | [branch-3.1-TDP](https://github.com/TOSIT-IO/hive/commits/branch-3.1-TDP) | [compare](https://github.com/TOSIT-IO/hive/compare/branch-3.1...branch-3.1-TDP) |
| Apache Hive 2 (for Spark 3) | 2.3.9-1.0 | [branch-2.3](https://github.com/apache/hive/commits/branch-2.3) | [branch-2.3-TDP](https://github.com/TOSIT-IO/hive/commits/branch-2.3-TDP) | [compare](https://github.com/TOSIT-IO/hive/compare/branch-2.3...branch-2.3-TDP) |
| Apache Hive 1 (for Spark 2) | 1.2.3-1.0 | [branch-1.2](https://github.com/apache/hive/commits/branch-1.2) | [branch-1.2-TDP](https://github.com/TOSIT-IO/hive/commits/branch-1.2-TDP) | [compare](https://github.com/TOSIT-IO/hive/compare/branch-1.2...branch-1.2-TDP) |
| Apache Tez | 0.9.1-1.0 | [branch-0.9.1](https://github.com/apache/tez/commits/branch-0.9.1) | [branch-0.9.1-TDP](https://github.com/TOSIT-IO/tez/commits/branch-0.9.1-TDP) | [compare](https://github.com/TOSIT-IO/tez/compare/branch-0.9.1...branch-0.9.1-TDP) |
| Apache Spark | 2.3.4-1.0 | [branch-2.3](https://github.com/apache/spark/commits/branch-2.3) | [branch-2.3-TDP](https://github.com/TOSIT-IO/spark/commits/branch-2.3-TDP) | [compare](https://github.com/TOSIT-IO/spark/compare/branch-2.3...branch-2.3-TDP) |
| Apache Spark 3 | 3.2.2-0.0 | [branch-3.2](https://github.com/apache/spark/commits/branch-3.2) | [branch-3.2-TDP](https://github.com/TOSIT-IO/spark/commits/branch-3.2-TDP) | [compare](https://github.com/TOSIT-IO/spark/compare/branch-3.2...branch-3.2-TDP) |
| Apache Ranger | 2.0.0-1.0 | [ranger-2.0](https://github.com/apache/ranger/commits/ranger-2.0) | [ranger-2.0-TDP](https://github.com/TOSIT-IO/ranger/commits/ranger-2.0-TDP) | [compare](https://github.com/TOSIT-IO/ranger/compare/ranger-2.0...ranger-2.0-TDP) |
| Apache Solr (for Ranger) | 7.7.3 | releases/lucene-solr/7.7.3 | XXX | X.X.X |
| Apache HBase | 2.1.10-1.0 | [branch-2.1](https://github.com/apache/hbase/commits/branch-2.1) | [branch-2.1-TDP](https://github.com/TOSIT-IO/hbase/commits/branch-2.1-TDP) | [compare](https://github.com/TOSIT-IO/hbase/compare/branch-2.1...branch-2.1-TDP) |
| Apache Phoenix | 5.1.3-1.0 | [5.1](https://github.com/apache/phoenix/commits/5.1) | [5.1.3-TDP](https://github.com/TOSIT-IO/phoenix/commits/5.1.3-TDP) | [compare](https://github.com/TOSIT-IO/phoenix/compare/5.1...5.1.3-TDP) |
| Apache Phoenix Query Server | 6.0.0-0.0 | [6.0.0](https://github.com/apache/phoenix-queryserver/commits/6.0.0) | [6.0.0-TDP](https://github.com/TOSIT-IO/phoenix-queryserver/commits/6.0.0-TDP) | [compare](https://github.com/TOSIT-IO/phoenix-queryserver/compare/6.0.0...6.0.0-TDP) |
| Apache Knox | 1.6.1-0.0 | [v1.6.1](https://github.com/apache/knox/commits/v1.6.1) | [v1.6.1-TDP](https://github.com/TOSIT-IO/knox/commits/v1.6.1-TDP) | [compare](https://github.com/TOSIT-IO/knox/compare/v1.6.1...v1.6.1-TDP) |
| Apache HBase Connectors | 1.0.0-0.0 | [rel/1.0.0](https://github.com/apache/hbase-connectors/commits/rel/1.0.0) | [branch-2.3.4-1.0.0-TDP](https://github.com/TOSIT-IO/hbase-connectors/commits/branch-2.3.4-1.0.0-TDP) | [compare](https://github.com/TOSIT-IO/hbase-connectors/compare/1.0.0...branch-2.3.4-1.0.0-TDP) |
| Apache HBase Operator tools | 1.1.0-0.0 | [rel/1.1.0](https://github.com/apache/hbase-operator-tools/commits/rel/1.1.0) | [branch-1.1.0-TDP](https://github.com/TOSIT-IO/hbase-operator-tools/commits/branch-1.1.0-TDP) | [compare](https://github.com/TOSIT-IO/hbase-operator-tools/compare/branch-1.1.0...branch-1.1.0-TDP) |
The components' versions and their repositories are found in [tdp-core](https://www.trunkdataplatform.io/en/discover/stacks/tdp2-0) and must be compiled in the follwing order:

Versions are approximately based on the [HDP 3.1.5 release](https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/release-notes/content/hdp_relnotes.html).
- Zookeeper
- Hadoop
- Tez
- Spark3
- Hive
- HBase
- Ranger
- Phoenix
- Phoenix-queryserver
- Knox
- HBase Operator tools
- Iceberg

**Note**: For some projects, the Apache foundation maintains sometimes a branch with this the components on which are backported fixes and features. We will be using these branches as much as possible if they are maintained and compatible.
The Maven compilation commands of the different components can be found in the `tdp/README.md` file of each project.

### TDP Extras
### tdp-builder-python

"TDP Extras" carries some projects that cannot be integrated to "TDP Core". There can be different reasons that keep the project outside of the core:
Although the python coded components use the same [tdp-builder-python](build-env-python/README.md) image, they must be packaged seperately in different containers since each component needs its own envrionment:

- The project is not judged as a key component of the Hadoop ecosystem. This is the case of Airflow.
- The project is not active enough. This is the case of Livy that has not been updated in 2 years.
- The project has some incompatibilities with other "TDP Core" projects' releases. This is the case of Kafka 2.X that relies on ZooKeeper 3.5.X (and cannot use the ZooKeeper 3.4.6 of "TDP Core").
- [JupyterHub](https://github.com/TOSIT-IO/jupyterhub/tree/branch-2.3.1-basic/tdp)
- [JupyterLab](https://github.com/TOSIT-IO/jupyterlab/tree/branch-3.2.9-basic/tdp)
- [Sparkmagic](https://github.com/TOSIT-IO/sparkmagic/tree/branch-0.21.0-basic/tdp)
- [Hue](https://github.com/TOSIT-IO/hue/tree/branch-4.11.0-fix/tdp)

| Component | Version | Apache Git branch | TDP Git Branch | TDP commits |
| ---------------------------------- | ------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- |
| Apache ZooKeeper 3.5.9 (for Kafka) | 3.5.9 | release-3.5.9 | XXX | X.X.X |
| Apache Kafka | 2.8.2 | [2.8](https://github.com/TOSIT-IO/kafka/tree/2.8) | [2.8-TDP](https://github.com/TOSIT-IO/kafka/tree/2.8-TDP) | [compare](https://github.com/TOSIT-IO/kafka/compare/2.8...2.8-TDP) |
| Apache Livy | 0.8.0 | [master](https://github.com/TOSIT-IO/incubator-livy/tree/master) | [branch-0.8.0-TDP](https://github.com/TOSIT-IO/incubator-livy/tree/branch-0.8.0-TDP) | [compare](https://github.com/TOSIT-IO/incubator-livy/compare/master...branch-0.8.0-TDP) |
| Apache Airflow | 2.2.2 | 2.2.2 | XXX | X.X.X |
### Special case for the Apache Livy compilation

**Note:** A project can graduate from "TDP Extras" to "TDP Core" if enough people are supporting it and/or if it is made compatible with all the other projects of the stack.
Apache Incubator Livy has its own compilation environment with its own instrsuctions which can be found in the [Incubator Livy project](https://github.com/TOSIT-IO/incubator-livy/tree/branch-0.9.0-fix/tdp).

## Tested operating system (OS)
## Contributing

Only bare metal and virtual machine deployment are tested. Container based OS may work but are not guaranteed.
Contributions are always welcome!

- Centos 7
- Rocky 8

Redhat like OS may work but are not guaranteed.

## TDP Components release

Every TDP initial release is built from a reference branch on the Apache Git repository according to the above tables. The main change from the original branches is the version declaration in the pom.xml files.

## Building / Testing environment

The builds / unit testing of the Maven Java projects of each component above can be run in Kubernetes pods which are scheduled by a Jenkins installation also running on Kubernetes.
Kubernetes pods scheduling allows for **truly** reproducible and isolated builds. Jenkins' strong integration with the Java ecosystem is a perfect match to build the components of the distribution.

### Build order

- hadoop
- tez
- hive1
- spark
- hive2
- spark3
- hive
- hbase
- ranger
- phoenix
- phoenix-queryserver
- knox
- hbase-spark-connector
- hbase-operator-tools

### Kubernetes

Kubernetes was installed on Ubuntu 20.04 Virtual Machines with [kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/).

**Note:** It is strongly recommended to deploy a Storage Class in order to have persistence on the Kubernetes cluster (useful for Jenkins among others). In our case, we are using [Rook](https://rook.io/) on physical drives attached to the Kubernetes cluster's VMs.

### Jenkins

Jenkins is used to trigger the builds which is the same process for every component of the stack:

- Git clone the sources
- Build the project
- Run the unit tests
- Publish the artifacts to a remote repository

Jenkins was installed on the Kubernetes cluster with the official [jenkinsci Helm chart](https://github.com/jenkinsci/helm-charts).

### Nexus / Docker registry

The building environment needs multiple registries:

- Maven to host the compiled Jars
- Docker to host the images that we use to build the projects
- File registry to host the .tar.gz files with the binaries and jars for every compiled projects.

Nexus Repository OSS can assume all three roles, is free and open source.

Nexus OSS was install on the Kubernetes cluster with the [helm chart](https://github.com/Oteemo/charts/tree/master/charts/sonatype-nexus) provided by [Oteemo](https://github.com/Oteemo).

## Local build environment

It is possible to run a local environment for building / small scale testing.

Prerequisite:

- Docker installed and available to your local user

You can start a local building environment with the `bin/start-build-env.sh` script.

**Note:** See `build-env/README.md` for details.

To build TDP component binaries, attach to the running `tdp-builder` container and `git clone` the TDP component repository to it. Each TDP component's `tdp/README.md` has custom instructions to launch the build process.
Assign a directory path to the `TDP_HOME` variable in the `bin/start-build-env.sh` to control the local path of built TDP binaries.
See [CONTRIBUTING.md](./CONTRIBUTING.md) for ways to get started.
2 changes: 1 addition & 1 deletion build-env-python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,4 @@ docker build . -t tdp-builder-python

Contrary to the `tdp-builder` container where components are compiled with maven putting the jar files in the `.m2` cache it is not the case here and therefore volumes, working directories and even users are different for each component.

Check the documentation of the concerned component for the command to start the container.
Check the documentation of the concerned component for the command to start the container.