docs(2.9): document indexes and DBs compatibility

README.md

@@ -204,14 +204,21 @@ helm show values codefresh/codefresh
 
 ### Persistent services
 
-The following table displays the list of **persistent** services created as part of the on-premises installation:
+Codefresh relies on several persistent services to store its data:
 
-| Database      | Purpose | Required version     |
-| :---        | :----   |  :--- |
-| MongoDB      | Stores all account data (account settings, users, projects, pipelines, builds etc.)       | 7.x   |
-| Postgresql   | Stores data about events for the account (pipeline updates, deletes, etc.). The audit log uses the data from this database.        | 16.x or 17.x      |
-| Redis   | Used for caching, and as a key-value store for cron trigger manager.        | 7.0.x      |
-| RabbitMQ  | Used for message queueing.        | 3.13 \| 4.0.x      |
+- **MongoDB**: Stores all account data (account settings, users, projects, pipelines, builds etc.)
+- **PostgreSQL**: Stores data about events for the account (pipeline updates, deletes, etc.). The audit log uses the data from this database.
+- **Redis**: Used for caching, and as a key-value store for cron trigger manager.
+- **RabbitMQ**: Used for message queueing.
+
+The following table reflects the recommended and supported versions of these databases for different Codefresh releases:
+
+| Codefresh version | MongoDB | PostgreSQL | Redis | RabbitMQ |
+| :---              | :---    | :---       | :---  | :---     |
+| 2.9.x             | \>=4.2 \<=7.x <br> Recommended: 7.x (`featureCompatibilityVersion: 7.0`)| \>= 16.x \<= 17.x <br> Recommended: 17.x | \>= 7.0.x \<= 7.4.x <br> Recommended: 7.4.x | 3.13.x \| 4.0.x \| 4.1.x <br> Recommended: 4.1.x |
+| 2.8.x             | \>=4.2 \<=7.x <br> Recommended: 7.x (`featureCompatibilityVersion: 6.0`)| \>= 13.x \<= 17.x <br> Recommended: 16.x \| 17.x | \>= 7.0.x \<= 7.4.x <br> Recommended: 7.4.x | 3.13.x \| 4.0.x \| 4.1.x <br> Recommended: 4.0.x |
+| 2.7.x             | \>=4.2 \<=6.x <br> Recommended: 6.x (`featureCompatibilityVersion: 6.0`)| 13.x | 7.0.x | 3.13.x |
+| 2.6.x             | \>=4.2 \<=6.x <br> Recommended: 6.x (`featureCompatibilityVersion: 5.0`)| 13.x | 7.0.x | 3.13.x |
 
 > Running on netfs (nfs, cifs) is not recommended.
 
@@ -1484,7 +1491,8 @@ Use [obtain-oidc-id-token](https://github.com/codefresh-io/steps/blob/822afc0a9a
 
 Sometimes, in new releases of Codefresh On-Prem, index requirements change. When this happens, it's mentioned in the [Upgrading section](#upgrading) for the specific release.
 
-> ℹ️ If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
+> [!TIP]
+> If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
 
 ### Index alignment
 
@@ -1493,10 +1501,9 @@ The required index definitions for each release can be found at the following re
 - `2.6` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.6/indexes>
 - `2.7` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.7/indexes>
 - `2.8` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.8/indexes>
+- `2.9` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.9/indexes>
 
-The indexes are stored in JSON files with keys and options specified.
-
-The directory structure is:
+The indexes specifications are stored in JSON files. The directory structure is:
 
 ```console
 indexes
@@ -1507,11 +1514,12 @@ indexes
 **Overview of the index alignment process:**
 
 1. Identify the differences between the indexes in your MongoDB instance and the required index definitions.
-2. Create any missing indexes one by one. (It's important not to create them in bulk.)
+2. Create any missing indexes.
 3. Perform the upgrade of Codefresh On-Prem installation.
 4. Then remove any unnecessary indexes.
 
-> ⚠️ **Note! Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
+> [!IMPORTANT]
+> Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
 >
 > Building indexes during time periods where the target collection is under heavy write load can result in reduced write performance and longer index builds. ([*Source: MongoDB official documentation*](https://www.mongodb.com/docs/manual/core/index-creation/#index-build-impact-on-database-performance))
 >
@@ -1533,19 +1541,32 @@ mongosh "<connection_string>"
 db.getSiblingDB('<db_name>').getCollection('<collection_name>').getIndexes()
 ```
 
-- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones
+- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones.
 
 **Index creation**
 
-> ⚠ **Note! Always create indexes sequentially, one by one. Don't create them in bulk.**
+- To create an indexes, we recommend using the `createIndexes` command ([ref](https://www.mongodb.com/docs/manual/reference/command/createIndexes/)):
 
-- To create an index, use the `createIndex()` method:
+> [!IMPORTANT]
+> We recommend to create indexes in batches of 3 indexes at a time.
+> However, it's highly recommended before creating indexes in production DB to test performance impact on a staging instance with prod-like amount of data.
+>
+> Previous command should be completed before starting the next batch.
 
 ```js
-db.getSiblingDB('<db_name>').getCollection('<collection_name>').createIndex(<keys_object>, <options_object>)
+db.getSiblingDB('<db_name>').runCommand(
+  {
+    createIndexes: '<collection_name>',
+    indexes: [
+        { ... },  // Index definition from the doc above
+        { ... },  // Index definition from the doc above
+        { ... }   // Index definition from the doc above
+    ],
+  }
+)
 ```
 
-After executing the `createIndex()` command, you should see a result indicating that the index was created successfully.
+After executing the command, you should see a result indicating that the indexes were created successfully.
 
 **Index removal**
 
@@ -1559,7 +1580,8 @@ db.getSiblingDB('<db_name>').getCollection('<collection_name>').dropIndex('<inde
 
 If you're hosting MongoDB on [Atlas](https://www.mongodb.com/atlas/database), use the following [Manage Indexes](https://www.mongodb.com/docs/atlas/atlas-ui/indexes/) guide to View, Create or Remove indexes.
 
-> ⚠️ **Important!** In Atlas, for production environments, it is recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v6.0/tutorial/build-indexes-on-replica-sets/))
+> [!IMPORTANT]
+> In Atlas, for production environments, it may be recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v7.0/tutorial/build-indexes-on-replica-sets/))
 
 ## Upgrading
 
@@ -2356,10 +2378,26 @@ For built-in RabbitMQ `bitnami/rabbitmq` subchart, pre-upgrade hook was added to
 
 ### To 2-9-0
 
+> [!WARNING]
+> **BREAKING CHANGES**
+>
+> Default DinD image has been upgraded to 28.x, which removes support for pushing and pulling with legacy image manifest v2 schema 1 ([ref](https://docs.docker.com/engine/deprecated/#pushing-and-pulling-with-image-manifest-v2-schema-1)).
+>
+> Before upgrading Codefresh, please follow the instruction in [this doc](https://codefresh.io/docs/docs/kb/articles/upgrade-deprecated-docker-images/) to identify deprecated images, upgrade them, and then proceed with upgrading the platform.
+
 ####  Affected values
 
 - `.Values.runner` is removed
 
+#### Changes in MongoDB schema
+
+Changes in indexes: follow [Maintaining MongoDB indexes](#maintaining-mongodb-indexes) guide to meet index requirements *before* the upgrade process.
+
+Changes in collections: following collections can be safely dropped *after* the upgrade to 2.9.x if they exist. These collections are no longer used and should be removed to maintain optimal database performance and prevent the accumulation of obsolete data.
+
+- `read-models.application-tree`
+- `read-models.<entity>-history` — every collection with `~-history` suffix, such as `read-models.applications-history`, `read-models.services-history`, etc.
+
 ## Troubleshooting
 
 ### Error: Failed to validate connection to Docker daemon; caused by Error: certificate has expired

README.md.gotmpl

@@ -205,14 +205,21 @@ helm show values codefresh/codefresh
 
 ### Persistent services
 
-The following table displays the list of **persistent** services created as part of the on-premises installation:
+Codefresh relies on several persistent services to store its data:
 
-| Database      | Purpose | Required version     |
-| :---        | :----   |  :--- |
-| MongoDB      | Stores all account data (account settings, users, projects, pipelines, builds etc.)       | 7.x   |
-| Postgresql   | Stores data about events for the account (pipeline updates, deletes, etc.). The audit log uses the data from this database.        | 16.x or 17.x      |
-| Redis   | Used for caching, and as a key-value store for cron trigger manager.        | 7.0.x      |
-| RabbitMQ  | Used for message queueing.        | 3.13 \| 4.0.x      |
+- **MongoDB**: Stores all account data (account settings, users, projects, pipelines, builds etc.)
+- **PostgreSQL**: Stores data about events for the account (pipeline updates, deletes, etc.). The audit log uses the data from this database.
+- **Redis**: Used for caching, and as a key-value store for cron trigger manager.
+- **RabbitMQ**: Used for message queueing.
+
+The following table reflects the recommended and supported versions of these databases for different Codefresh releases:
+
+| Codefresh version | MongoDB | PostgreSQL | Redis | RabbitMQ |
+| :---              | :---    | :---       | :---  | :---     |
+| 2.9.x             | \>=4.2 \<=7.x <br> Recommended: 7.x (`featureCompatibilityVersion: 7.0`)| \>= 16.x \<= 17.x <br> Recommended: 17.x | \>= 7.0.x \<= 7.4.x <br> Recommended: 7.4.x | 3.13.x \| 4.0.x \| 4.1.x <br> Recommended: 4.1.x |
+| 2.8.x             | \>=4.2 \<=7.x <br> Recommended: 7.x (`featureCompatibilityVersion: 6.0`)| \>= 13.x \<= 17.x <br> Recommended: 16.x \| 17.x | \>= 7.0.x \<= 7.4.x <br> Recommended: 7.4.x | 3.13.x \| 4.0.x \| 4.1.x <br> Recommended: 4.0.x |
+| 2.7.x             | \>=4.2 \<=6.x <br> Recommended: 6.x (`featureCompatibilityVersion: 6.0`)| 13.x | 7.0.x | 3.13.x |
+| 2.6.x             | \>=4.2 \<=6.x <br> Recommended: 6.x (`featureCompatibilityVersion: 5.0`)| 13.x | 7.0.x | 3.13.x |
 
 > Running on netfs (nfs, cifs) is not recommended.
 
@@ -1492,7 +1499,8 @@ Use [obtain-oidc-id-token](https://github.com/codefresh-io/steps/blob/822afc0a9a
 
 Sometimes, in new releases of Codefresh On-Prem, index requirements change. When this happens, it's mentioned in the [Upgrading section](#upgrading) for the specific release.
 
-> ℹ️ If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
+> [!TIP]
+> If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
 
 ### Index alignment
 
@@ -1501,10 +1509,9 @@ The required index definitions for each release can be found at the following re
 - `2.6` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.6/indexes>
 - `2.7` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.7/indexes>
 - `2.8` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.8/indexes>
+- `2.9` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.9/indexes>
 
-The indexes are stored in JSON files with keys and options specified.
-
-The directory structure is:
+The indexes specifications are stored in JSON files. The directory structure is:
 
 ```console
 indexes
@@ -1515,11 +1522,12 @@ indexes
 **Overview of the index alignment process:**
 
 1. Identify the differences between the indexes in your MongoDB instance and the required index definitions.
-2. Create any missing indexes one by one. (It's important not to create them in bulk.)
+2. Create any missing indexes.
 3. Perform the upgrade of Codefresh On-Prem installation.
 4. Then remove any unnecessary indexes.
 
-> ⚠️ **Note! Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
+> [!IMPORTANT]
+> Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
 >
 > Building indexes during time periods where the target collection is under heavy write load can result in reduced write performance and longer index builds. ([*Source: MongoDB official documentation*](https://www.mongodb.com/docs/manual/core/index-creation/#index-build-impact-on-database-performance))
 >
@@ -1542,19 +1550,32 @@ mongosh "<connection_string>"
 db.getSiblingDB('<db_name>').getCollection('<collection_name>').getIndexes()
 ```
 
-- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones
+- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones.
 
 **Index creation**
 
-> ⚠ **Note! Always create indexes sequentially, one by one. Don't create them in bulk.**
+- To create an indexes, we recommend using the `createIndexes` command ([ref](https://www.mongodb.com/docs/manual/reference/command/createIndexes/)):
 
-- To create an index, use the `createIndex()` method:
+> [!IMPORTANT]
+> We recommend to create indexes in batches of 3 indexes at a time.
+> However, it's highly recommended before creating indexes in production DB to test performance impact on a staging instance with prod-like amount of data.
+>
+> Previous command should be completed before starting the next batch.
 
 ```js
-db.getSiblingDB('<db_name>').getCollection('<collection_name>').createIndex(<keys_object>, <options_object>)
+db.getSiblingDB('<db_name>').runCommand(
+  {
+    createIndexes: '<collection_name>',
+    indexes: [
+        { ... },  // Index definition from the doc above
+        { ... },  // Index definition from the doc above
+        { ... }   // Index definition from the doc above
+    ],
+  }
+)
 ```
 
-After executing the `createIndex()` command, you should see a result indicating that the index was created successfully.
+After executing the command, you should see a result indicating that the indexes were created successfully.
 
 **Index removal**
 
@@ -1568,7 +1589,8 @@ db.getSiblingDB('<db_name>').getCollection('<collection_name>').dropIndex('<inde
 
 If you're hosting MongoDB on [Atlas](https://www.mongodb.com/atlas/database), use the following [Manage Indexes](https://www.mongodb.com/docs/atlas/atlas-ui/indexes/) guide to View, Create or Remove indexes.
 
-> ⚠️ **Important!** In Atlas, for production environments, it is recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v6.0/tutorial/build-indexes-on-replica-sets/))
+> [!IMPORTANT]
+> In Atlas, for production environments, it may be recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v7.0/tutorial/build-indexes-on-replica-sets/))
 
 ## Upgrading
 
@@ -2366,10 +2388,26 @@ For built-in RabbitMQ `bitnami/rabbitmq` subchart, pre-upgrade hook was added to
 
 ### To 2-9-0
 
+> [!WARNING]
+> **BREAKING CHANGES**
+> 
+> Default DinD image has been upgraded to 28.x, which removes support for pushing and pulling with legacy image manifest v2 schema 1 ([ref](https://docs.docker.com/engine/deprecated/#pushing-and-pulling-with-image-manifest-v2-schema-1)).
+>
+> Before upgrading Codefresh, please follow the instruction in [this doc](https://codefresh.io/docs/docs/kb/articles/upgrade-deprecated-docker-images/) to identify deprecated images, upgrade them, and then proceed with upgrading the platform.
+
 ####  Affected values
 
 - `.Values.runner` is removed
 
+#### Changes in MongoDB schema
+
+Changes in indexes: follow [Maintaining MongoDB indexes](#maintaining-mongodb-indexes) guide to meet index requirements *before* the upgrade process.
+
+Changes in collections: following collections can be safely dropped *after* the upgrade to 2.9.x if they exist. These collections are no longer used and should be removed to maintain optimal database performance and prevent the accumulation of obsolete data.
+
+- `read-models.application-tree`
+- `read-models.<entity>-history` — every collection with `~-history` suffix, such as `read-models.applications-history`, `read-models.services-history`, etc.
+
 
 ## Troubleshooting
 

indexes/codefresh/agenttasks.json

@@ -1,6 +1,6 @@
 [
     {
-        "keys": {
+        "key": {
             "metadata.accountId": 1,
             "metadata.reIdentifier": 1,
             "metadata.shouldExecute": 1,
@@ -10,11 +10,9 @@
         }
     },
     {
-        "keys": {
+        "key": {
             "metadata.expireAt": 1
         },
-        "options": {
-            "expireAfterSeconds": 0
-        }
+        "expireAfterSeconds": 0
     }
-]
+]

indexes/codefresh/feature-store-versioned.json

@@ -1,16 +1,14 @@
 [
     {
-        "keys": {
+        "key": {
             "createdAt": 1
         },
-        "options": {
-            "expireAfterSeconds": 43200
-        }
+        "expireAfterSeconds": 43200
     },
     {
-        "keys": {
+        "key": {
             "_id": -1,
             "LDRedisStoreVersion": 1
         }
     }
-]
+]

indexes/codefresh/users.json

@@ -1,15 +1,15 @@
 [
     {
-        "keys": {
+        "key": {
             "account": 1,
             "_id": 1
         }
     },
     {
-        "keys": {
+        "key": {
             "role": 1,
             "account": 1,
             "_id": 1
         }
     }
-]
+]