datanav · djalil-bouhachi · May 2, 2025 · May 2, 2025 · May 2, 2025 · May 2, 2025
@@ -5,6 +5,6 @@ Environment Variables
 
 An :ref:`environment variable <environment_variables>` is a named value that you can reference in your configuration. Environment variables are used to parameterize your configuration so that you can easily enable/disable or change certain aspects of your configuration. If you have an environment variable called ``myvariable`` then you can reference it in configuration like this: ``"$ENV(myvariable)"``.
 
-.. NOTE::
+.. important::
 
     Do not use environment variables for sensitive values; use :ref:`secrets <concepts-secrets>` instead. Environment variables are global only.
@@ -344,9 +344,10 @@ Properties
        be redacted in the ``pump-completed``/``pump-completed`` events in the execution-log. The
        purpose of this setting is to redact enough of the secrets to render them safe to log, but to
        potentially leave some of the secret for debugging purposes.
-       A value of ``-1`` means to redact all bytes of the secrets. Note that the redaction is only a best-effort
-       attempt to prevent secrets from ending up in the logs, there may be cases where secrets leak through in any
-       case, so it is best to always check that what ends up being logged looks ok.
+       A value of ``-1`` means to redact all bytes of the secrets. 
+
+       .. note::
+        The redaction is only a best-effort attempt to prevent secrets from ending up in the logs, there may be cases where secrets leak through in any case, so it is best to always check that what ends up being logged looks ok.
      - 600
      - No
 

@@ -64,7 +64,7 @@ Prototype
 
 Variant 1: Explicit equality-rules with the ``equality`` property
 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
-::
+.. code-block :: json
 
     {
         "type": "merge",
@@ -80,7 +80,7 @@ Variant 1: Explicit equality-rules with the ``equality`` property
 
 Variant 2: Implicit equality-rules with the ``equality_sets`` property
 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
-::
+.. code-block :: json
 
     {
         "type": "merge",
@@ -266,46 +266,56 @@ Properties
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Equality is resolved transitively, so if A is the same as B and B is the same as C then A,
 B and C are all considered the same. With the ``equality`` property, these rules must be specified
-one at a time, like this::
+one at a time, like this:
 
-        "equality": [
-            ["eq", "a.x", "b.x"],
-            ["eq", "b.x", "c.y"],
-            ["eq", "c.z", "d.z"],
-        ],
+.. code-block :: json
+
+    "equality": [
+        ["eq", "a.x", "b.x"],
+        ["eq", "b.x", "c.y"],
+        ["eq", "c.z", "d.z"],
+    ],
 
 The ``equality_sets`` property was added as a way to makes it clearer which equality-rules belong together.
-The equality-rules above could be expressed like this::
+The equality-rules above could be expressed like this:
 
-        "equality_sets": [
-            ["a.x", "b.x", "c.y"],
-            ["c.z", "d.z"],
-        ],
+.. code-block :: json
+
+    "equality_sets": [
+        ["a.x", "b.x", "c.y"],
+        ["c.z", "d.z"],
+    ],
 
 Note that the ``equality_sets`` property is just a bit of syntactic sugar; behind the scenes the implicit
 equality-rules are added to the rules in the ``equality`` property. This means that you can use both the
 ``equality_sets`` and the ``equality`` property at the same time if you want (although this is not recommended, since
 it makes it harder to figure out the equality-rules). It also means that you will not get a configuration warning if
-if you accidentally specify two equality-sets that are actually overlapping. If you for example specify this::
+if you accidentally specify two equality-sets that are actually overlapping. If you for example specify this:
 
-        "equality_sets": [
-            ["a.x", "b.x", "c.y"],
-            ["c.y", "d.y"],
-        ],
+.. code-block :: json
 
-you won't actually get two equality-sets, since behind the scenes you end up with these equality-rules::
+    "equality_sets": [
+        ["a.x", "b.x", "c.y"],
+        ["c.y", "d.y"],
+    ],
 
-        "equality": [
-            ["eq", "a.x", "b.x"],
-            ["eq", "b.x", "c.y"],
-            ["eq", "c.y", "d.y"]
-        ],
+you won't actually get two equality-sets, since behind the scenes you end up with these equality-rules:
 
-, which is equivalent to specifying a single equality-set, like this::
+.. code-block :: json
 
-        "equality_sets": [
-            ["a.x", "b.x", "c.y", "d.y"],
-        ],
+    "equality": [
+        ["eq", "a.x", "b.x"],
+        ["eq", "b.x", "c.y"],
+        ["eq", "c.y", "d.y"]
+    ],
+
+which is equivalent to specifying a single equality-set, like this:
+
+.. code-block :: json
+
+    "equality_sets": [
+        ["a.x", "b.x", "c.y", "d.y"],
+    ],
 
 Continuation support
 ^^^^^^^^^^^^^^^^^^^^
@@ -336,7 +346,7 @@ that uses the ``merge`` source.
 
 Dataset ``A``:
 
-::
+.. code-block :: json
 
    [
        {"_id": "a1", "f1": 1},
@@ -345,7 +355,7 @@ Dataset ``A``:
 
 Dataset ``B``:
 
-::
+.. code-block :: json
 
    [
        {"_id": "b1", "f1": 1, "f2": "x"},
@@ -354,7 +364,7 @@ Dataset ``B``:
 
 Dataset ``C``:
 
-::
+.. code-block :: json
 
    [
        {"_id": "c1", "f3": "X"},
@@ -365,7 +375,7 @@ Dataset ``C``:
 
 Pipe configuration:
 
-::
+.. code-block :: json
 
    {
        "_id": "result",
@@ -381,7 +391,7 @@ Pipe configuration:
 
 Given the above we should expect an output that looks like this:
 
-::
+.. code-block :: json
 
    [
        {"$ids": ["a1", "b1", "c1"], "_id": "0|a1|1|b1|2|c1", "_updated": 0,

@@ -16,13 +16,13 @@ Prototype
 
 The following *JSON* snippet shows the general form of a source definition.
 
-..code-block::json
+.. code-block:: json
 
-    {
-        "type": "a-source-type",
-        "comment": "This is a comment"
+  {
+    "type": "a-source-type",
+    "comment": "This is a comment"
 
-    }
+  }
 
 The only universally required property is ``type``.
 
@@ -64,9 +64,9 @@ the Sesam components and it is assumed to be interpretable *only by
 the source*. Within an entity the marker is carried in the
 ``_updated`` property if supported by its source.
 
-    .. important::
+.. important::
 
-        When using continuation support, Sesam will not be able to do automatic deletion tracking. If you wish to include deleted entities in your import, make sure you regularly set a full sync on the imported data.
+  When using continuation support, Sesam will not be able to do automatic deletion tracking. If you wish to include deleted entities in your import, make sure you regularly set a full sync on the imported data.
 
 
 
@@ -201,9 +201,9 @@ The table below shows which strategy is chosen depending on the value of the pro
    :header-rows: 1
    :widths: 25, 25, 25, 25
 
-   * - ``supports_since``
-     - ``is_since_comparable``
-     - ``is_chronological``
+   * - supports_since
+     - is_since_comparable
+     - is_chronological
      - Strategy
 
    * - ``false``
@@ -298,6 +298,8 @@ If you wish to activate continuation support for a :ref:`microservice <microserv
 
    </details>
 
+|
+
 The microservice needs to pass on an entity property named "_updated" to Sesam for each entity from the source. This property should take the value corresponding to the time-stamp or sequence value of the source data representing the last data update for that entity (the same column as for the "updated_column" for SQL type sources). When the entities have been passed on into Sesam, the inbound pipe will go through all these "_updated" values and pick the max value as the new "pipe_offset".
 
 The first time the inbound pipe runs (or if the pipe is reset), the "pipe_offset" will not have a value, resulting in a complete import of all the data from the endpoint. Once data has been imported, the new "pipe_offset" will get passed to the microservice as the query parameter "since". This parameter can in turn be used as a query parameter to the API ensuring that only data updated after the last "since" value will be included in the GET request. An example of this is shown in the Python code snippet below.
@@ -335,6 +337,8 @@ The first time the inbound pipe runs (or if the pipe is reset), the "pipe_offset
 
    </details>
 
+|
+
 In this case the data from the source is not ordered chronologically, which means we can not use the "is_chronological" tag. The benefit of chronologically ordered data in the source system is that if the pipe's pump for some reason should fail in the middle of a request, Sesam can use the chronological order of the source data to continue requesting data from the last received entity. If the data is not ordered, Sesam has to re-run the whole last request.
 
 Type of sources

@@ -10,7 +10,7 @@ used in conjunction with the :ref:`Elasticsearch sink <elasticsearch_sink>`.
 Prototype
 ^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "id-of-system",
@@ -62,7 +62,7 @@ Properties
 Example configuration
 ^^^^^^^^^^^^^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "our-elasticsearch-server",
@@ -75,7 +75,7 @@ Example configuration
 Example configuration with authentication
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "our-elasticsearch-server",

@@ -14,7 +14,7 @@ It supports the following properties:
 Prototype
 ^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "host": "FQDN of LDAP host",
@@ -108,7 +108,7 @@ Properties
 Example configuration
 ^^^^^^^^^^^^^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "example_ldap",

@@ -261,7 +261,7 @@ service:
 Example configuration
 ^^^^^^^^^^^^^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "our-http-server",

@@ -15,7 +15,7 @@ are supported and how they are mapped to :ref:`Sesam types <entity_data_types>`.
 Prototype
 ^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "sql_system_id",
@@ -97,7 +97,7 @@ Example configuration
 
 Example MS SQL Server configuration:
 
-::
+.. code-block :: json
 
     {
         "_id": "sqlserver_db",

@@ -17,7 +17,7 @@ are supported and how they are mapped to :ref:`Sesam types <entity_data_types>`.
 Prototype
 ^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "sql_system_id",
@@ -85,7 +85,7 @@ Example configuration
 
 Example MS SQL Server configuration:
 
-::
+.. code-block :: json
 
     {
         "_id": "sqlserver_db",

@@ -11,7 +11,7 @@ how they are mapped to :ref:`Sesam types <entity_data_types>`.
 Prototype
 ^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "sql_system_id",
@@ -72,7 +72,7 @@ Example configuration
 
 Example MySQL configuration:
 
-::
+.. code-block :: json
 
     {
         "_id": "sqlserver_db",

@@ -11,7 +11,7 @@ and how they are mapped to :ref:`Sesam types <entity_data_types>`.
 Prototype
 ^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "sql_system_id",
@@ -69,15 +69,15 @@ Example configuration
 
 Example Oracle TNS configuration:
 
-::
+.. code-block :: json
 
     {
         "_id": "oracle_db",
         "name": "Oracle test database",
         "type": "system:oracle_tns",
         "username": "$ENV(username-variable)",
         "password": "$SECRET(password-variable)",
-        "tns_name": "(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = foo)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = BAR)))"",
+        "tns_name": "(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = foo)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = BAR)))",
         "coerce_to_decimal": true
     }
 
@@ -10,7 +10,7 @@ and how they are mapped to :ref:`Sesam types <entity_data_types>`.
 Prototype
 ^^^^^^^^^
 
-::
+.. code-block :: json
 
     {
         "_id": "sql_system_id",
@@ -81,7 +81,7 @@ Example configuration
 
 Example Oracle configuration:
 
-::
+.. code-block :: json
 
     {
         "_id": "oracle_db",