added extractor info

Author: andreasDroid

Gemfile

@@ -5,3 +5,4 @@ gem "jekyll", "~> 4.3.4" # installed by `gem jekyll`
 
 gem "just-the-docs", "0.10.0" # pinned to the current release
 # gem "just-the-docs"        # always download the latest release
+gem "jekyll-relative-links", "0.7.0"

Gemfile.lock

@@ -39,6 +39,8 @@ GEM
       webrick (~> 1.7)
     jekyll-include-cache (0.2.1)
       jekyll (>= 3.7, < 5.0)
+    jekyll-relative-links (0.7.0)
+      jekyll (>= 3.3, < 5.0)
     jekyll-sass-converter (3.0.0)
       sass-embedded (~> 1.54)
     jekyll-seo-tag (2.8.0)
@@ -83,6 +85,7 @@ PLATFORMS
 
 DEPENDENCIES
   jekyll (~> 4.3.4)
+  jekyll-relative-links (= 0.7.0)
   just-the-docs (= 0.10.0)
 
 BUNDLED WITH

README.md

@@ -8,3 +8,6 @@ logo: /assets/fologos.png
 
 aux_links:
   FleetOptimiser OS2: https://github.com/OS2fleetoptimiser/OS2fleetoptimiser
+
+plugins:
+  - jekyll-relative-links

_config.yml

@@ -128,5 +128,5 @@ docker run -p 6379:6379 redis -d
 npm run dev
 ```
 
-See [production](production.html) for environment variables for the backend. 
+See [production](production.md) for environment variables for the backend. 
 

developing.md

@@ -0,0 +1,30 @@
+---
+title: Clevertrack
+layout: default
+parent: Extractors
+has_children: false
+---
+# Clevertrack
+
+Clevertrack uses a method of patching already read trips. 
+If one does not patch you would continuously have to load already _known_ trips, because it's not possible to query
+based off of vehicle id. Hence, you have to define a date from which you want to load trips, and manage patching.
+
+The connection requires a `TOKEN` environment which is a classic uid format. 
+In addition, you should provide a pickle path to;
+- `CAR_TRIPS_IDS` : path to a pickle that stores mapping between vehicle ids and patched trips
+- `PATCHED_IDS` : path to a pickle that stores record of patched trips
+
+
+Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor:
+
+| function         | description                          |
+|------------------|--------------------------------------|
+| set-vehicles     | syncs the vehicles                   |
+| set-roundtrips   | loads, aggregate and save roundtrips | 
+| patch-roundtrips | patches read trips on the connection |
+
+
+```
+["/bin/sh", "-c", "python -m fleetmanager.extractors.clevertrack set-vehicles && python -m fleetmanager.extractors.clevertrack set-roundtrips && python -m fleetmanager.extractors.clevertrack patch-roundtrips"]
+```

extractors.md

@@ -0,0 +1,32 @@
+---
+title: Extractors
+layout: default
+nav_order: 4
+has_children: true
+---
+
+# Extractors
+The extractors are an important part of FleetOptimiser, which asserts that data is synced from the fleetmanagement systems.
+The extractor will synchronise available metadata with vehicles, and will delta load the latest trips. 
+Trips will be aggregated from single GPS logs, to an aggregated format that will only contain coordinates of a saved start location, 
+such that the system does not save sensitive information.
+
+
+The amount of exposed metadata differ from system to system, so with some integrations there are more sanitation of data. 
+
+
+All extractors expect the following env variables:
+
+| env | description |
+|-----|-------------|
+| DB_NAME          | name of the database |
+| DB_USER          | user of the database |
+| DB_PASSWORD      | password of the user on the database |
+| DB_URL           | host and port to the database |
+| DB_SERVER        | dialect and driver |
+| TRIP_DURATION | maximum duration of a roundtrip |
+| MAX_DATE | first delta load date |
+| DESCRIPTION_FIELDS | additional fields to save from APIs
+
+
+

extractors/clevertrack.md

@@ -0,0 +1,29 @@
+---
+title: FleetComplete
+layout: default
+parent: Extractors
+has_children: false
+---
+# FleetComplete
+
+The FleetComplete extractor expects the connection key to be present in the env as a single string, like:
+
+```
+KEYS=keyToFleetComplete
+```
+
+
+Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor:
+
+| function         | description |
+|------------------| ----------- |
+| set-starts       | loads and saves new locations |
+| set-vehicles     | syncs the vehicles |
+| set-roundtrips   | loads, aggregate and save roundtrips | 
+| clean-roundtrips | deletes X (keep_data in DB) month old roundtrips |
+
+
+
+```
+entrypoint: ["/bin/sh", "-c", "python -m fleetmanager.extractors.fleetcomplete set-starts && python -m fleetmanager.extractors.fleetcomplete set-vehicles && python -m fleetmanager.extractors.fleetcomplete set-roundtrips && python -m fleetmanager.extractors.fleetcomplete clean-roundtrips"]
+```

extractors/extractors.md

@@ -0,0 +1,30 @@
+---
+title: Gamfleet
+layout: default
+parent: Extractors
+has_children: false
+---
+# Gamfleet
+
+You must modify the urls in the Gamfleet extractor code in order to point to the correct endpoints.
+
+To make the load more effective, we use a `LOAD_RECORD_PATH` to store load history of each vehicle. 
+This asserts that we don't load _old_ unusable trips. 
+
+- `LOAD_RECORD_PATH` : path to json history (optional)
+- `KEY` : key to the Gamfleet API
+
+Use the `docker-compose-extractor.yaml` file and define the functions you want to use in the extractor:
+
+| function         | description |
+|------------------| ----------- |
+| set-starts       | loads and saves new locations |
+| set-vehicles     | syncs the vehicles |
+| set-roundtrips   | loads, aggregate and save roundtrips | 
+| clean-roundtrips | deletes X (keep_data in DB) month old roundtrips |
+
+
+
+```
+entrypoint: ["/bin/sh", "-c", "python -m fleetmanager.extractors.gamfleet set-starts && python -m fleetmanager.extractors.gamfleet set-vehicles && python -m fleetmanager.extractors.gamfleet set-roundtrips && python -m fleetmanager.extractors.gamfleet clean-roundtrips"]
+```