Fix & refactor dev docker-entrypoint.sh script

.gitignore

@@ -54,3 +54,6 @@ vagrant/*.log
 
 # Exclude debugging profiles
 profiles/
+
+# Ignore local dev key
+.dev_secret_key

docker/dev/README.md

@@ -1,121 +1,100 @@
-## Docker for development
+## Docker for Development
 
 You can run Timesketch on Docker in development mode.
-Make sure to follow the docker [post-install](https://docs.docker.com/engine/install/linux-postinstall/) to run without superuser. If not then make sure to execute all `docker` commands here as *superuser*.
+Make sure to follow the docker [post-install](https://docs.docker.com/engine/install/linux-postinstall/)
+to run without superuser. If not, make sure to execute all `docker` commands here
+as *superuser*.
 
-NOTE: It is not recommended to try to run on a system with less than 8 GB of RAM.
+**Note:** It is not recommended to try to run on a system with less than 8 GB of RAM.
 
-### Start a developer version of docker containers in this directory
+### 1. Start the Developer Containers
 
 ```bash
+cd timesketch/docker/dev/
+
 docker compose up -d
 ```
 
-The provided container definition runs Timesketch in development mode as a volume from your cloned repo. Any changes you make will appear in Timesketch automatically.
-
-If you see the following message you can continue
+The provided container definition runs Timesketch in development mode as a
+volume from your cloned repo. Any changes you make will appear in Timesketch
+automatically.
 
-```text
-Timesketch development server is ready!
-```
-### Find out container ID for the timesketch container
+Wait until you see the "Timesketch development server is ready!" message in the logs:
 
 ```bash
-CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
+docker compose logs -f timesketch
 ```
 
-In the output look for CONTAINER ID for the timesketch container
+### 2. Start the Application Services
 
-To write the ID to a variable, use:
+Since the container starts in a "sleeping" state to allow for debugging, you
+need to manually start the worker and the webserver.
 
-```bash
-export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-```
+**Option A: Interactive (Recommended for debugging)**
+Open two new terminal tabs/windows and run:
 
-and test with
+*Terminal 1 (Celery Worker):*
 
 ```bash
-echo $CONTAINER_ID
+docker compose exec timesketch celery -A timesketch.lib.tasks worker --loglevel info
 ```
 
-### Start a celery container shell
+OR
 
 ```bash
-docker exec -it $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info
+bash utils/tsdev.sh celery
 ```
 
-### Start development webserver (and metrics server)
+*Terminal 2 (Webserver):*
 
 ```bash
-docker exec -it $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 600 -c /usr/local/src/timesketch/data/gunicorn_config.py timesketch.wsgi:application
+docker compose exec timesketch gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 600 -c /usr/local/src/timesketch/timesketch/gunicorn_config.py timesketch.wsgi:application
 ```
 
-You now can access your development version at http://127.0.0.1:5000/
-
-Log in with user: dev password: dev
-
-You can also access a metrics dashboard at http://127.0.0.1:3000/
-
-### Non-interactive
-
-Running the following as a script after `docker compose up -d` will bring up the development environment in the background for you.
+OR
 
 ```bash
-export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-docker exec $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info
-docker exec $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 120 timesketch.wsgi:application
+bash utils/tsdev.sh web
 ```
 
-### Run tests
+**Option B: Non-interactive (Background)**
+Run these commands to start everything in the background:
 
 ```bash
-docker exec -w /usr/local/src/timesketch -it $CONTAINER_ID python3 run_tests.py --coverage
+docker compose exec -d timesketch celery -A timesketch.lib.tasks worker --loglevel info
+docker compose exec -d timesketch gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 120 timesketch.wsgi:application
 ```
 
-That will run all tests in your docker container. It is recommended to run all tests at least before creating a pull request.
+**Option C: Tilt**
 
-### Jupyter Notebook
+For instructions on how to run the dev environment using Tilt check this guide:
+* https://github.com/google/timesketch/blob/master/docs/developers/tilt-development.md
 
-To access a Jupyter notebook that has access to the Timesketch development
-environment start a browser and visit http://localhost:8844/ . The password to
-gain access is "timesketch".
+### 3. Access the Application
 
-By default, the /tmp directory is mapped as the data directory to store all
-notebooks. To change that, modify the line:
+* **Timesketch UI:** http://127.0.0.1:5000/
+* **User:** `dev`
+* **Password:** `dev`
+* Note: Your login session will persist across container restarts.
+  * The "secret key" is randomly generated and stored in a `.dev_secret_key` file.
 
-```yaml
-      - /tmp/:/usr/local/src/picadata/
-```
 
-in the docker-compose.yml file to point to a directory of your choosing.
-In order for the jupyter notebook to be able to make use of that folder it has
-to have read and write permission for the user with the UID 1000.
+* **Metrics (Prometheus):** http://127.0.0.1:9090/
 
-By default, the latest checked in code of the timesketch API client and
-timesketch import client are installed. In order to install a new version, if
-you are modifying the clients you'll need to make sure that the timesketch
-source code on your machine is readable by the user with the UID 1000 and
-gid 1000.
-If that is done, then the code is mapped into the `/usr/local/src/timesketch`
-folder on the docker container.
+### Run Tests
 
-New versions of timesketch api client can then be installed using:
+To run all tests inside the container:
 
 ```bash
-!pip install -e /usr/local/src/timesketch/api_client/python/
+docker compose exec -w /usr/local/src/timesketch timesketch python3 run_tests.py --coverage
 ```
 
-And the importer client:
-
-```bash
-!pip install -e /usr/local/src/timesketch/importer_client/python
-```
+It is recommended to run all tests at least before creating a pull request.
 
-Just remember to restart the kernel runtime in order for the changes to be
-active.
+### Pull new images
 
 To update the docker image run:
 
 ```bash
-$ sudo docker image pull us-docker.pkg.dev/osdfir-registry/timesketch/notebook:latest
+docker compose pull
 ```

docker/dev/build/docker-entrypoint.sh

@@ -5,60 +5,74 @@ setup_config() {
   # Copy config files
   mkdir -p /etc/timesketch
   cp /usr/local/src/timesketch/data/timesketch.conf /etc/timesketch/
-  cp /usr/local/src/timesketch/data/regex_features.yaml /etc/timesketch/
-  cp /usr/local/src/timesketch/data/winevt_features.yaml /etc/timesketch/
-  cp /usr/local/src/timesketch/data/tags.yaml /etc/timesketch/
-  cp /usr/local/src/timesketch/data/intelligence_tag_metadata.yaml /etc/timesketch/
-  cp /usr/local/src/timesketch/data/plaso.mappings /etc/timesketch/
-  cp /usr/local/src/timesketch/data/generic.mappings /etc/timesketch/
-  cp /usr/local/src/timesketch/data/ontology.yaml /etc/timesketch/
-  cp /usr/local/src/timesketch/data/data_finder.yaml /etc/timesketch/
-  cp /usr/local/src/timesketch/data/bigquery_matcher.yaml /etc/timesketch/
-
+
   # Use -f to ignore error if link already exists
-  ln -sf /usr/local/src/timesketch/data/sigma_config.yaml /etc/timesketch/sigma_config.yaml
   ln -sf /usr/local/src/timesketch/data/sigma /etc/timesketch/
   ln -sf /usr/local/src/timesketch/data/dfiq /etc/timesketch/
-  ln -sf /usr/local/src/timesketch/data/context_links.yaml /etc/timesketch/context_links.yaml
-  ln -sf /usr/local/src/timesketch/data/plaso_formatters.yaml /etc/timesketch/plaso_formatters.yaml
   ln -sf /usr/local/src/timesketch/data/nl2q /etc/timesketch/
   ln -sf /usr/local/src/timesketch/data/llm_summarize /etc/timesketch/
+  CONFIG_FILES=(
+    "regex_features.yaml"
+    "winevt_features.yaml"
+    "tags.yaml"
+    "intelligence_tag_metadata.yaml"
+    "plaso.mappings"
+    "generic.mappings"
+    "ontology.yaml"
+    "data_finder.yaml"
+    "bigquery_matcher.yaml"
+    "plaso_formatters.yaml"
+    "context_links.yaml"
+    "sigma_config.yaml"
+  )
+  for f in "${CONFIG_FILES[@]}"; do
+    ln -sf "/usr/local/src/timesketch/data/$f" "/etc/timesketch/$f"
+  done
 
   # Set SECRET_KEY in /etc/timesketch/timesketch.conf if it isn't already set
-  if grep -q "SECRET_KEY = '<KEY_GOES_HERE>'" /etc/timesketch/timesketch.conf; then
-    OPENSSL_RAND=$( openssl rand -base64 32 )
-    # Using the pound sign as a delimiter to avoid problems with / being output from openssl
-    sed -i 's#SECRET_KEY = \x27\x3CKEY_GOES_HERE\x3E\x27#SECRET_KEY = \x27'$OPENSSL_RAND'\x27#' /etc/timesketch/timesketch.conf
+  # We generate a locally stored key on the first start that gets re-used for
+  # later sessions. If you need a new key, just delete the ".dev_secret_key"
+  # file in your project root dir and restart the container.
+  KEY_FILE="/usr/local/src/timesketch/.dev_secret_key"
+
+  if [ ! -f "$KEY_FILE" ]; then
+    echo "Generating new persistent development secret key..."
+    openssl rand -base64 32 > "$KEY_FILE"
+  fi
+
+  SECRET_KEY=$(cat "$KEY_FILE")
+  if grep -q 'SECRET_KEY = "<KEY_GOES_HERE>"' /etc/timesketch/timesketch.conf; then
+    # Use double quotes for the variable expansion to be safe
+    sed -i 's#SECRET_KEY = "<KEY_GOES_HERE>"#SECRET_KEY = "'"$SECRET_KEY"'"#' /etc/timesketch/timesketch.conf
   fi
 
   # Set up the Postgres connection
-  if [ $POSTGRES_USER ] && [ $POSTGRES_PASSWORD ] && [ $POSTGRES_ADDRESS ] && [ $POSTGRES_PORT ]; then
+  if [ "$POSTGRES_USER" ] && [ "$POSTGRES_PASSWORD" ] && [ "$POSTGRES_ADDRESS" ] && [ "$POSTGRES_PORT" ]; then
     sed -i 's#postgresql://<USERNAME>:<PASSWORD>@localhost#postgresql://'$POSTGRES_USER':'$POSTGRES_PASSWORD'@'$POSTGRES_ADDRESS':'$POSTGRES_PORT'#' /etc/timesketch/timesketch.conf
   fi
 
   # Set up the OpenSearch connection
-  if [ $OPENSEARCH_HOST ] && [ $OPENSEARCH_PORT ]; then
-    sed -i 's#OPENSEARCH_HOST = \x27127.0.0.1\x27#OPENSEARCH_HOST = \x27'$OPENSEARCH_HOST'\x27#' /etc/timesketch/timesketch.conf
-    sed -i 's#OPENSEARCH_PORT = 9200#OPENSEARCH_PORT = '$OPENSEARCH_PORT'#' /etc/timesketch/timesketch.conf
+  if [ "$OPENSEARCH_HOST" ] && [ "$OPENSEARCH_PORT" ]; then
+    sed -i 's#OPENSEARCH_HOSTS = \[{"host": "opensearch", "port": 9200}\]#OPENSEARCH_HOSTS = [{"host": "'$OPENSEARCH_HOST'", "port": '$OPENSEARCH_PORT'}]#' /etc/timesketch/timesketch.conf
   fi
 
   # Set up the Redis connection
-  if [ $REDIS_ADDRESS ] && [ $REDIS_PORT ]; then
-    sed -i 's#UPLOAD_ENABLED = False#UPLOAD_ENABLED = True#' /etc/timesketch/timesketch.conf
-    sed -i 's#^CELERY_BROKER_URL =.*#CELERY_BROKER_URL = \x27redis://'$REDIS_ADDRESS':'$REDIS_PORT'\x27#' /etc/timesketch/timesketch.conf
-    sed -i 's#^CELERY_RESULT_BACKEND =.*#CELERY_RESULT_BACKEND = \x27redis://'$REDIS_ADDRESS':'$REDIS_PORT'\x27#' /etc/timesketch/timesketch.conf
+  if [ "$REDIS_ADDRESS" ] && [ "$REDIS_PORT" ]; then
+    sed -i 's#^UPLOAD_ENABLED = False#UPLOAD_ENABLED = True#' /etc/timesketch/timesketch.conf
+    sed -i 's#^CELERY_BROKER_URL =.*#CELERY_BROKER_URL = "redis://'$REDIS_ADDRESS':'$REDIS_PORT'"#' /etc/timesketch/timesketch.conf
+    sed -i 's#^CELERY_RESULT_BACKEND =.*#CELERY_RESULT_BACKEND = "redis://'$REDIS_ADDRESS':'$REDIS_PORT'"#' /etc/timesketch/timesketch.conf
   fi
 
   # Enable debug for the development server
-  sed -i s/"DEBUG = False"/"DEBUG = True"/ /etc/timesketch/timesketch.conf
+  sed -i 's/DEBUG = False/DEBUG = True/' /etc/timesketch/timesketch.conf
 
   # Enable index and sketch analyzers
-  sed -i s/"ENABLE_INDEX_ANALYZERS = False"/"ENABLE_INDEX_ANALYZERS = True"/ /etc/timesketch/timesketch.conf
-  sed -i s/"ENABLE_SKETCH_ANALYZERS = False"/"ENABLE_SKETCH_ANALYZERS = True"/ /etc/timesketch/timesketch.conf
-  sed -i s/"ENABLE_EXPERIMENTAL_UI = False"/"ENABLE_EXPERIMENTAL_UI = True"/ /etc/timesketch/timesketch.conf
+  sed -i 's/ENABLE_INDEX_ANALYZERS = False/ENABLE_INDEX_ANALYZERS = True/' /etc/timesketch/timesketch.conf
+  sed -i 's/ENABLE_SKETCH_ANALYZERS = False/ENABLE_SKETCH_ANALYZERS = True/' /etc/timesketch/timesketch.conf
+  sed -i 's/ENABLE_EXPERIMENTAL_UI = False/ENABLE_EXPERIMENTAL_UI = True/' /etc/timesketch/timesketch.conf
 
   # Disable CSRF checks for the development server
-  echo "WTF_CSRF_ENABLED = False" >> /etc/timesketch/timesketch.conf
+  sed -i 's/^# WTF_CSRF_ENABLED = False.*/WTF_CSRF_ENABLED = False/' /etc/timesketch/timesketch.conf
 }
 
 # Run the container the default way