diff --git a/.gitignore b/.gitignore index 45ed349e37..0a0a5a0a35 100644 --- a/.gitignore +++ b/.gitignore @@ -54,3 +54,6 @@ vagrant/*.log # Exclude debugging profiles profiles/ + +# Ignore local dev key +.dev_secret_key diff --git a/docker/dev/README.md b/docker/dev/README.md index 7231b53612..1fa48ac099 100644 --- a/docker/dev/README.md +++ b/docker/dev/README.md @@ -1,121 +1,100 @@ -## Docker for development +## Docker for Development You can run Timesketch on Docker in development mode. -Make sure to follow the docker [post-install](https://docs.docker.com/engine/install/linux-postinstall/) to run without superuser. If not then make sure to execute all `docker` commands here as *superuser*. +Make sure to follow the docker [post-install](https://docs.docker.com/engine/install/linux-postinstall/) +to run without superuser. If not, make sure to execute all `docker` commands here +as *superuser*. -NOTE: It is not recommended to try to run on a system with less than 8 GB of RAM. +**Note:** It is not recommended to try to run on a system with less than 8 GB of RAM. -### Start a developer version of docker containers in this directory +### 1. Start the Developer Containers ```bash +cd timesketch/docker/dev/ + docker compose up -d ``` -The provided container definition runs Timesketch in development mode as a volume from your cloned repo. Any changes you make will appear in Timesketch automatically. - -If you see the following message you can continue +The provided container definition runs Timesketch in development mode as a +volume from your cloned repo. Any changes you make will appear in Timesketch +automatically. -```text -Timesketch development server is ready! -``` -### Find out container ID for the timesketch container +Wait until you see the "Timesketch development server is ready!" message in the logs: ```bash -CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)" +docker compose logs -f timesketch ``` -In the output look for CONTAINER ID for the timesketch container +### 2. Start the Application Services -To write the ID to a variable, use: +Since the container starts in a "sleeping" state to allow for debugging, you +need to manually start the worker and the webserver. -```bash -export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)" -``` +**Option A: Interactive (Recommended for debugging)** +Open two new terminal tabs/windows and run: -and test with +*Terminal 1 (Celery Worker):* ```bash -echo $CONTAINER_ID +docker compose exec timesketch celery -A timesketch.lib.tasks worker --loglevel info ``` -### Start a celery container shell +OR ```bash -docker exec -it $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info +bash utils/tsdev.sh celery ``` -### Start development webserver (and metrics server) +*Terminal 2 (Webserver):* ```bash -docker exec -it $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 600 -c /usr/local/src/timesketch/data/gunicorn_config.py timesketch.wsgi:application +docker compose exec timesketch gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 600 -c /usr/local/src/timesketch/timesketch/gunicorn_config.py timesketch.wsgi:application ``` -You now can access your development version at http://127.0.0.1:5000/ - -Log in with user: dev password: dev - -You can also access a metrics dashboard at http://127.0.0.1:3000/ - -### Non-interactive - -Running the following as a script after `docker compose up -d` will bring up the development environment in the background for you. +OR ```bash -export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)" -docker exec $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info -docker exec $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 120 timesketch.wsgi:application +bash utils/tsdev.sh web ``` -### Run tests +**Option B: Non-interactive (Background)** +Run these commands to start everything in the background: ```bash -docker exec -w /usr/local/src/timesketch -it $CONTAINER_ID python3 run_tests.py --coverage +docker compose exec -d timesketch celery -A timesketch.lib.tasks worker --loglevel info +docker compose exec -d timesketch gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 120 timesketch.wsgi:application ``` -That will run all tests in your docker container. It is recommended to run all tests at least before creating a pull request. +**Option C: Tilt** -### Jupyter Notebook +For instructions on how to run the dev environment using Tilt check this guide: +* https://github.com/google/timesketch/blob/master/docs/developers/tilt-development.md -To access a Jupyter notebook that has access to the Timesketch development -environment start a browser and visit http://localhost:8844/ . The password to -gain access is "timesketch". +### 3. Access the Application -By default, the /tmp directory is mapped as the data directory to store all -notebooks. To change that, modify the line: +* **Timesketch UI:** http://127.0.0.1:5000/ +* **User:** `dev` +* **Password:** `dev` +* Note: Your login session will persist across container restarts. + * The "secret key" is randomly generated and stored in a `.dev_secret_key` file. -```yaml - - /tmp/:/usr/local/src/picadata/ -``` -in the docker-compose.yml file to point to a directory of your choosing. -In order for the jupyter notebook to be able to make use of that folder it has -to have read and write permission for the user with the UID 1000. +* **Metrics (Prometheus):** http://127.0.0.1:9090/ -By default, the latest checked in code of the timesketch API client and -timesketch import client are installed. In order to install a new version, if -you are modifying the clients you'll need to make sure that the timesketch -source code on your machine is readable by the user with the UID 1000 and -gid 1000. -If that is done, then the code is mapped into the `/usr/local/src/timesketch` -folder on the docker container. +### Run Tests -New versions of timesketch api client can then be installed using: +To run all tests inside the container: ```bash -!pip install -e /usr/local/src/timesketch/api_client/python/ +docker compose exec -w /usr/local/src/timesketch timesketch python3 run_tests.py --coverage ``` -And the importer client: - -```bash -!pip install -e /usr/local/src/timesketch/importer_client/python -``` +It is recommended to run all tests at least before creating a pull request. -Just remember to restart the kernel runtime in order for the changes to be -active. +### Pull new images To update the docker image run: ```bash -$ sudo docker image pull us-docker.pkg.dev/osdfir-registry/timesketch/notebook:latest +docker compose pull ``` diff --git a/docker/dev/build/docker-entrypoint.sh b/docker/dev/build/docker-entrypoint.sh index 6d690ade50..8692755d39 100755 --- a/docker/dev/build/docker-entrypoint.sh +++ b/docker/dev/build/docker-entrypoint.sh @@ -5,60 +5,74 @@ setup_config() { # Copy config files mkdir -p /etc/timesketch cp /usr/local/src/timesketch/data/timesketch.conf /etc/timesketch/ - cp /usr/local/src/timesketch/data/regex_features.yaml /etc/timesketch/ - cp /usr/local/src/timesketch/data/winevt_features.yaml /etc/timesketch/ - cp /usr/local/src/timesketch/data/tags.yaml /etc/timesketch/ - cp /usr/local/src/timesketch/data/intelligence_tag_metadata.yaml /etc/timesketch/ - cp /usr/local/src/timesketch/data/plaso.mappings /etc/timesketch/ - cp /usr/local/src/timesketch/data/generic.mappings /etc/timesketch/ - cp /usr/local/src/timesketch/data/ontology.yaml /etc/timesketch/ - cp /usr/local/src/timesketch/data/data_finder.yaml /etc/timesketch/ - cp /usr/local/src/timesketch/data/bigquery_matcher.yaml /etc/timesketch/ - + # Use -f to ignore error if link already exists - ln -sf /usr/local/src/timesketch/data/sigma_config.yaml /etc/timesketch/sigma_config.yaml ln -sf /usr/local/src/timesketch/data/sigma /etc/timesketch/ ln -sf /usr/local/src/timesketch/data/dfiq /etc/timesketch/ - ln -sf /usr/local/src/timesketch/data/context_links.yaml /etc/timesketch/context_links.yaml - ln -sf /usr/local/src/timesketch/data/plaso_formatters.yaml /etc/timesketch/plaso_formatters.yaml ln -sf /usr/local/src/timesketch/data/nl2q /etc/timesketch/ ln -sf /usr/local/src/timesketch/data/llm_summarize /etc/timesketch/ + CONFIG_FILES=( + "regex_features.yaml" + "winevt_features.yaml" + "tags.yaml" + "intelligence_tag_metadata.yaml" + "plaso.mappings" + "generic.mappings" + "ontology.yaml" + "data_finder.yaml" + "bigquery_matcher.yaml" + "plaso_formatters.yaml" + "context_links.yaml" + "sigma_config.yaml" + ) + for f in "${CONFIG_FILES[@]}"; do + ln -sf "/usr/local/src/timesketch/data/$f" "/etc/timesketch/$f" + done # Set SECRET_KEY in /etc/timesketch/timesketch.conf if it isn't already set - if grep -q "SECRET_KEY = ''" /etc/timesketch/timesketch.conf; then - OPENSSL_RAND=$( openssl rand -base64 32 ) - # Using the pound sign as a delimiter to avoid problems with / being output from openssl - sed -i 's#SECRET_KEY = \x27\x3CKEY_GOES_HERE\x3E\x27#SECRET_KEY = \x27'$OPENSSL_RAND'\x27#' /etc/timesketch/timesketch.conf + # We generate a locally stored key on the first start that gets re-used for + # later sessions. If you need a new key, just delete the ".dev_secret_key" + # file in your project root dir and restart the container. + KEY_FILE="/usr/local/src/timesketch/.dev_secret_key" + + if [ ! -f "$KEY_FILE" ]; then + echo "Generating new persistent development secret key..." + openssl rand -base64 32 > "$KEY_FILE" + fi + + SECRET_KEY=$(cat "$KEY_FILE") + if grep -q 'SECRET_KEY = ""' /etc/timesketch/timesketch.conf; then + # Use double quotes for the variable expansion to be safe + sed -i 's#SECRET_KEY = ""#SECRET_KEY = "'"$SECRET_KEY"'"#' /etc/timesketch/timesketch.conf fi # Set up the Postgres connection - if [ $POSTGRES_USER ] && [ $POSTGRES_PASSWORD ] && [ $POSTGRES_ADDRESS ] && [ $POSTGRES_PORT ]; then + if [ "$POSTGRES_USER" ] && [ "$POSTGRES_PASSWORD" ] && [ "$POSTGRES_ADDRESS" ] && [ "$POSTGRES_PORT" ]; then sed -i 's#postgresql://:@localhost#postgresql://'$POSTGRES_USER':'$POSTGRES_PASSWORD'@'$POSTGRES_ADDRESS':'$POSTGRES_PORT'#' /etc/timesketch/timesketch.conf fi # Set up the OpenSearch connection - if [ $OPENSEARCH_HOST ] && [ $OPENSEARCH_PORT ]; then - sed -i 's#OPENSEARCH_HOST = \x27127.0.0.1\x27#OPENSEARCH_HOST = \x27'$OPENSEARCH_HOST'\x27#' /etc/timesketch/timesketch.conf - sed -i 's#OPENSEARCH_PORT = 9200#OPENSEARCH_PORT = '$OPENSEARCH_PORT'#' /etc/timesketch/timesketch.conf + if [ "$OPENSEARCH_HOST" ] && [ "$OPENSEARCH_PORT" ]; then + sed -i 's#OPENSEARCH_HOSTS = \[{"host": "opensearch", "port": 9200}\]#OPENSEARCH_HOSTS = [{"host": "'$OPENSEARCH_HOST'", "port": '$OPENSEARCH_PORT'}]#' /etc/timesketch/timesketch.conf fi # Set up the Redis connection - if [ $REDIS_ADDRESS ] && [ $REDIS_PORT ]; then - sed -i 's#UPLOAD_ENABLED = False#UPLOAD_ENABLED = True#' /etc/timesketch/timesketch.conf - sed -i 's#^CELERY_BROKER_URL =.*#CELERY_BROKER_URL = \x27redis://'$REDIS_ADDRESS':'$REDIS_PORT'\x27#' /etc/timesketch/timesketch.conf - sed -i 's#^CELERY_RESULT_BACKEND =.*#CELERY_RESULT_BACKEND = \x27redis://'$REDIS_ADDRESS':'$REDIS_PORT'\x27#' /etc/timesketch/timesketch.conf + if [ "$REDIS_ADDRESS" ] && [ "$REDIS_PORT" ]; then + sed -i 's#^UPLOAD_ENABLED = False#UPLOAD_ENABLED = True#' /etc/timesketch/timesketch.conf + sed -i 's#^CELERY_BROKER_URL =.*#CELERY_BROKER_URL = "redis://'$REDIS_ADDRESS':'$REDIS_PORT'"#' /etc/timesketch/timesketch.conf + sed -i 's#^CELERY_RESULT_BACKEND =.*#CELERY_RESULT_BACKEND = "redis://'$REDIS_ADDRESS':'$REDIS_PORT'"#' /etc/timesketch/timesketch.conf fi # Enable debug for the development server - sed -i s/"DEBUG = False"/"DEBUG = True"/ /etc/timesketch/timesketch.conf + sed -i 's/DEBUG = False/DEBUG = True/' /etc/timesketch/timesketch.conf # Enable index and sketch analyzers - sed -i s/"ENABLE_INDEX_ANALYZERS = False"/"ENABLE_INDEX_ANALYZERS = True"/ /etc/timesketch/timesketch.conf - sed -i s/"ENABLE_SKETCH_ANALYZERS = False"/"ENABLE_SKETCH_ANALYZERS = True"/ /etc/timesketch/timesketch.conf - sed -i s/"ENABLE_EXPERIMENTAL_UI = False"/"ENABLE_EXPERIMENTAL_UI = True"/ /etc/timesketch/timesketch.conf + sed -i 's/ENABLE_INDEX_ANALYZERS = False/ENABLE_INDEX_ANALYZERS = True/' /etc/timesketch/timesketch.conf + sed -i 's/ENABLE_SKETCH_ANALYZERS = False/ENABLE_SKETCH_ANALYZERS = True/' /etc/timesketch/timesketch.conf + sed -i 's/ENABLE_EXPERIMENTAL_UI = False/ENABLE_EXPERIMENTAL_UI = True/' /etc/timesketch/timesketch.conf # Disable CSRF checks for the development server - echo "WTF_CSRF_ENABLED = False" >> /etc/timesketch/timesketch.conf + sed -i 's/^# WTF_CSRF_ENABLED = False.*/WTF_CSRF_ENABLED = False/' /etc/timesketch/timesketch.conf } # Run the container the default way