Index to search #2

galmuhammadi · 2025-11-02T09:50:22Z

Purpose

Describe the purpose of this pull request.

Proposal

item 1...
item 2...

External contributions

Thank you for your contribution! 🎉

Please ensure the following items are checked before submitting your pull request:

I have read and followed the contributing guidelines
I have read and agreed to the Code of Conduct
I have signed off my commits with git commit --signoff (DCO compliance)
I have signed my commits with my SSH or GPG key (git commit -S)
My commit messages follow the required format: <gitmoji>(type) title description
I have added a changelog entry under ## [Unreleased] section (if noticeable change)
I have added corresponding tests for new features or bug fixes (if applicable)

Search in Docs relies on an external project like "La Suite Find". We need to declare a common external network in order to connect to the search app and index our documents.

We need to content in our demo documents so that we can test indexing.

Add indexer that loops across documents in the database, formats them as json objects and indexes them in the remote "Find" mico-service.

On document content or permission changes, start a celery job that will call the indexation API of the app "Find". Signed-off-by: Fabre Florian <ffabre@hybird.org>

Signed-off-by: Fabre Florian <ffabre@hybird.org>

New API view that calls the indexed documents search view (resource server) of app "Find". Signed-off-by: Fabre Florian <ffabre@hybird.org>

New SEARCH_INDEXER_CLASS setting to define the indexer service class. Raise ImpoperlyConfigured errors instead of RuntimeError in index service. Signed-off-by: Fabre Florian <ffabre@hybird.org>

Signed-off-by: Fabre Florian <ffabre@hybird.org>

Filter deleted documents from visited ones. Set default ordering to the Find API search call (-updated_at) BaseDocumentIndexer.search now returns a list of document ids instead of models. Do not call the indexer in signals when SEARCH_INDEXER_CLASS is not defined or properly configured. Signed-off-by: Fabre Florian <ffabre@hybird.org>

Only documents without title and content are ignored by indexer.

Add SEARCH_INDEXER_COUNTDOWN as configurable setting. Make the search backend creation simplier (only 'get_document_indexer' now). Allow indexation of deleted documents. Signed-off-by: Fabre Florian <ffabre@hybird.org>

Generate a fernet key for the OIDC_STORE_REFRESH_TOKEN_KEY in development settings if not set. Signed-off-by: Fabre Florian <ffabre@hybird.org>

Add nginx with 'nginx' alias to the 'lasuite-net' network (keycloak calls) Add celery-dev to the 'lasuite-net' network (Find API calls in jobs) Set app-dev alias as 'impress' in the 'lasuite-net' network Add indexer configuration in common settings Signed-off-by: Fabre Florian <ffabre@hybird.org>

Set a valid Ferney for OIDC_STORE_REFRESH_TOKEN_KEY in env.d/development/common Add .gitguardian.yaml configuration to ignore this key. Signed-off-by: Fabre Florian <ffabre@hybird.org>

Rename FindDocumentIndexer as SearchIndexer Rename FindDocumentSerializer as SearchDocumentSerializer Rename package core.tasks.find as core.task.search Remove logs on http errors in SearchIndexer Factorise some code in search API view. Signed-off-by: Fabre Florian <ffabre@hybird.org>

Replace indexer_debounce_lock|release functions by indexer_throttle_acquire() Instead of mutex-like mechanism, simply set a flag in cache for an amount of time that block any other task creation. Signed-off-by: Fabre Florian <ffabre@hybird.org>

Keep ordering by score from Find API on search/ results and fallback search still uses "-update_at" ordering as default Refactor pagination to work with a list instead of a queryset Signed-off-by: Fabre Florian <ffabre@hybird.org>

galmuhammadi · 2025-11-04T11:10:04Z

.gitguardian.yaml

@@ -0,0 +1,5 @@
+secret:
+  ignored_matches:


The ignored value contains a 44‐character key; storing static secrets in configuration is risky and may be better done via environment variables.

galmuhammadi · 2025-11-04T11:10:04Z

.gitignore

 env.d/terraform

+# Docker
+compose.override.yml


Adding "docker/auth/*.local" could inadvertently ignore local authentication files for all Docker builds; confirm this is intended.

galmuhammadi · 2025-11-04T11:10:05Z

CHANGELOG.md

 - 🍱(frontend) Fonts GDPR compliants #1453
 - ♻️(service-worker) improve SW registration and update handling #1473
+- ✨(backend) add async indexation of documents on save (or access save) #1276
+- ✨(backend) add debounce mechanism to limit indexation jobs #1276


The same ticket number suitenumerique#1276 appears twice in the added feature list; duplicates can confuse readers and should be consolidated.

galmuhammadi · 2025-11-04T11:10:05Z

Makefile

 .PHONY: demo

+index: ## index all documents to remote search
+	@$(MANAGE) index


The new index target references $(MANAGE) index but no phony dependency is added for 'demo', potentially causing unintended dependency ordering.

galmuhammadi · 2025-11-04T11:10:06Z

bin/fernetkey

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+


The generated key is output with an extra newline; while harmless, it may cause downstream parsing errors if the consumer expects a plain key.

galmuhammadi · 2025-11-04T11:10:07Z

compose.yml

    ports:
      - "8071:8000"
+    networks:
+      default: {}


Network alias "impress" defined for the backend service may conflict with existing aliases; verify that no other service uses the same alias to avoid name clashes.

galmuhammadi · 2025-11-04T11:10:07Z

env.d/development/common

 OIDC_AUTH_REQUEST_EXTRA_PARAMS={"acr_values": "eidas1"}

+# Store OIDC tokens in the session
+OIDC_STORE_ACCESS_TOKEN = True


The placeholder value for OIDC_STORE_REFRESH_TOKEN_KEY looks like a toy 44‐character string; replace it with a real Fernet key before deployment.

galmuhammadi · 2025-11-04T11:10:08Z

src/backend/core/api/serializers.py

        default=enums.MoveNodePositionChoices.LAST_CHILD,
    )
+
+


The string literal for the page field ends with an unmatched single quote, causing a syntax error that will prevent the module from loading.

galmuhammadi · 2025-11-04T11:10:09Z

src/backend/core/api/viewsets.py

 from django.core.exceptions import ValidationError
 from django.core.files.storage import default_storage
+from django.core.paginator import InvalidPage, Paginator
 from django.core.validators import URLValidator


search_serializer_class is set to ListDocumentSerializer, but the new SearchDocumentSerializer was added; this mismatch will make search responses use the wrong serializer.

galmuhammadi · 2025-11-04T11:12:47Z

.gitguardian.yaml

@@ -0,0 +1,5 @@
+secret:
+  ignored_matches:


The ignored value contains a 44‐character key; storing static secrets in configuration is risky and may be better done via environment variables.

galmuhammadi · 2025-11-04T11:12:47Z

.gitignore

 env.d/terraform

+# Docker
+compose.override.yml


Adding "docker/auth/*.local" could inadvertently ignore local authentication files for all Docker builds; confirm this is intended.

galmuhammadi · 2025-11-04T11:12:48Z

CHANGELOG.md

 - 🍱(frontend) Fonts GDPR compliants #1453
 - ♻️(service-worker) improve SW registration and update handling #1473
+- ✨(backend) add async indexation of documents on save (or access save) #1276
+- ✨(backend) add debounce mechanism to limit indexation jobs #1276


The same ticket number suitenumerique#1276 appears twice in the added feature list; duplicates can confuse readers and should be consolidated.

galmuhammadi · 2025-11-04T11:12:48Z

Makefile

 .PHONY: demo

+index: ## index all documents to remote search
+	@$(MANAGE) index


The new index target references $(MANAGE) index but no phony dependency is added for 'demo', potentially causing unintended dependency ordering.

galmuhammadi · 2025-11-04T11:12:49Z

bin/fernetkey

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+


The generated key is output with an extra newline; while harmless, it may cause downstream parsing errors if the consumer expects a plain key.

galmuhammadi · 2025-11-04T11:12:49Z

compose.yml

    ports:
      - "8071:8000"
+    networks:
+      default: {}


Network alias "impress" defined for the backend service may conflict with existing aliases; verify that no other service uses the same alias to avoid name clashes.

galmuhammadi · 2025-11-04T11:12:50Z

env.d/development/common

 OIDC_AUTH_REQUEST_EXTRA_PARAMS={"acr_values": "eidas1"}

+# Store OIDC tokens in the session
+OIDC_STORE_ACCESS_TOKEN = True


The placeholder value for OIDC_STORE_REFRESH_TOKEN_KEY looks like a toy 44‐character string; replace it with a real Fernet key before deployment.

galmuhammadi · 2025-11-04T11:12:50Z

src/backend/core/api/serializers.py

        default=enums.MoveNodePositionChoices.LAST_CHILD,
    )
+
+


The string literal for the page field ends with an unmatched single quote, causing a syntax error that will prevent the module from loading.

galmuhammadi · 2025-11-04T11:12:51Z

src/backend/core/api/viewsets.py

 from django.core.exceptions import ValidationError
 from django.core.files.storage import default_storage
+from django.core.paginator import InvalidPage, Paginator
 from django.core.validators import URLValidator


search_serializer_class is set to ListDocumentSerializer, but the new SearchDocumentSerializer was added; this mismatch will make search responses use the wrong serializer.

galmuhammadi · 2025-11-04T11:12:52Z

src/backend/core/models.py

-            file_key = self.file_key
-            bytes_content = self._content.encode("utf-8")
+            self.save_content(self._content)



The save method is not properly handling the case where the document's content is empty.

galmuhammadi · 2025-11-04T11:12:52Z

src/backend/core/services/search_indexers.py

@@ -0,0 +1,295 @@
+"""Document search index management utilities and indexers"""
+


The get_document_indexer function is not properly handling the case where the indexer class is not configured.

galmuhammadi · 2025-11-04T11:12:53Z

src/backend/core/tests/commands/test_index.py

@@ -0,0 +1,65 @@
+"""
+Unit test for `index` command.


The test_index command is not properly handling the case where the indexer is not configured.

galmuhammadi · 2025-11-04T11:12:54Z

src/backend/core/tests/documents/test_api_documents_search.py

@@ -0,0 +1,429 @@
+"""
+Tests for Documents API endpoint in impress's core app: list


The test_api_documents_search endpoint is not properly handling the case where the indexer is not configured.

galmuhammadi · 2025-11-04T11:12:54Z

src/backend/core/tests/test_models_documents.py

 import smtplib
 from logging import Logger
+from operator import itemgetter
 from unittest import mock


The test_models_documents test case is not properly handling the case where the document's content is empty.

galmuhammadi · 2025-11-04T11:12:55Z

src/backend/core/tests/test_services_search_indexers.py

@@ -0,0 +1,540 @@
+"""Tests for Documents search indexers"""
+


The test_services_search_indexers test case is not properly handling the case where the indexer is not configured.

galmuhammadi · 2025-11-04T11:12:55Z

src/backend/core/utils.py

 import base64
 import re
+from collections import defaultdict



The get_ancestor_to_descendants_map function is not properly handling the case where the path is empty.

galmuhammadi · 2025-11-04T11:12:56Z

src/backend/demo/management/commands/create_demo.py

 """create_demo management command"""

+import base64
 import logging


The create_demo command is not properly handling the case where the demo data is not properly formatted.

galmuhammadi · 2025-11-04T11:12:57Z

src/backend/impress/settings.py

    DEFAULT_AUTO_FIELD = "django.db.models.AutoField"

+    # Search
+    SEARCH_INDEXER_CLASS = values.Value(


The SEARCH_INDEXER_CLASS setting is not properly configured.

galmuhammadi · 2025-11-04T11:12:57Z

src/frontend/apps/impress/src/features/docs/doc-management/api/useDocs.tsx

  const searchParams = constructParams(params);
-  const response = await fetchAPI(`documents/?${searchParams.toString()}`);
+  let response
+


The getDocs function is not properly handling the case where the search query is empty.

sampaccoud and others added 19 commits October 31, 2025 11:31

🔧(compose) configure external network for communication with search

272d396

Search in Docs relies on an external project like "La Suite Find". We need to declare a common external network in order to connect to the search app and index our documents.

✨(backend) add dummy content to demo documents

f3b2907

We need to content in our demo documents so that we can test indexing.

✨(backend) add document search indexer

bc48ec9

Add indexer that loops across documents in the database, formats them as json objects and indexes them in the remote "Find" mico-service.

✨(backend) add async triggers to enable document indexation with find

7728b56

On document content or permission changes, start a celery job that will call the indexation API of the app "Find". Signed-off-by: Fabre Florian <ffabre@hybird.org>

🔧(compose) Add some ignore for docker-compose local overrides

04cd06a

Signed-off-by: Fabre Florian <ffabre@hybird.org>

✨(backend) add unit test for the 'index' command

da2f360

Signed-off-by: Fabre Florian <ffabre@hybird.org>

✨(backend) add document search view

409918e

New API view that calls the indexed documents search view (resource server) of app "Find". Signed-off-by: Fabre Florian <ffabre@hybird.org>

✨(backend) improve search indexer service configuration

43f3e50

New SEARCH_INDEXER_CLASS setting to define the indexer service class. Raise ImpoperlyConfigured errors instead of RuntimeError in index service. Signed-off-by: Fabre Florian <ffabre@hybird.org>

✨(backend) refactor indexation signals and fix circular import issues

9554640

Signed-off-by: Fabre Florian <ffabre@hybird.org>

✨(backend) Index partially empty documents

65d4d75

Only documents without title and content are ignored by indexer.

✨(backend) Index deleted documents

2af7937

Add SEARCH_INDEXER_COUNTDOWN as configurable setting. Make the search backend creation simplier (only 'get_document_indexer' now). Allow indexation of deleted documents. Signed-off-by: Fabre Florian <ffabre@hybird.org>

🔧(backend) force a valid key for token storage in development mode

e1fdace

Generate a fernet key for the OIDC_STORE_REFRESH_TOKEN_KEY in development settings if not set. Signed-off-by: Fabre Florian <ffabre@hybird.org>

🔧(backend) force a valid key for token storage in development mode

c71c0d7

Set a valid Ferney for OIDC_STORE_REFRESH_TOKEN_KEY in env.d/development/common Add .gitguardian.yaml configuration to ignore this key. Signed-off-by: Fabre Florian <ffabre@hybird.org>

WIP 💩(front) hack to use the fulltext search api

b1475df

galmuhammadi commented Nov 4, 2025

View reviewed changes

		@@ -0,0 +1,295 @@
		"""Document search index management utilities and indexers"""

		@@ -0,0 +1,429 @@
		"""
		Tests for Documents API endpoint in impress's core app: list

Index to search #2

Are you sure you want to change the base?

Index to search #2

Uh oh!

Conversation

galmuhammadi commented Nov 2, 2025

Purpose

Proposal

External contributions

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants