diff --git a/README.md b/README.md index 3467baa..488355e 100644 --- a/README.md +++ b/README.md @@ -141,15 +141,15 @@ You can test it's running by: `ps aux | grep -v grep | grep mongod` +#### NOTE: Manual operations on the MongoDB database +Whatever solution you choose to use to deploy your MongoDB database, if you perform manual operations on it such as document copies, document deletions or backup recovering, it's very important not to create duplicated documents. This precaution will prevent inconsistent and unexepected behaviour during Matilda's workflow. + ### Accessing the interface Each option you chose before you can now simply navigate to http://localhost:5000 if you installed the server locally or navigate to the remote server address. Keep in mind you may need to open the correct ports on your firewall(s) in order to reach the server. - -HTTP Requests from your client may not reach your server in some configuration environment, -in those few cases please check and edit the backend address in MATILDA's file `/web/server/gui/source/utils/backend.js`. Other configuration options are exposed in `/Configuration/conf.json`. ### First username and password @@ -166,19 +166,23 @@ All configuration changes that you may wish to make to MATILDA network and datab There you can change: - App ports (default 5000) and address (127.0.0.1) - Database location with address:port combination (127.0.0.1:27017) or mongoDB URI (mongodb://mongo:27017/?retryWrites=true&w=majority) - - The annotation models you want to be available inside MATILDA. The json files you are referring to must be included in the Configuration folder. + - The annotation models you want to be available inside MATILDA. The json files you are referring to, the models, must be included in the Configuration folder. + - Whether or not enforce session security (which is strongly advised) with the session_guard parameter. + - The event logging level saved in `/web/server/matilda.log` file. -If you are using the Docker version you can also perform additional configuration with `/Configuration/gunicorn_run.sh`. +If you are using the Docker version you can also perform additional configuration with `/Configuration/gunicorn_run.sh` in order to set the workers number and other gunicorn options. ### Annotation Models All configuration changes that you may wish to make to MATILDA's annotation model can be done by editing the json file `/Configuration/lida_model.json` or by adding a new one. This script contains a configuration dictionary that describes -which labels will appear in MATILDA's front end. +which labels will appear in MATILDA's annotation interface. You can also add an entire new annotation model file and put a reference to it in the `/Configuration/conf.json` file in order to instruct the program to load it on start. -You can currently add three different types of new labels to MATILDA: +## 3. Advanced Configuration + +Any annotation model has up to four different types of labels in MATILDA: 1. `multilabel_classification` :: will display as checkboxes which you can select one or more of. @@ -191,9 +195,9 @@ You can currently add three different types of new labels to MATILDA: 3. `string` :: will display underneath the user's utterance as a string response. This is the label field that would be used for a response to the user's query. - - -## 3. Advanced Configuration + +4. `global_classification_string` :: similar to multilabel_classification_string + but it's not dialogue turn-related, it refers to the entire dialogue. ### New Labels diff --git a/changelog.md b/changelog.md new file mode 100644 index 0000000..06cc165 --- /dev/null +++ b/changelog.md @@ -0,0 +1,67 @@ +![WLUPER AND UNIPI](images/research_collaboration_matilda.png) + +# MATILDA: Multi-AnnoTator multi-language Interactive Lightweight Dialogue Annotator + +**Authors:** Davide Cucurnia, Nikolai Rozanov, Irene Sucameli, Augusto Ciuffoletti, Maria Simi + +**Contact:** contact@wluper.com + +**Paper:** [link to the EACL paper](https://www.aclweb.org/anthology/2021.eacl-demos.5/) + +### Citation at bottom of README! (Please cite when using) + +## 1.5 + +- Configuration view in Admin Panel +- Supervision annotation rate bars +- Supervision view now allows to upload an already annotated dialogue collection +- Supervision view now allows to edit turn utterances of collections +- Inter-annotator stats for multilabel-string-classification +- Annotation Rate for single dialogue is calculated anew when entering dialogue annotation mode + +## 1.4 + +- Dialogue annotation view displays few annotation customizable options: + - Resizable layout, useful for very large or very small screen layouts. + - Character limit for long utterances: after that number of character a scroll-bar will be shown. + - Auto-save on turn change on/off switch. + + +## Citation +Please cite these two papers when using. +``` +@inproceedings{cucurnia-etal-2021-matilda, + title = "{MATILDA} - Multi-{A}nno{T}ator multi-language {I}nteractive{L}ight-weight Dialogue Annotator", + author = "Cucurnia, Davide and + Rozanov, Nikolai and + Sucameli, Irene and + Ciuffoletti, Augusto and + Simi, Maria", + booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations", + month = apr, + year = "2021", + address = "Online", + publisher = "Association for Computational Linguistics", + url = "https://www.aclweb.org/anthology/2021.eacl-demos.5", + pages = "32--39", + abstract = "Dialogue Systems are becoming ubiquitous in various forms and shapes - virtual assistants(Siri, Alexa, etc.), chat-bots, customer sup-port, chit-chat systems just to name a few.The advances in language models and their publication have democratised advanced NLP.However, data remains a crucial bottleneck.Our contribution to this essential pillar isMATILDA, to the best of our knowledge the first multi-annotator, multi-language dialogue annotation tool. MATILDA allows the creation of corpora, the management of users, the annotation of dialogues, the quick adaptation of the user interface to any language and the resolution of inter-annotator disagreement. We evaluate the tool on ease of use, annotation speed and interannotation resolution for both experts and novices and conclude that this tool not only supports the full pipeline for dialogue annotation, but also allows non-technical people to easily use it. We are completely open-sourcing the tool at https://github.com/wluper/matilda and provide a tutorial video1.", +} +``` + +``` +@inproceedings{collins-etal-2019-lida, + title = "{LIDA}: Lightweight Interactive Dialogue Annotator", + author = "Collins, Edward and + Rozanov, Nikolai and + Zhang, Bingbing", + booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations", + month = nov, + year = "2019", + address = "Hong Kong, China", + publisher = "Association for Computational Linguistics", + url = "https://www.aclweb.org/anthology/D19-3021", + doi = "10.18653/v1/D19-3021", + pages = "121--126", + abstract = "Dialogue systems have the potential to change how people interact with machines but are highly dependent on the quality of the data used to train them.It is therefore important to develop good dialogue annotation tools which can improve the speed and quality of dialogue data annotation. With this in mind, we introduce LIDA, an annotation tool designed specifically for conversation data. As far as we know, LIDA is the first dialogue annotation system that handles the entire dialogue annotation pipeline from raw text, as may be the output of transcription services, to structured conversation data. Furthermore it supports the integration of arbitrary machine learning mod-els as annotation recommenders and also has a dedicated interface to resolve inter-annotator disagreements such as after crowdsourcing an-notations for a dataset. LIDA is fully open source, documented and publicly available.[https://github.com/Wluper/lida] {--}{\textgreater} Screen Cast: https://vimeo.com/329824847", +} +``` diff --git a/configuration/conf.json b/configuration/conf.json index f335e53..c11c016 100644 --- a/configuration/conf.json +++ b/configuration/conf.json @@ -7,18 +7,17 @@ "lida_model.json", "unipi_model_v2.json" ], - "docker": false, "session_guard": true, - "full_log": false + "full_log": true }, "database": { - "name": "matilda", + "name": "matilda_wsgi", "legacy_configuration": { "address": "localhost", "port": 27017, "username": null, "password": null }, - "optional_uri": null + "optional_uri": "mongodb://mongo:27017/?retryWrites=true&w=majority" } -} \ No newline at end of file +} diff --git a/configuration/gunicorn_run.sh b/configuration/gunicorn_run.sh index 7c644dc..02cb197 100755 --- a/configuration/gunicorn_run.sh +++ b/configuration/gunicorn_run.sh @@ -1,2 +1,2 @@ #!/bin/sh -cd server; gunicorn --bind 0.0.0.0:5000 matilda_app:MatildaApp --log-file matilda.log --log-level 'info' +cd server; gunicorn --bind 0.0.0.0:5000 matilda_app:MatildaApp --log-file matilda.log --log-level 'info' \ No newline at end of file diff --git a/configuration/unipi_model.json b/configuration/unipi_model.json index 3c8065a..d7678c1 100644 --- a/configuration/unipi_model.json +++ b/configuration/unipi_model.json @@ -1,83 +1,68 @@ { - - "global_slot": { - - "description" : "General info related to the dialogue", - "label_type" : "multilabel_global_string", - "required" : false, - "labels" : [ - "result" - ] - }, - - "usr": { - "description" : "The user's query", - "label_type" : "string", - "required" : true - }, - - "sys": { - "description" : "The system's response", - "label_type" : "string", - "required" : true - }, - - "Dialogue_act": { - - "description" : "Type of dialogue act", - "label_type" : "multilabel_classification", - "required" : false, - "labels" :[ - "sys_greet", - "sys_inform_basic", - "sys_inform_proactive", - "sys_request", - "sys_select", - "sys_deny", - "usr_greet", - "usr_inform_basic", - "usr_inform_proactive", - "usr_request", - "usr_select", - "usr_deny" - ] - }, - - "Slot": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "Async": { - - "description": "To annotate async messages", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - "turn_ref" - - ] - - } - + "Async": { + "description": "To annotate async messages", + "label_type": "multilabel_classification_string", + "labels": [ + "turn_ref" + ], + "required": false + }, + "Dialogue_act": { + "description": "Type of dialogue act", + "label_type": "multilabel_classification", + "labels": [ + "sys_greet", + "sys_inform_basic", + "sys_inform_proactive", + "sys_request", + "sys_select", + "sys_deny", + "usr_greet", + "usr_inform_basic", + "usr_inform_proactive", + "usr_request", + "usr_select", + "usr_deny" + ], + "required": false + }, + "Slot": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "job_description", + "contract", + "duties", + "skill", + "past_experience", + "degree", + "age", + "languages", + "area", + "company_name", + "company_size", + "location", + "contact", + "other" + ], + "required": false + }, + "global_slot": { + "description": "General info related to the dialogue", + "label_type": "multilabel_global_string", + "labels": [ + "result" + ], + "required": false + }, + "sys": { + "description": "The system's response", + "label_type": "string", + "required": true + }, + "usr": { + "description": "The user's query", + "label_type": "string", + "required": false } +} \ No newline at end of file diff --git a/configuration/unipi_model_v2.json b/configuration/unipi_model_v2.json index e6b5503..d3b1e4d 100644 --- a/configuration/unipi_model_v2.json +++ b/configuration/unipi_model_v2.json @@ -1,279 +1,228 @@ -{ - - "sys": { - "description" : "The system's query", - "label_type" : "string", - "required" : true - }, - - "usr": { - "description" : "The user's query", - "label_type" : "string", - "required" : true - }, - - "sys_greet": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none" - ] - - }, - - "usr_greet": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none" - ] - - }, - - "sys_inform_basic": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "usr_inform_basic": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "sys_inform_proactive": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "sys_request": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_size", - "location" - ] - - }, - - "usr_inform_proactive": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description", - "contract", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_size", - "location", - "other" - ] - - }, - - "usr_request": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_size", - "location" - ] - - }, - - "sys_select": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description" - ] - - }, - - "usr_select": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description" - ] - - }, - - "sys_deny": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_size", - "location" - ] - - }, - - "usr_deny": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "none", - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_size", - "location" - ] - - }, - - "async": { - - "description": "To annotate async messages", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - "turn_ref" - ] - }, - - "global_slot": { - - "description" : "General info related to the dialogue", - "label_type" : "multilabel_global_string", - "required" : false, - "labels" : [ - "result" - ] - - } - - } +{ + "async": { + "description": "To annotate async messages", + "label_type": "multilabel_classification_string", + "labels": [ + "turn_ref" + ], + "required": false + }, + "global_slot": { + "description": "General info related to the dialogue", + "label_type": "multilabel_global_string", + "labels": [ + "result" + ], + "required": false + }, + "sys": { + "description": "The system's query", + "label_type": "string", + "required": true + }, + "sys_deny": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description", + "contract", + "duties", + "skill", + "past_experience", + "degree", + "age", + "languages", + "area", + "company_size", + "location" + ], + "required": false + }, + "sys_greet": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none" + ], + "required": false + }, + "sys_inform_basic": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description", + "contract", + "duties", + "skill", + "past_experience", + "degree", + "age", + "languages", + "area", + "company_name", + "company_size", + "location", + "contact", + "other" + ], + "required": false + }, + "sys_inform_proactive": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description", + "contract", + "duties", + "skill", + "past_experience", + "degree", + "age", + "languages", + "area", + "company_name", + "company_size", + "location", + "contact", + "other" + ], + "required": false + }, + "sys_request": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description", + "contract", + "duties", + "skill", + "past_experience", + "degree", + "other", + "company_name", + "age", + "languages", + "area", + "company_size", + "location" + ], + "required": false + }, + "sys_select": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description" + ], + "required": false + }, + "usr": { + "description": "The user's query", + "label_type": "string", + "required": true + }, + "usr_deny": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description", + "contract", + "duties", + "skill", + "past_experience", + "degree", + "age", + "languages", + "area", + "company_size", + "location" + ], + "required": false + }, + "usr_greet": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none" + ], + "required": false + }, + "usr_inform_basic": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description", + "contract", + "duties", + "skill", + "past_experience", + "degree", + "age", + "languages", + "area", + "company_name", + "company_size", + "location", + "contact", + "other" + ], + "required": false + }, + "usr_inform_proactive": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description", + "contract", + "skill", + "past_experience", + "degree", + "age", + "languages", + "area", + "company_size", + "location", + "other" + ], + "required": false + }, + "usr_request": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description", + "contract", + "contact", + "duties", + "skill", + "past_experience", + "degree", + "age", + "languages", + "area", + "company_size", + "company_name", + "location", + "other" + ], + "required": false + }, + "usr_select": { + "description": "Entity's value", + "label_type": "multilabel_classification_string", + "labels": [ + "none", + "job_description" + ], + "required": false + } +} diff --git a/configuration/unipi_v2.json b/configuration/unipi_v2.json deleted file mode 100644 index 5e81eab..0000000 --- a/configuration/unipi_v2.json +++ /dev/null @@ -1,277 +0,0 @@ -{ - - "sys": { - "description" : "The user's query", - "label_type" : "string", - "required" : true - }, - - "usr": { - "description" : "The user's query", - "label_type" : "string", - "required" : true - }, - - - "sys_inform_basic": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "usr_inform_basic": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "sys_inform_proactive": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "sys_request": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "sys_select": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "usr_greet": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "usr_inform_proactive": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "usr_request": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "usr_select": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "usr_deny": { - - "description" : "Entity's value", - "label_type" : "multilabel_classification_string", - "required" : false, - "labels" : [ - - "job_description", - "contract", - "duties", - "skill", - "past_experience", - "degree", - "age", - "languages", - "area", - "company_name", - "company_size", - "location", - "contact", - "other" - ] - - }, - - "global_slot": { - - "description" : "General info related to the dialogue", - "label_type" : "multilabel_global_string", - "required" : false, - "labels" : [ - "goal" - ] - - } - - } diff --git a/docker-compose.yml b/docker-compose.yml index 03c1b04..4cbdb5c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,4 +1,4 @@ -version: '3.6' +version: '3' services: mongo: image: mongo:latest @@ -10,10 +10,8 @@ services: driver: "json-file" networks: - mongo_net - ports: - - 27017:27017 matilda: - image: davivcu/matilda:multimodel + image: davivcu/matilda:latest volumes: - ./configuration:/configuration restart: always diff --git a/nginx_configuration/conf.d/matilda.conf b/nginx_configuration/conf.d/matilda.conf deleted file mode 100644 index 100df23..0000000 --- a/nginx_configuration/conf.d/matilda.conf +++ /dev/null @@ -1,12 +0,0 @@ -upstream matilda { - server matilda:5000; - } -server { - listen 80; - listen 443 ssl; - ssl_certificate /etc/nginx/cert.pem; - ssl_certificate_key /etc/nginx/key.pem; - location / { - proxy_pass http://matilda; - } -} diff --git a/nginx_configuration/docker-compose_nginx.yml b/nginx_configuration/docker-compose_nginx.yml index 717ef15..f1d3ddf 100644 --- a/nginx_configuration/docker-compose_nginx.yml +++ b/nginx_configuration/docker-compose_nginx.yml @@ -1,4 +1,4 @@ -version: '3.6' +version: '3' services: mongo: image: mongo:latest @@ -13,7 +13,7 @@ services: ports: - 27017:27017 matilda: - image: davivcu/matilda:beta + image: davivcu/matilda:latest volumes: - ./configuration:/configuration restart: always @@ -26,8 +26,8 @@ services: nginx: image: nginx volumes: - - ./nginx_configuration/conf.d:/etc/nginx/conf.d - - ./nginx_configuration/make-ssl-certs.sh:/docker-entrypoint.d/40-make-ssl-certs.sh + - ./nginx_configuration:/etc/nginx/conf.d + - ./nginx_configuration/make-ssl-certs.sh:/docker-entrypoint.d/make-ssl-certs.sh restart: always networks: - nginx_net diff --git a/nginx_configuration/make-ssl-certs.sh b/nginx_configuration/make-ssl-certs.sh index b7fc8e6..d3ed59d 100755 --- a/nginx_configuration/make-ssl-certs.sh +++ b/nginx_configuration/make-ssl-certs.sh @@ -1,3 +1 @@ -#!/bin/sh -CONFDIR="/etc/nginx/" -openssl req -subj '/CN=localhost' -x509 -newkey rsa:4096 -nodes -keyout $CONFDIR/key.pem -out $CONFDIR/cert.pem -days 365 +openssl req -subj '/CN=localhost' -x509 -newkey rsa:4096 -nodes -keyout /etc/nginx/conf.d/key.pem -out /etc/nginx/conf.d/cert.pem -days 365 \ No newline at end of file diff --git a/nginx_configuration/nginx.conf b/nginx_configuration/nginx.conf index 100df23..e0fa0eb 100644 --- a/nginx_configuration/nginx.conf +++ b/nginx_configuration/nginx.conf @@ -4,9 +4,9 @@ upstream matilda { server { listen 80; listen 443 ssl; - ssl_certificate /etc/nginx/cert.pem; - ssl_certificate_key /etc/nginx/key.pem; + ssl_certificate /etc/nginx/conf.d/cert.pem; + ssl_certificate_key /etc/nginx/conf.d/key.pem; location / { proxy_pass http://matilda; } -} +} \ No newline at end of file diff --git a/web/server/annotator_config.py b/web/server/annotator_config.py index 3e0c2a0..a3a4fb0 100644 --- a/web/server/annotator_config.py +++ b/web/server/annotator_config.py @@ -15,7 +15,7 @@ # >>>> Local <<<< from dummy_models import TypeDummyModel, BeliefStateDummyModel, PolicyDummyModel, SysDummyModel -#logging.basicConfig(filename='matilda.log', level=logging.DEBUG, format='%(asctime)s VALIDATOR %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p', style='%') +logging.basicConfig(filename='matilda.log', level=logging.DEBUG, format='%(asctime)s VALIDATOR %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p', style='%') ############################################## # CONFIG Dict @@ -47,7 +47,6 @@ class responsible for configuration and valid annotation structure """ DEFAULT_PATH = "" - DOCKER = False #importing json configuration file try: @@ -56,7 +55,6 @@ class responsible for configuration and valid annotation structure conf = json.load(json_file) __DEFAULT_PATH = "configuration/" DEFAULT_PATH = __DEFAULT_PATH - DOCKER = True except: #standalone with open('../../configuration/conf.json') as json_file: @@ -110,13 +108,13 @@ def validate_dialogue(annotation_style, dialogue: List[Dict[str, Any]]) -> Union message = ("ERROR1: Label \'{}\' is listed as \"required\" in the " \ "config.py file, but is missing from the provided " \ "dialogue in turn {}.".format(labelName, i)) - logger.info(message, turn) + logging.info(message, turn) return message if info["required"] and not turn[labelName]: message = ("ERROR2: Required label, \'{}\', does not have a value " \ "provided in the dialogue in turn {}".format(labelName, i)) - logger.info(message, turn) + logging.info(message, turn) return message if info["required"] and ("multilabel_classification" == info["label_type"]): @@ -127,11 +125,11 @@ def validate_dialogue(annotation_style, dialogue: List[Dict[str, Any]]) -> Union message = "ERROR3: One of the provided labels in the list: " \ "\'{}\' is not in allowed list according to " \ "config.py in turn {}".format(providedLabels, i) - logger.info(message, turn) + logging.info(message, turn) return message except: message = "ERROR4: dialogue in list couldn't validate with the current annotation style model" - logger.info(message) + logging.info(message) return message return dialogue diff --git a/web/server/database.py b/web/server/database.py index f39b1a9..c1db4fe 100644 --- a/web/server/database.py +++ b/web/server/database.py @@ -45,7 +45,7 @@ class DatabaseManagement(object): logging.info(" * MATILDA: Connecting to database "+str(databaseURI)) client.server_info() except Exception as e: - logging.warning(" * "+e+"\n * Connecting Errror. Trying again with legacy configuration...") + logging.warning(" * "+str(e)+"\n * Error connecting to Database. Trying again with legacy configuration...") conf["database"]["optional_uri"] = None databaseURI = database_uri_compose(conf["database"]) client = MongoClient(databaseURI) @@ -139,6 +139,11 @@ def updateDoc(searchFields, collection, updateFields): DatabaseManagement.selected(collection).update(searchFields, { "$set": updateFields }) + def updateDocs(searchFields, collection, updateFields): + + DatabaseManagement.selected(collection).update_many(searchFields, { "$set": updateFields }) + + def pullFromDoc(doc_id, collection, field): value = field["dialogue"] @@ -169,7 +174,6 @@ def pullFromDoc(doc_id, collection, field): def dumpDatabase(): - collections = DatabaseManagement.db.collection_names() dump = {} for i, collection_name in enumerate(collections): @@ -270,7 +274,7 @@ def checkSession(): def start(): if DatabaseManagement.users.count_documents({"id":"admin"}) == 0: DatabaseManagement.users.insert_one(LoginFuncs.administratorDefault) - logging.info(" * Default admin account created: please log-in with username 'admin' and password 'admin'") + logging.warning(" * Default admin account created: please log-in with username 'admin' and password 'admin'") else: - logging.info(" * Connected to database "+str(DatabaseManagement.databaseURI)) + logging.warning(" * Connected to database "+str(DatabaseManagement.databaseURI)) diff --git a/web/server/gui/assets/css/all_dialogues.css b/web/server/gui/assets/css/all_dialogues.css index 18ee77b..b10c25f 100644 --- a/web/server/gui/assets/css/all_dialogues.css +++ b/web/server/gui/assets/css/all_dialogues.css @@ -80,8 +80,7 @@ list-style-type: none; margin-left: 0; padding-left: 0; - - padding: 0; + padding: 0 0 3% 0; margin: 0px -10px 35px; overflow:hidden; } @@ -254,8 +253,12 @@ background-color: rgb(0 0 0 / 10%); } -.container-bar .annotated-bar:nth-child(2n) { - border-left:4px solid black; +.container-bar .annotated-bar:nth-child(1) { + border-left:none; +} + +.container-bar .annotated-bar:nth-child(n) { + border-left:4px solid #000000ab; } .annotated-fill { diff --git a/web/server/gui/assets/css/annotation_app.css b/web/server/gui/assets/css/annotation_app.css index 6e2e62d..fa5eaed 100644 --- a/web/server/gui/assets/css/annotation_app.css +++ b/web/server/gui/assets/css/annotation_app.css @@ -123,13 +123,13 @@ textarea { resize: none; width: 100%; height: 100%; - overflow-y: scroll; + overflow-y: auto; display: inline-block; justify-self: center; margin: 0; outline: none; border: 1px solid #e1e1e3; - padding: 1px 15px; + padding: 10px 15px; border-left:none; } textarea:focus { @@ -533,7 +533,7 @@ WluperDark: 0e181e } .slider { - height: 8px; + height: 7px; background: #c2d5e3; outline: none; opacity: 0.7; @@ -589,10 +589,18 @@ WluperDark: 0e181e outline: none; border: 1px solid #e1e1e3; padding: 10px 15px; + overflow-y: scroll; -webkit-touch-callout: none; /* iOS Safari */ -webkit-user-select: none; /* Safari */ -khtml-user-select: none; /* Konqueror HTML */ -moz-user-select: none; /* Old versions of Firefox */ -ms-user-select: none; /* Internet Explorer/Edge */ user-select: none; +} + +.turn-editing-btn { + float:right; + background-color:red; + box-shadow: 0 2px darkred; + margin-top: -0.5%; } \ No newline at end of file diff --git a/web/server/gui/assets/css/database.css b/web/server/gui/assets/css/database.css index 0d45b5e..c95b5bf 100644 --- a/web/server/gui/assets/css/database.css +++ b/web/server/gui/assets/css/database.css @@ -92,6 +92,12 @@ float:left; } +.collection-list .entry-assigned { + max-width: 27em; + overflow: hidden; + height: 1em; +} + .annotation-list .listed-entry { margin: 10px 0; color: white; @@ -481,6 +487,10 @@ input:not(:checked) + div { padding-right: 1%; } +.tc-no-min { + height:auto; +} + .two-columns .listed-entry { width:100% !important; float:none !important; @@ -588,7 +598,7 @@ input[type="password"]:hover { } .dialogue-entry-list { - grid-template: [row1-start] "info del" [row1-end] / 3fr 0.6fr; + grid-template: [row1-start] "info edit del" [row1-end] / 3fr 0.6fr 0.6fr; display: grid; box-shadow: 0px 5px 5px rgba(0,0,0,0.05); transition: ease 175ms all; @@ -606,6 +616,15 @@ input[type="password"]:hover { transition: ease 250ms all; } +.dialogue-entry-info .entry-annotated { + font-size:10px; + padding-right:6px; +} + +.dialogue-entry-list .edit-dialogue-button { + margin-top: 10px; +} + /******************************* // CONFIGURATION VIEW *******************************/ diff --git a/web/server/gui/assets/css/main.css b/web/server/gui/assets/css/main.css index 00484cd..ab1172a 100644 --- a/web/server/gui/assets/css/main.css +++ b/web/server/gui/assets/css/main.css @@ -359,6 +359,7 @@ select { #supervision #dialogue-menu { height:3em; + grid-column-start: 1; } #supervision #dialogue-menu .back-button { @@ -442,7 +443,7 @@ select { .copyright { font-size: .8em; line-height: 4em; - color: #8a95a0; + color: #cad1d9; margin-left: 20px; } .foot-bar { diff --git a/web/server/gui/index.html b/web/server/gui/index.html index 9c7d6d8..ac698fe 100644 --- a/web/server/gui/index.html +++ b/web/server/gui/index.html @@ -17,21 +17,21 @@ CSS ---------------------------------------------------------------> - - + + - + - - - - - - - - + + + + + + + + @@ -70,7 +70,7 @@
- MATILDA v.1.5 · Copyright © 2020 Wluper Ltd. · Developed in collaboration with University of Pisa + MATILDA v.1.5.1 · Copyright © 2020 Wluper Ltd. · Developed in collaboration with University of Pisa
@@ -88,34 +88,34 @@ SCRIPTS ============================================= --> - - - - - + + + + + - - - - - - - - - - + + + + + + + + + + - + - - - - - - - + + + + + + + diff --git a/web/server/gui/source/components/all_dialogues_components.js b/web/server/gui/source/components/all_dialogues_components.js index 52c193a..18e3485 100644 --- a/web/server/gui/source/components/all_dialogues_components.js +++ b/web/server/gui/source/components/all_dialogues_components.js @@ -70,7 +70,7 @@ Vue.component("all-dialogues", { mainApp.collectionRate = "0%" } if (mainApp.collectionRate != tempRate) { - backend.update_collection_fields(mainApp.activeCollection,{"status":mainApp.collectionRate}, false); + backend.update_collection_fields(mainApp.activeCollection,{"status":mainApp.collectionRate}); } }, @@ -191,30 +191,7 @@ Vue.component("all-dialogues", { - - - -
` diff --git a/web/server/gui/source/components/annotation_app_components.js b/web/server/gui/source/components/annotation_app_components.js index cba6ac1..01aebf6 100644 --- a/web/server/gui/source/components/annotation_app_components.js +++ b/web/server/gui/source/components/annotation_app_components.js @@ -65,7 +65,6 @@ Vue.component("annotation-app", { annotationAppEventBus.$on("go_back", this.go_back ); // DIALOGUE TURNS EVENTS - //annotationAppEventBus.$on( "turn_updated_string", this.turn_update ); annotationAppEventBus.$on( "update_turn_id", this.id_updated_from_ids_list ); annotationAppEventBus.$on( "delete_turn", this.remove_turn ); @@ -97,7 +96,6 @@ Vue.component("annotation-app", { annotationAppEventBus.$off("go_back", this.go_back ); // DIALOGUE TURNS EVENTS - //annotationAppEventBus.$off( "turn_updated_string", this.turn_update ); annotationAppEventBus.$off( "update_turn_id", this.id_updated_from_ids_list ); annotationAppEventBus.$off( "delete_turn", this.remove_turn ); @@ -306,6 +304,7 @@ Vue.component("annotation-app", { if (status == "success") { this.allDataSaved = true; //fields = {"status":mainApp.collectionRate}; + //this saves updated annotation rate together with each click on save button //backend.update_annotations(mainApp.activeCollection, fields, false); } else { this.allDataSaved = false; @@ -456,7 +455,7 @@ Vue.component('dialogue-menu',{ v-bind:value="dialogueTitle"> + v-on:focusout="toggleTitleEdit()"> --> @@ -550,7 +549,8 @@ Vue.component('dialogue-turns',{ v-bind:currentId="currentId" v-bind:myId="index" v-bind:selectingText="selectingText" - v-bind:style="{ maxWidth:maxWidth+'%' }"> + v-bind:style="{ maxWidth:maxWidth+'%' }" + v-bind:readOnly="readOnly"> @@ -570,24 +570,21 @@ Vue.component('dialogue-meta',{ } }, methods :{ - turn_updated_string : function(event){ - annotationAppEventBus.$emit("turn_updated_string", event ) - }, check_if_selected(){ return this.currentId==this.myId; }, update_id(){ annotationAppEventBus.$emit("update_turn_id", this.myId); }, - resize_turn_width(newValue) { + resize_turn_width: function(newValue) { this.maxWidth = newValue; annotationAppEventBus.$emit("change_option", "change_width", newValue); }, - change_max_chars(newValue) { + change_max_chars: function(newValue) { this.maxChars = newValue; annotationAppEventBus.$emit("change_option", "change_chars", newValue); }, - auto_save_value(event) { + auto_save_value: function(event) { annotationAppEventBus.$emit("change_auto_save", event.target.checked); }, }, @@ -641,18 +638,19 @@ Vue.component('dialogue-meta',{ Vue.component('dialogue-turn',{ // primaryElementClass is the class used to select the correct input field // to correctly set the focus when turns are changed with arrow keys or enter - props : ["turn","currentId","myId", "primaryElementClass", "selectingText"], + props : ["turn","currentId","myId", "primaryElementClass", "selectingText", "readOnly"], data: function (){ return { guiMessages, + editing:false, maxWidth: mainApp.maxWidth, maxChars: mainApp.maxChars, selectedWords:{"sys":{},"usr":{}} } }, methods :{ - turn_updated_string : function(event){ - annotationAppEventBus.$emit("turn_updated_string", event ) + turn_updated_string : function(){ + annotationAppEventBus.$emit("turn_updated_string") }, check_if_selected(){ return this.currentId==this.myId; @@ -660,9 +658,6 @@ Vue.component('dialogue-turn',{ update_id(){ annotationAppEventBus.$emit("update_turn_id", this.myId) }, - delete_this_turn(event) { - annotationAppEventBus.$emit("delete_turn", this.myId) - }, selection_done() { var out = ""; for (dict in this.selectedWords) { @@ -698,6 +693,26 @@ Vue.component('dialogue-turn',{ this.selectedWords = null; this.selectedWords = {"sys":{},"usr":{}}; }, + editing_mode: function() { + console.log("Editing turn content..."); + this.editing = true; + }, + delete_this_turn(event) { + if (confirm(guiMessages.selected.annotation_app.confirmEditing)) { + annotationAppEventBus.$emit("delete_turn", this.myId) + } + }, + send_new_turn: function() { + if (confirm(guiMessages.selected.annotation_app.confirmEditing)) { + this.turn_updated_string(); + this.editing = false; + } + }, + send_new_empty_turn: function() { + if (confirm(guiMessages.selected.annotation_app.confirmEditing)) { + annotationAppEventBus.$emit("new_empty_turn"); + } + } }, mounted(){ if (this.currentId==this.myId){ @@ -727,11 +742,29 @@ Vue.component('dialogue-turn',{
+ v-show="selectingText != false && readOnly != true" class="help-button btn btn-sm btn-primary" + v-on:click="selection_done()" v-bind:id="'selection-done-'+currentId" + style="float:right"> + {{guiMessages.selected.annotation_app.doneSelection}} + + + + + + +
@@ -746,11 +779,17 @@ Vue.component('dialogue-turn',{
+ + - + - + - - - - +
-
` @@ -814,13 +851,12 @@ Vue.component('annotations',{ template: `
-
- Current Turn: {{currentId}} -
+
Current Turn: {{currentId}}
+ v-bind:dialogueId="dialogueId" + v-bind:supervision="readOnly"> + v-bind:turn="currentId" + v-bind:supervision="readOnly"> -
+
- {{uniqueName.replace(/_/g, ' ')}}
@@ -129,7 +127,7 @@ Vue.component('classification-annotation',{
-
+
{{uniqueName.replace(/_/g, ' ')}}
@@ -205,7 +203,7 @@ Vue.component('classification-string-annotation', { adminEventBus.$on("switch_slot_values", this.switchSlotValue); this.collapsed = "new"; } else { - if ((this.classes.length > 1) && (this.slotView == "new")) { + if ((this.classes.length > 1 || this.supervision) && (this.slotView == "new")) { this.collapsed = "new"; } } @@ -544,7 +542,7 @@ Vue.component('classification-string-annotation', {
- Label: - +
- + + + Assigned: + + {{collection.assignedTo[0]}} + {{collection.assignedTo[1]}} + {{collection.assignedTo[2]}} + and {{collection.assignedTo.length-3}} more + + + Assigned: {{collection.assignedTo.join(", ")}} @@ -303,10 +338,10 @@ Vue.component("datamanagement-view", {
@@ -479,7 +514,12 @@ Vue.component('collection-users-reverse', { Gold: False
-
+ +
+ Assigned: {{collection.assignedTo.length}} +
+ +
Assigned: {{collection.assignedTo.join(", ")}}
@@ -504,10 +544,10 @@ Vue.component('collection-users-reverse', {
@@ -535,7 +575,10 @@ Vue.component('collection-entry-details', { role: mainApp.role, checkedUsers:[], changesSaved:"", - showGold: {boo: false, code:""} + showGold: {boo: false, code:""}, + clickedDialogueToEdit: "", + selectedAnnotator:"matilda_editor", + editorMode:"true", } }, @@ -543,6 +586,14 @@ Vue.component('collection-entry-details', { this.init(); }, + created () { + adminEventBus.$on("go_back_from_editor", this.clicked_entry); + }, + + beforeDestroyed () { + adminEventBus.$off("go_back_from_editor", this.clicked_entry); + }, + methods: { init : function(){ @@ -550,7 +601,7 @@ Vue.component('collection-entry-details', { .then( (response) => { console.log("Collection details",response[0]); this.entry = response[0]; - this.checkedUsers = response[0]["assignedTo"] + this.checkedUsers = response[0]["assignedTo"]; document.body.style.cursor = null; this.showGold.boo = this.check_if_gold(this.entry); }); @@ -614,13 +665,39 @@ Vue.component('collection-entry-details', { } ) } + }, + + clicked_entry(clickedDialogue) { + // this function is used to communicate the choosen dialogue to the editor component + // and also to close the editor by sending an empty id\\\\\\ + this.clickedDialogueToEdit = clickedDialogue; + databaseEventBus.$emit("load_dialogue_editor", clickedDialogue); + if (clickedDialogue != "") + adminEventBus.$emit("open_dialogue_editor"); + }, + + add_dialogue(){ + let new_dialogue_name = prompt(guiMessages.selected.admin.newDialogueNameInput); + if ((new_dialogue_name == "") || (new_dialogue_name == null) || (new_dialogue_name == undefined) + || (new_dialogue_name.includes(".",0)) || (new_dialogue_name.length > 20)) { + alert(guiMessages.selected.admin.cancelledDialogueNameInput); + return; + } + let params = { + "new_dialogue_name":new_dialogue_name + } + backend.admin_change_dialogue_content(this.selectedCollection, params, "new_dialogue", this.editorMode) + .then((response) => { + this.init(); + } + ); } }, template: `
-
-
+
+

{{guiMessages.selected.admin.editButton}}: {{entry.id}}

{{guiMessages.selected.collection.collTitle}}: @@ -636,7 +713,7 @@ Vue.component('collection-entry-details', {
-
    +

      {{guiMessages.selected.admin.assignedTo}}

    • @@ -648,10 +725,10 @@ Vue.component('collection-entry-details', {
      @@ -660,12 +737,12 @@ Vue.component('collection-entry-details', {
    -
    +
    @@ -673,22 +750,37 @@ Vue.component('collection-entry-details', {

    {{entry.documentLength}} {{guiMessages.selected.admin.dataItems}}

  • +
    + {{guiMessages.selected.admin.editButton}} +
    {{guiMessages.selected.lida.button_delete}}
    -
    +
    {{name}}
    - {{guiMessages.selected.lida.turns}}: {{content.length}} + {{guiMessages.selected.lida.turns}}: {{content.length-1}}
+
+ + +
+ +
+ +
` }); @@ -847,7 +939,7 @@ Vue.component('collection-creation', {
-
    +

      {{guiMessages.selected.admin.assignedTo}}

    • @@ -859,10 +951,10 @@ Vue.component('collection-creation', {
      @@ -882,4 +974,4 @@ Vue.component('collection-creation', {
` -}); +}); \ No newline at end of file diff --git a/web/server/gui/source/components/interannotator_components.js b/web/server/gui/source/components/interannotator_components.js index ed7fa33..863dcfa 100644 --- a/web/server/gui/source/components/interannotator_components.js +++ b/web/server/gui/source/components/interannotator_components.js @@ -330,16 +330,6 @@ Vue.component("interannotator-app", { }, - create_new_dialogue(event) { - - backend.admin_post_empty_dialogue() - .then( (newDialogueId) => { - - this.allDialogueMetadata.push({id: newDialogueId, num_turns: 0}); - - }); - }, - open_file(event){ let file = event.target.files[0]; this.handle_file(file); diff --git a/web/server/gui/source/components/login_components.js b/web/server/gui/source/components/login_components.js index 06b601d..e93d8b6 100644 --- a/web/server/gui/source/components/login_components.js +++ b/web/server/gui/source/components/login_components.js @@ -54,7 +54,6 @@ Vue.component("login-view", {
` }); diff --git a/web/server/gui/source/lida_view.js b/web/server/gui/source/lida_view.js index 044a5dd..0f021d3 100644 --- a/web/server/gui/source/lida_view.js +++ b/web/server/gui/source/lida_view.js @@ -65,7 +65,7 @@ var mainApp = new Vue({ setTurnWidth : function(){ if ((localStorage["turnWidth"] == null) || (localStorage["turnWidth"] == undefined)) { - localStorage.setItem("turnWidth", 80); + localStorage.setItem("turnWidth", 87); return localStorage["turnWidth"]; } else { return localStorage["turnWidth"]; @@ -74,7 +74,7 @@ var mainApp = new Vue({ setMaxChars : function(){ if (localStorage["maxChars"] == null) { - localStorage.setItem("maxChars", 80); + localStorage.setItem("maxChars",100); return localStorage["maxChars"]; } else { return localStorage["maxChars"]; @@ -305,7 +305,7 @@ var mainApp = new Vue({ v-bind:userName="userName"> - diff --git a/web/server/gui/source/utils/backend.js b/web/server/gui/source/utils/backend.js index c6be34a..b22c789 100644 --- a/web/server/gui/source/utils/backend.js +++ b/web/server/gui/source/utils/backend.js @@ -21,8 +21,7 @@ async function annotate_query(query){ } catch (error) { - console.log(error); - alert(error); + utils.handle_error_message(error); } @@ -104,7 +103,7 @@ async function manage_configuration_file(mode,id,jsonFile) { } } -async function get_annotation_style_async(collection,id,supervision){ +async function get_annotation_style_async(collection, id, supervision, editorMode=false){ var dialogues = {} @@ -114,7 +113,7 @@ async function get_annotation_style_async(collection,id,supervision){ } else { if (supervision != undefined) { - var apiLink = API_BASE+"/supervision/"+mainApp.userName+`/dialogue_annotationstyle/${collection}/${id}` + var apiLink = API_BASE+"/supervision/"+mainApp.userName+`/dialogue_annotationstyle/${collection}/${id}/${editorMode}` } else { var apiLink = API_BASE+"/"+mainApp.userName+`/dialogue_annotationstyle/${collection}/${id}` } @@ -124,12 +123,7 @@ async function get_annotation_style_async(collection,id,supervision){ var response = await axios.get(apiLink) - if (response["data"]["error"] != undefined) { - alert(response["data"]["error"]) - if (response["data"]["status"] == "logout") { - mainApp.force_logout() - } - } + utils.handle_auth_rejection(response) dialogueStyle = response.data console.log("============= ANNOTATION CLASSES ==============") @@ -157,6 +151,8 @@ async function get_logs(complete=undefined){ try { var response = await axios.get(apiLink) + utils.handle_auth_rejection(response) + let logs = response.data console.log("============= APP LOGS ==============") console.log(logs) @@ -164,8 +160,7 @@ async function get_logs(complete=undefined){ } catch (error) { - console.log(error); - alert(error); + utils.handle_error_message(error); } }; @@ -185,8 +180,7 @@ async function get_registered_annotation_styles(){ } catch (error) { - console.log(error); - alert(error); + utils.handle_error_message(error); } } @@ -209,8 +203,7 @@ async function write_tag(id,tag,value) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } @@ -242,12 +235,7 @@ async function get_all_dialogue_ids_async(admin, interannotatorCollection=undefi console.log(response); - if (response["data"]["error"] != undefined) { - alert(response["data"]["error"]) - if (response["data"]["status"] == "logout") { - mainApp.force_logout() - } - } + utils.handle_auth_rejection(response) dialoguesList = response.data console.log("=========== ALL DIALOGUE METADATA LIST ===========") @@ -255,8 +243,7 @@ async function get_all_dialogue_ids_async(admin, interannotatorCollection=undefi } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } @@ -275,7 +262,7 @@ async function change_dialogue_name_async(oldName, newName) { } catch(error) { - console.log(error); + utils.handle_error_message(error); } return false; @@ -306,7 +293,7 @@ async function get_all_dialogues_async(admin) { } catch(error) { - console.log(error) + utils.handle_error_message(error); return response } @@ -322,18 +309,18 @@ async function get_all_dialogues_async(admin) { } catch(error) { - console.log(error); + utils.handle_error_message(error); } } }; -async function get_single_dialogue_async(id, collection, supervision) { +async function get_single_dialogue_async(id, collection, supervision, editorMode=false) { if (supervision != undefined) { - apiLink = API_BASE+"/supervision/"+mainApp.userName+"/dialogues/"+id; + apiLink = API_BASE+"/supervision/"+mainApp.userName+"/"+collection+"/"+id+"/"+editorMode; var response = await axios.get( apiLink ) @@ -363,38 +350,11 @@ async function get_single_dialogue_async(id, collection, supervision) { } catch(error) { - console.log(error) - alert(error); - //logout? - } - -} - - - -async function post_empty_dialogue(collection) { - - try { - - if (collection == undefined) { - var response = await RESTdialogues("POST", null, null); - } else { - var response = await RESTdialogues("POST", null, null, collection); - } - - console.log(response) - - return response.data.id - - } catch(error) { - - console.log(error); - alert(error); + utils.handle_error_message(error); } } - async function post_new_dialogues_from_string_lists_async(stringLists) { try { @@ -407,8 +367,7 @@ async function post_new_dialogues_from_string_lists_async(stringLists) { } catch(error) { - console.log(error); - alert(error); + utils.handle_error_message(error); } } @@ -427,8 +386,7 @@ async function post_new_dialogue_from_json_string_async(jsonString, fileName) { } catch(error) { - console.log(error); - alert(error); + utils.handle_error_message(error); } @@ -444,19 +402,13 @@ async function put_single_dialogue_async(event, dialogueId, dTurns, collection) status = response.data.status console.log('status', status) - if (response["data"]["error"] != undefined) { - alert(response["data"]["error"]) - if (response["data"]["status"] == "logout") { - mainApp.force_logout() - } - } + utils.handle_auth_rejection(response) return status } catch(error) { - console.log(error); - alert(error) + utils.handle_error_message(error); } @@ -472,8 +424,7 @@ async function del_single_dialogue_async(dialogueId) { } catch(error) { - console.log(error); - alert(error) + utils.handle_error_message(error); } }; @@ -496,14 +447,13 @@ async function del_all_dialogues_async(admin) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } async function recover_dialogues(id) { -const apiLink = API_BASE+"/"+mainApp.userName+`/annotations_recover/${id}` + const apiLink = API_BASE+"/"+mainApp.userName+`/annotations_recover/${id}` try { @@ -514,8 +464,7 @@ const apiLink = API_BASE+"/"+mainApp.userName+`/annotations_recover/${id}` } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } @@ -528,12 +477,13 @@ async function load_dialogues(doc) { try { var response = await axios.put(apiLink) + + utils.handle_auth_rejection(response) return response } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } @@ -548,8 +498,7 @@ async function supervision(annotator,doc) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } @@ -608,8 +557,7 @@ async function get_all_db_entries_ids() { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } @@ -636,17 +584,13 @@ async function update_annotations(activeColl,fields,backup) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } -async function update_collection_fields(activeColl,fields, annotator) { - - if (annotator == undefined) - annotator = mainApp.userName; +async function update_collection_fields(activeColl,fields, annotator=mainApp.userName) { var apiLink = API_BASE+"/"+annotator+`/database/fields/${activeColl}` @@ -660,8 +604,7 @@ async function update_collection_fields(activeColl,fields, annotator) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } @@ -680,7 +623,7 @@ async function del_db_entry_async(pairs, collectionDB) { } catch(error) { - console.log(error); + utils.handle_error_message(error); } } @@ -699,7 +642,7 @@ async function get_db_entry_async(entryId,DBcollection) { } catch(error) { - console.log(error); + utils.handle_error_message(error); } } @@ -722,8 +665,7 @@ async function get_database_dump_async() { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } @@ -745,8 +687,7 @@ async function login(loginName,loginPass) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } @@ -768,12 +709,7 @@ async function get_collections_ids_async(DBcollection) { console.log(response) - if (response["data"]["error"] != undefined) { - alert(response["data"]["error"]) - if (response["data"]["status"] == "logout") { - mainApp.force_logout() - } - } + utils.handle_auth_rejection(response) entriesList = response.data @@ -781,9 +717,7 @@ async function get_collections_ids_async(DBcollection) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) - + utils.handle_error_message(error); } } @@ -803,12 +737,7 @@ async function get_specific_collections(DBcollection,fields,projection) { console.log(response) - if (response["data"]["error"] != undefined) { - alert(response["data"]["error"]) - if (response["data"]["status"] == "logout") { - mainApp.force_logout() - } - } + utils.handle_auth_rejection(response) entriesList = response.data @@ -816,8 +745,7 @@ async function get_specific_collections(DBcollection,fields,projection) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } @@ -834,21 +762,15 @@ async function get_collections_and_annotations_meta() { console.log(response) - if (response["data"]["error"] != undefined) { - alert(response["data"]["error"]) - if (response["data"]["status"] == "logout") { - mainApp.force_logout() - } - } + utils.handle_auth_rejection(response); - entriesList = response.data + entriesList = response.data - return entriesList + return entriesList } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } @@ -865,8 +787,7 @@ async function remove_from_collection_async(DBcollection, id, fields) { } catch(error) { - console.log(error); - alert(guiMessages.selected.lida.connectionError) + utils.handle_error_message(error); } } @@ -975,8 +896,7 @@ async function get_scores_async(collection){ } catch (error) { - console.log(error); - alert(error) + utils.handle_error_message(error); } } @@ -995,8 +915,7 @@ async function get_errors_async(collection,dialogueId){ } catch (error) { - console.log(error); - alert(error) + utils.handle_error_message(error); } } @@ -1014,8 +933,7 @@ async function get_collection_errors_async(collectionId){ } catch (error) { - console.log(error); - alert(error) + utils.handle_error_message(error); } } @@ -1032,52 +950,57 @@ async function put_error_async(listOfErrors){ } catch (error) { - console.log(error); - alert(error) + utils.handle_error_message(error); return false } } -/* -async function admin_post_empty_dialogue() { +async function admin_import_for_interannotation(collection, newFile=undefined) { - const apiLink = API_BASE+"/dialogues"; + var apiLink = API_BASE+`/interannotation_import/${collection}` try { - var response = await axios.post( apiLink, null, null ) - console.log(response) + if (newFile == undefined) + var response = await axios.get( apiLink ) + else + var response = await axios.post( apiLink, { payload:JSON.parse(newFile) } ) - return response.data.id + console.log('---- RESPONSE TO POST DATA ----', response); + return response; } catch(error) { - console.log(error) + utils.handle_error_message(error); } + return false; } -*/ -async function admin_import_for_interannotation(collection, newFile=undefined) { - var apiLink = API_BASE+`/interannotation_import/${collection}` +async function admin_change_dialogue_content(activeColl, fields, mode, editorMode=false) { + + document.body.style.cursor = "progress"; + var apiLink = API_BASE+"/"+mainApp.userName+`/database/${mode}/${activeColl}/${editorMode}` try { - if (newFile == undefined) - var response = await axios.get( apiLink ) - else - var response = await axios.post( apiLink, { payload:JSON.parse(newFile) } ) + var response = await axios.put(apiLink, fields) - console.log('---- RESPONSE TO POST DATA ----', response); - return response; + if (utils.handle_auth_rejection(response) === false) { + console.log("======== UPDATING DATABASE ========") + console.log(response) + document.body.style = null; + return response + } } catch(error) { - console.log(error); + utils.handle_error_message(error); + document.body.style = null; } - return false; } + async function import_new_dialogues_from_string_lists_async(stringLists) { var apiLink = API_BASE+`/dialogues_import` @@ -1090,7 +1013,7 @@ async function import_new_dialogues_from_string_lists_async(stringLists) { } catch(error) { - console.log(error); + utils.handle_error_message(error); } return false; } @@ -1104,20 +1027,14 @@ async function get_all_users(){ try { var response = await axios.get( apiLink ) - if (response["data"]["error"] != undefined) { - alert(response["data"]["error"]) - if (response["data"]["status"] == "logout") { - mainApp.force_logout() - } - } + utils.handle_auth_rejection(response) users = response.data return users } catch(error) { - console.log(error); - alert(error) + utils.handle_error_message(error); } } @@ -1136,12 +1053,12 @@ async function create_user(parameters,update=false){ var response = await axios.put( apiLink, parameters ) } - return response + if (utils.handle_auth_rejection(response) === false) + return response } catch(error) { - console.log(error); - alert(error) + utils.handle_error_message(error); } } @@ -1171,7 +1088,7 @@ backend = load_dialogues : load_dialogues, recover_dialogues : recover_dialogues, - post_empty_dialogue : post_empty_dialogue, + admin_change_dialogue_content : admin_change_dialogue_content, post_new_dialogues_from_string_lists_async : post_new_dialogues_from_string_lists_async, post_new_dialogue_from_json_string_async : post_new_dialogue_from_json_string_async, diff --git a/web/server/gui/source/utils/languages.js b/web/server/gui/source/utils/languages.js index 8acbf1a..badada8 100644 --- a/web/server/gui/source/utils/languages.js +++ b/web/server/gui/source/utils/languages.js @@ -29,6 +29,7 @@ guiMessages = { userButton: "Users Management", users:"Users", createButton: "Add", + addDialogue: "Add a dialogue", createAnno:"Create new annotation style", newAnnotations: "Import annotations from a file", editButton:"Edit", @@ -43,6 +44,7 @@ guiMessages = { annotation:"Annotation", goToAnnotation: "Annotation View", backToColl:"Back", + backFromEditor:"Close editor", importConflictsResult:"Nothing to import, no annotated versions have been produced yet", cantDeleteAdmin:"Sorry, default admin account can't be deleted", annotationInProgress:"Annotations in progress for collection", @@ -62,6 +64,9 @@ guiMessages = { loadedAnno: "Loaded annotation models", confirmSave: "Do you want to save the new configuration?", confGuide: "Configuration Guide", + noAnnotations: "No users started to annotate this collection yet.", + newDialogueNameInput: "Insert the name of the new dialogue. The name must be unique in the selected collection and must not contain dots nor be longer than 20 chars. The dialogue will be empty unless you enter it and edit its turns.", + cancelledDialogueNameInput: "Not a valid name. Operation cancelled." }, lida: { button_fileFormatInfo: "Format Info", @@ -75,6 +80,8 @@ guiMessages = { drop:"Drop Files Anywhere to Upload!", buttonCollections:"Collections", connectionError: "Couldn't connect to server, check that it's running or the address in backend_config", + genericError: "Server Error. Something went wrong.", + serverError: "Server Error. The server received your request but couldn't process it correctly.", backupDone:"Backup done.", backupPost:"Backup postponed: nothing to save.", backupFailed:"Backup failed.", @@ -132,12 +139,12 @@ guiMessages = { dataManagementTitle:"Select a user or a collection", collectionToAnnotator:"Assign collections to selected user", annotatorToCollection:"Assign annotators to selected collection" - }, annotation_app: { turnId: "Turn Id", enter: "Enter", save: "Save", + saveAssignement: "Save new assignement and info", enterQuery:"Enter New Query", allSaved: "All Changes Saved", unsaved: "Unsaved Changes", @@ -150,9 +157,17 @@ guiMessages = { switchAnnotationView: "Expanded Slots View", annotationPref:"Annotation Preferences", turnWidth:"Turn Width", - scrollAfter:"Scroll-bar after", + scrollAfter:"Scroll-bar with sentence longer than", chars:"characters", autoSave:"Auto-save on turn changed", + doneSelection:"Selection done", + doneEditing: "Turn editing done", + sendEmptyTurn: "Add a new empty turn", + deleteTurn: "Delete turn", + confirmEditing: "This will change the turn content of this collection for every annotator. The annotation associated to this turn will not be affected.", + lastTurn: "The last turn of the dialogue can't be deleted. Please go back to the dialogue list and delete the dialogue instead.", + alsoAssignedTo: "Also assigned to", + done: "Done", }, resolution_app: { errorId: "Error Id:", @@ -216,9 +231,9 @@ guiMessages = { the annotator_config.py file in the "server" folder. `, ` - By default, the only required key-value pair in each turn - is called "usr" and should be the user's query as - a string. + By default, the only two required key-value pairs in each turn + are called "usr" and "sys". + They should be the user's and system's query as a string. ` ], modal_agreementScores: @@ -261,8 +276,8 @@ guiMessages = { ], modal_annotationStyle: [ - `String usually are used to represent the utterances in the dialogue. There should be at least one for turn, so the default value for - required field in this category is set to true. The String category is shown in interface as a read-only text-area.`, + `String usually are used to represent the utterances in the dialogue. There should be two for turn and their names should be "usr" and "sys". + The String category is shown in interface as a read-only text-area.`, `Multilabel Classification are used to identify if a certain characteristic is present on the selected turn or not. The labels you may insert in this category don't have a value, they are just selected or unselected. In interface Multilabel Classification is shown as a group of checkboxes the annotators can mark.`, @@ -391,6 +406,7 @@ guiMessages = { userButton: "Gestione Utenti", users:"Utenti", createButton: "Crea", + addDialogue: "Aggiungi un dialogo", createAnno:"Crea modello di annotazione", editButton:"Modifica", hideButton:"Nascondi", @@ -404,6 +420,7 @@ guiMessages = { annotation:"Annotazione", goToAnnotation: "Vai alla vista Annotazione", backToColl: "Indietro", + backFromEditor: "Chiudi editor", newAnnotations: "Carica annotazioni da file", importConflictsResult:"Non è stato trovato niente da importare, nessuna versione annotata è stata ancora prodotta.", cantDeleteAdmin:"L'account speciale 'admin' non può essere eliminato", @@ -424,6 +441,9 @@ guiMessages = { loadedAnno: "Modelli di annotazione caricati", confirmSave: "Vuoi salvare la nuova configurazione?", confGuide: "Guida alla Configurazione", + noAnnotations: "Nessun utente ha iniziato ad annotare questa collezione.", + newDialogueNameInput: "Inserisci un nome per il nuovo dialogo. Il nome deve essere univoco nella collezione selezionata e non può contenere punti o essere più lungo di 20 caratteri. Il nuovo dialogo sarà vuoto, una volta inserito potrai modificarlo come avviene per gli altri dialoghi.", + cancelledDialogueNameInput: "Il nome inserito non è valido. Operazione cancellata." }, lida: { button_fileFormatInfo: "Info sui formati", @@ -437,7 +457,9 @@ guiMessages = { drop:"Rilascia i file per caricarli!", buttonCollections: "Collezioni", button_downloadAll: "Scarica i dati", - connectionError:"Impossibile connettersi al server, controlla che sia in funzione o verifica l'indirizzo in backend_config", + connectionError:"Impossibile connettersi al server, controlla che sia in funzione o verificane l'indirizzo.", + serverError:"Server Error. Il server ha ricevuto la tua richiesta ma non ha potuto processarla correttamente.", + genericError: "Server Error. Qualcosa è andato storto.", backupDone:"Backup completato.", backupPost:"Backup postposto: niente da salvare.", backupFailed:"Backup fallito.", @@ -500,6 +522,7 @@ guiMessages = { turnId: "Id Turno", enter: "Invia", save: "Salva", + saveAssignement: "Salva nuovi assegnamenti e informazioni", enterQuery: "Nuova richiesta", allSaved: "Modifiche salvate", unsaved: "Cambiamenti non salvati", @@ -512,9 +535,17 @@ guiMessages = { switchAnnotationView: "Visuale Slot-Espansi", annotationPref:"Preferenze di Annotazione", turnWidth:"Larghezza finestra turni", - scrollAfter:"Scroll-bar dopo", + scrollAfter:"Scroll-bar con frasi più lunghe di", chars:"caratteri", autoSave:"Auto-save al cambio di turno", + doneSelection: "Selezione completata", + doneEditing: "Modifica turno completata", + sendEmptyTurn: "Aggiungi un nuovo turno vuoto", + sendDeleteTurn: "Elimina turno", + confirmEditing: "Questa operazione cambierà il contenuto del turno selezionato per tutti gli annotatori. Non influirà su eventuali annotazioni associate.", + lastTurn: "L'ultimo turno di un dialogo non può essere eliminato. Puoi tornare indietro alla lista dei dialoghi ed eliminare l'intero dialogo.", + alsoAssignedTo: "Assegnato anche a", + done: "Completato", }, resolution_app: { errorId: "Id Conflitto:", @@ -724,6 +755,7 @@ guiMessages = { users: "Benutzer", fileName:"File Name", createButton: "Hinzufügen", + addDialogue: "Add a dialogue", createAnno:"Create new annotation style", editButton: "Bearbeiten", hideButton:"Hide", @@ -737,6 +769,7 @@ guiMessages = { annotation: "Annotation", goToAnnotation: "Annotation View", backToColl: "Zurück", + backFromEditor: "Close editor", importConflictsResult: "Es wurde nichts importiert, es wurden noch keine kommentierten Versionen erstellt", cantDeleteAdmin: "Das Standard-Administratorkonto kann leider nicht gelöscht werden", annotationInProgress: "Anmerkungen zur Sammlung werden ausgeführt", @@ -756,6 +789,9 @@ guiMessages = { loadedAnno: "Loaded annotation models", confirmSave: "Do you want to save the new configuration?", confGuide: "Configuration Guide", + noAnnotations: "No users started to annotate this collection yet.", + newDialogueNameInput: "Insert the name of the new dialogue. The name must be unique in the selected collection and must not contain dots. The dialogue will be empty unless you enter it and edit its turns.", + cancelledDialogueNameInput: "Not a valid name. Operation cancelled." }, lida: { button_fileFormatInfo: "Info formatieren", @@ -770,6 +806,8 @@ guiMessages = { button_downloadAll: "Daten herunterladen", buttonCollections: "Sammlungen", connectionError: "Es konnte keine Verbindung zum Server hergestellt werden. Überprüfen Sie, ob der Server ausgeführt wird, oder die Adresse in backend_config.", + serverError: "Server Error. The server received your request but couldn't process it correctly.", + genericError: "Server Error. Something went wrong.", backupDone: "Sicherung abgeschlossen.", backupPost: "Backup verschoben: nichts zu speichern.", backupFailed: "Sicherung fehlgeschlagen.", @@ -833,6 +871,7 @@ guiMessages = { turnId: "Dialogueschritt", enter: "Eintretten", save: "Speichern", + saveAssignement: "Save new assignement and info", enterQuery: "Neue Abfrage eingeben", allSaved: "Alle Änderungen gespeichert", unsaved: "Nicht gespeicherte Änderungen", @@ -848,6 +887,14 @@ guiMessages = { scrollAfter:"Scroll-bar after", chars:"characters", autoSave:"Auto-save on turn changed", + doneSelection: "Selction done", + doneEditing: "Turn editing done", + sendEmptyTurn: "Add a new empty turn", + sendDeleteTurn: "Delete turn", + confirmEditing: "This will change the turn content of this collection for every annotator. The annotation associated to this turn will not be affected.", + lastTurn: "The last turn of the dialogue can't be deleted. Please go back to the dialogue list and delete the dialogue instead.", + alsoAssignedTo: "Also assigned to", + done:"Done", }, resolution_app: { errorId: "Error Id:", diff --git a/web/server/gui/source/utils/utils.js b/web/server/gui/source/utils/utils.js index 22a053e..812991c 100644 --- a/web/server/gui/source/utils/utils.js +++ b/web/server/gui/source/utils/utils.js @@ -199,6 +199,27 @@ function get_token_range(event,selectionText) { return range } +function handle_error_message(error) { + if (error.request) { + alert(guiMessages.selected.lida.connectionError) + } else if (error.response) { + alert(guiMessages.selected.lida.serverError) + } else { + alert(guiMessages.selected.lida.genericError+" "+error) + console.log(error); + } +} + +function handle_auth_rejection(response) { + if (response["data"]["error"] != undefined) { + alert(response["data"]["error"]) + if (response["data"]["status"] == "logout") { + mainApp.force_logout() + } + } else { + return false + } +} utils = { @@ -207,5 +228,8 @@ utils = get_turn_data : get_turn_data, get_all_turns_data : get_all_turns_data, create_date : create_date, - get_token_range : get_token_range + get_token_range : get_token_range, + + handle_error_message : handle_error_message, + handle_auth_rejection : handle_auth_rejection } diff --git a/web/server/matilda_app.py b/web/server/matilda_app.py index 73b79f5..a5dc26f 100644 --- a/web/server/matilda_app.py +++ b/web/server/matilda_app.py @@ -4,16 +4,14 @@ # == Native == import os -import sys import json -import copy import datetime import logging -from typing import Dict, List, Any, Tuple, Hashable, Iterable, Union +from typing import Dict, List, Any from collections import defaultdict # == Flask == -from flask import Flask, jsonify, request, render_template, session +from flask import Flask, jsonify, request, render_template from flask_cors import CORS # == Logging == @@ -59,7 +57,7 @@ @MatildaApp.route('/') def welcome(): - return render_template("index.html") + return render_template("index.html") ############################################## # FUNCTION HANDLERS @@ -84,8 +82,6 @@ def handle_configuration_file(annotationStyle=None, option=None): for setting in section: responseObject[setting][section] = setting - responseObject["app"]["docker"] = Configuration.DOCKER - #default mongodb databases not listed ignoreList = ["admin","config","local"] @@ -259,8 +255,9 @@ def handle_dialogues_metadata_resource(user, id=None, collection=None): except: #reload session dialogueFile.create_userspace(user) + #import and format new dialogues for docCollection in docRetrieved: - dialogueFile.update_dialogues(user, docCollection["document"]) + __add_new_dialogues_from_json_dict(user, collection, responseObject, dialogueDict=docCollection["document"]) dialogueFile.change_collection(user, collection) @@ -312,8 +309,8 @@ def handle_collections_and_annotations_metadata(): @MatildaApp.route('//dialogues//',methods=['GET','POST','DELETE']) @MatildaApp.route('//dialogues//',methods=['PUT']) @MatildaApp.route('//dialogues/collection/',methods=['POST']) -@MatildaApp.route('/supervision//dialogues/',methods=['GET']) -def handle_dialogues_resource(user=None, id=None, fileName=None, supervisor=None): +@MatildaApp.route('/supervision////',methods=['GET']) +def handle_dialogues_resource(user=None, id=None, fileName=None, supervisor=None, editorMode=None): """ GET - All dialogues @@ -342,7 +339,24 @@ def handle_dialogues_resource(user=None, id=None, fileName=None, supervisor=None responseObject = { "status": "success" } if supervisor: - responseObject = dialogueFile.get_dialogue("Su_"+supervisor, id = id) + + if editorMode == "true": + #direct access to database + docRetrieved = DatabaseManagement.readDatabase("dialogues_collections",{"id":fileName}) + return jsonify({"dialogue":docRetrieved[0]["document"][id]}) + + try: + responseObject = dialogueFile.get_dialogue("Su_"+supervisor, id = id) + except: + #reload session + dialogueFile.clean_workspace("Su_"+supervisor) + #then load and import dialogues + docRetrieved = DatabaseManagement.readDatabase("dialogues_collections",{"id":fileName}) + if len(docRetrieved) != 0: + for docCollection in docRetrieved: + __add_new_dialogues_from_json_dict("Su_"+supervisor, fileName, {}, dialogueDict=docCollection["document"]) + responseObject = dialogueFile.get_dialogue("Su_"+supervisor, id = id) + return jsonify(responseObject) elif fileName: @@ -389,8 +403,8 @@ def retrieve_and_return_annotation_styles(): @MatildaApp.route('/dialogue_annotationstyle/', methods=['GET']) @MatildaApp.route('//dialogue_annotationstyle//',methods=['GET']) -@MatildaApp.route('/supervision//dialogue_annotationstyle//', methods=['GET']) -def handle_annotation_style_resource(collection,user=None,id=None,supervisor=None): +@MatildaApp.route('/supervision//dialogue_annotationstyle///', methods=['GET']) +def handle_annotation_style_resource(collection,user=None,id=None,supervisor=None,editorMode=None): """ GET - Returns the annotation style for different workspace "global admin", "user specific" or "supervision" """ @@ -398,9 +412,20 @@ def handle_annotation_style_resource(collection,user=None,id=None,supervisor=Non #retrieve and load correct annotation style annotationStyle = retrieve_annotation_style_name(collection) - #return data for the right view + #return data for the correct view if supervisor: - dialogue = dialogueFile.get_dialogue("Su_"+supervisor, id = id) + try: + dialogue = dialogueFile.get_dialogue("Su_"+supervisor, id = id) + except: + #reload session + dialogueFile.clean_workspace("Su_"+supervisor) + #then load and import dialogues + docRetrieved = DatabaseManagement.readDatabase("dialogues_collections",{"id":collection}) + if len(docRetrieved) != 0: + for docCollection in docRetrieved: + __add_new_dialogues_from_json_dict("Su_"+supervisor, collection, {}, dialogueDict=docCollection["document"]) + dialogue = dialogueFile.get_dialogue("Su_"+supervisor, id = id) + Configuration.validate_dialogue(annotationStyle,dialogue["dialogue"]) return jsonify( Configuration.create_annotation_dict(annotationStyle) ) @@ -463,9 +488,10 @@ def handle_name_resource(user): @MatildaApp.route('/database', methods=['GET']) @MatildaApp.route('//database//',methods=['PUT']) +@MatildaApp.route('//database///',methods=['PUT']) @MatildaApp.route('/database/',methods=['POST']) @MatildaApp.route('/database//',methods=['GET','POST']) -def handle_database_resource(id=None, user=None, mode=None, DBcollection=None, activecollection=None): +def handle_database_resource(id=None, user=None, mode=None, DBcollection=None, activecollection=None, editorMode=False): """ GET - Gets the dialogues id in the database collection for the user or Gets an entire database document @@ -490,12 +516,85 @@ def handle_database_resource(id=None, user=None, mode=None, DBcollection=None, a if mode == "annotations": responseObject = DatabaseManagement.storeAnnotations( user, activecollection, values ) - else: + + elif mode == "fields": responseObject = DatabaseManagement.updateAnnotations( user, activecollection, values ) + + elif mode == "content": + DatabaseManagement.updateDoc({ + "id":activecollection}, + "dialogues_collections", + {"document."+str(values["dialogue"])+"."+str(values["turn"])+".usr":values["usr"], + "document."+str(values["dialogue"])+"."+str(values["turn"])+".sys":values["sys"]} + ) + DatabaseManagement.updateDocs({ + "id":activecollection, "fromCollection":activecollection}, + "annotated_collections", + {"document."+str(values["dialogue"])+"."+str(values["turn"])+".usr":values["usr"], + "document."+str(values["dialogue"])+"."+str(values["turn"])+".sys":values["sys"]} + ) + # update the viewed collection in the dialogue source to keep the gui in sync + if editorMode != "true": + docRetrieved = DatabaseManagement.readDatabase("annotated_collections",{"id":activecollection,"annotator":values["selected_annotator"]}) + __add_new_dialogues_from_json_dict("Su_"+user, activecollection, responseObject, dialogueDict=docRetrieved[0]["document"]) + responseObject = {"status":"success", "dialogue":docRetrieved[0]["document"][str(values["dialogue"])]} + else: + docRetrieved = DatabaseManagement.readDatabase("dialogues_collections",{"id":activecollection}) + responseObject = {"status":"success", "dialogue":docRetrieved[0]["document"][str(values["dialogue"])]} + + elif mode == "new_turn" or mode == "remove_turn": + # retrive the dialogue + collection = DatabaseManagement.readDatabase("dialogues_collections",{"id":activecollection},{"document":1, "_id":0}) + dialogue = collection[0]["document"][str(values["dialogue"])] + # add the empty turn + if mode == "new_turn": + dialogue.insert(values["turn"],{"usr":"","sys":""}) + else: + dialogue.pop(values["turn"]) + # update database + DatabaseManagement.updateDoc({ + "id":activecollection}, + "dialogues_collections", + {"document."+str(values["dialogue"]):dialogue} + ) + DatabaseManagement.updateDocs({ + "id":activecollection, "fromCollection":activecollection}, + "annotated_collections", + {"document."+str(values["dialogue"]):dialogue} + ) + + if editorMode != "true": + # update the viewed collection in the dialogue source to keep the gui in sync + docRetrieved = DatabaseManagement.readDatabase("annotated_collections",{"id":activecollection,"annotator":values["selected_annotator"]}) + if len(docRetrieved) != 0: + __add_new_dialogues_from_json_dict("Su_"+user, activecollection, responseObject, dialogueDict=docRetrieved[0]["document"]) + + responseObject = {"status":"success", "dialogue":dialogue} + + elif mode == "new_dialogue": + new_dialogue = [{"collection":activecollection,"status":"0%"},{"sys":"","usr":""}] + # update database + DatabaseManagement.updateDoc({ + "id":activecollection}, + "dialogues_collections", + {"document."+str(values["new_dialogue_name"]):new_dialogue} + ) + DatabaseManagement.updateDocs({ + "id":activecollection, "fromCollection":activecollection}, + "annotated_collections", + {"document."+str(values["new_dialogue_name"]):new_dialogue} + ) + # update the viewed collection in the dialogue source to keep the gui in sync + if editorMode != "true": + collection = DatabaseManagement.readDatabase("annotated_collections", {"id":activecollection, "annotator":values["selected_annotator"], "fromCollection":activecollection}, {"document":1, "_id":0}) + __add_new_dialogues_from_json_dict("Su_"+user, activecollection, responseObject, dialogueDict=collection[0]["document"]) + responseObject = {"status":"success", "collection":collection} + else: + collection = DatabaseManagement.readDatabase("dialogues_collections", {"id":activecollection}, {"document":1, "_id":0}) + responseObject = {"status":"success", "collection":collection} - #if request.method == "GET": - # responseObject = DatabaseManagement.getDatabaseIds() + elif id: if request.method == "GET": responseObject = DatabaseManagement.readDatabase(DBcollection,{"id":id}) @@ -592,9 +691,17 @@ def handle_switch_collection_request(user, doc): #import and format new dialogues for docCollection in docRetrieved: __add_new_dialogues_from_json_dict(user, doc, responseObject, dialogueDict=docCollection["document"]) + + annotationStyle = retrieve_annotation_style_name(doc) + if annotationStyle != False: + __add_new_dialogues_from_json_dict(user, doc, responseObject, dialogueDict=docCollection["document"]) + else: + responseObject["error"] = "The linked annotation model is not currently loaded." + return jsonify( responseObject ) - dialogueFile.change_collection(user, doc) - responseObject["status"] = "success" + if responseObject["status"] != "error": + dialogueFile.change_collection(user, doc) + responseObject["status"] = "success" return jsonify( responseObject ) @@ -1198,6 +1305,11 @@ def __add_new_dialogues_from_json_dict(user, fileName, currentResponseObject, di annotationStyle = retrieve_annotation_style_name(fileName) + if annotationStyle == False: + currentResponseObject["error"] = "The selected annotation style is no more in the database." + currentResponseObject["status"] = "error" + return currentResponseObject + for dialogue_name, dialogue in dialogueDict.items(): dialogue = Configuration.validate_dialogue(annotationStyle,dialogue) @@ -1365,10 +1477,19 @@ def retrieve_annotation_style_name(collection): annotationStyle = search[0]["annotationStyle"] #if annotation model name not valid or empty system falls back to default model - if len(annotationStyle) <= 1: + if len(annotationStyle) == 0: annotationStyle = Configuration.annotation_styles[0] - return annotationStyle + try: + if annotationStyle in Configuration.annotation_styles: + return annotationStyle + else: + logging.warning(" * The annotation style "+annotationStyle+" referenced is not loaded. Maybe it was deleted?") + return False + except: + logging.warning(" * The annotation style "+annotationStyle+" referenced is not passing the new check. Maybe it was changed since last reboot?") + return False + ################################ # STATIC METHODS @@ -1604,6 +1725,7 @@ def run_models_on_dialogue(dialogue, dontRun= tuple(["sys"])): @MatildaApp.before_request def guard(): requestedUri = request.path + #except this routes if (requestedUri != "/" and requestedUri != "/login" and requestedUri != "/favicon.ico" @@ -1618,4 +1740,4 @@ def guard(): logging.disable(logging.INFO) if __name__ == "__main__": - MatildaApp.run(port=jsonConf["port"],host=jsonConf["address"]) \ No newline at end of file + MatildaApp.run(port=jsonConf["port"],host=jsonConf["address"])