Updates to onboard_project.py

[bhartmoore]

Requesting a couple of quick updates to onboard_project.py. Let me know if you'd prefer these broken out as separate issues:

1. The latest version has lost the ability to handle multiple projects at once. It would be great to bring this back as it's a common workflow. Here I'm attempting to extract AAI and AIH at the same time:

(silnlp) root@VilleDell:~/silnlp# poetry run python -m silnlp.common.onboard_project AAI AIH --copy-from /mnt/c/Users/bmoor/Downloads --extract-corpora --collect-verse-counts --datestamp --wildebeest --stats
2026-03-24 11:04:05,758 - silnlp.common.environment - INFO - Using workspace: /root/M as per environment variable SIL_NLP_DATA_PATH.
INFO:silnlp.common.environment:Using workspace: /root/M as per environment variable SIL_NLP_DATA_PATH.
usage: onboard_project.py [-h] [--ref-projects REF_PROJECTS [REF_PROJECTS ...]]
                          [--copy-from [COPY_FROM]] [--config CONFIG] [--overwrite]
                          [--extract-corpora] [--collect-verse-counts] [--no-clean] [--datestamp]    
                          [--wildebeest] [--stats] [--align]
                          [--align-isos ALIGN_ISOS [ALIGN_ISOS ...]]
                          [--clearml-queue CLEARML_QUEUE] [--clearml-tag tag]
                          main_project
onboard_project.py: error: unrecognized arguments: AIH
(silnlp) root@VilleDell:~/silnlp# 

2. The most recent version errors on attempting to extract an empty project. It could be reasonable to just return a notification in this case instead of continuing. My reason for extracting this project was actually because I suspected that it was empty and shouldn't be activated for drafting. I can't think of a time we'd actually need to do further processing on a truly empty project. Example below:

(silnlp) root@VilleDell:~/silnlp# poetry run python -m silnlp.common.onboard_project AAI --copy-from /mnt/c/Users/bmoor/Downloads --extract-corpora --collect-verse-counts --datestamp --wildebeest --stats
2026-03-24 11:01:49,363 - silnlp.common.environment - INFO - Using workspace: /root/M as per environment variable SIL_NLP_DATA_PATH.
INFO:silnlp.common.environment:Using workspace: /root/M as per environment variable SIL_NLP_DATA_PATH.
2026-03-24 11:01:52,050 - silnlp.common.onboard_project - INFO - Preparing and Uploading main project 'AAI'
INFO:silnlp.common.onboard_project:Preparing and Uploading main project 'AAI'
2026-03-24 11:01:52,118 - silnlp.common.onboard_project - INFO - Datestamping project. New project name: AAI_2026_03_24
INFO:silnlp.common.onboard_project:Datestamping project. New project name: AAI_2026_03_24
2026-03-24 11:01:53,098 - silnlp.common.onboard_project - INFO - Cleaning Paratext project: AAI_2026_03_24.
INFO:silnlp.common.onboard_project:Cleaning Paratext project: AAI_2026_03_24.
2026-03-24 11:01:54,367 - silnlp.common.onboard_project - INFO - Copying project: AAI_2026_03_24 from /mnt/c/Users/bmoor/Downloads to /root/M/Paratext/projects/AAI_2026_03_24
INFO:silnlp.common.onboard_project:Copying project: AAI_2026_03_24 from /mnt/c/Users/bmoor/Downloads to /root/M/Paratext/projects/AAI_2026_03_24
2026-03-24 11:01:54,368 - silnlp.common.onboard_project - INFO - Paratext project folder '/root/M/Paratext/projects/AAI_2026_03_24' already exists.
INFO:silnlp.common.onboard_project:Paratext project folder '/root/M/Paratext/projects/AAI_2026_03_24' already exists.
2026-03-24 11:01:54,379 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/01GENAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/01GENAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,383 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/02EXOAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/02EXOAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,387 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/03LEVAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/03LEVAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,391 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/04NUMAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/04NUMAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,395 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/05DEUAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/05DEUAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,401 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/06JOSAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/06JOSAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,405 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/07JDGAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/07JDGAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,409 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/08RUTAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/08RUTAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,415 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/091SAAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/091SAAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,419 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/102SAAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/102SAAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,423 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/111KIAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/111KIAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,427 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/122KIAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/122KIAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,434 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/131CHAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/131CHAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,442 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/142CHAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/142CHAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,448 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/15EZRAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/15EZRAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,454 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/16NEHAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/16NEHAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,459 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/17ESTAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/17ESTAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,464 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/18JOBAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/18JOBAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,469 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/19PSAAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/19PSAAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,475 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/20PROAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/20PROAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,480 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/21ECCAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/21ECCAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,485 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/22SNGAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/22SNGAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,491 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/23ISAAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/23ISAAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,497 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/24JERAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/24JERAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,504 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/25LAMAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/25LAMAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,510 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/26EZKAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/26EZKAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,516 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/27DANAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/27DANAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,526 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/28HOSAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/28HOSAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,531 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/29JOLAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/29JOLAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,537 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/30AMOAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/30AMOAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,543 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/31OBAAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/31OBAAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,549 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/32JONAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/32JONAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,554 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/33MICAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/33MICAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,560 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/34NAMAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/34NAMAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,567 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/35HABAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/35HABAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,574 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/36ZEPAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/36ZEPAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,579 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/37HAGAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/37HAGAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,584 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/38ZECAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/38ZECAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,589 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/39MALAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/39MALAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,594 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/41MATAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/41MATAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,600 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/42MRKAAI.SFM' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/42MRKAAI.SFM' already exists. Skipping.
2026-03-24 11:01:54,607 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/anp.ldml' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/anp.ldml' already exists. Skipping.
2026-03-24 11:01:54,611 - silnlp.common.onboard_project - INFO - File '/root/M/Paratext/projects/AAI_2026_03_24/Settings.xml' already exists. Skipping.
INFO:silnlp.common.onboard_project:File '/root/M/Paratext/projects/AAI_2026_03_24/Settings.xml' already exists. Skipping.
2026-03-24 11:01:54,854 - silnlp.common.onboard_project - INFO - Processing onboarding request for main project 'AAI_2026_03_24'
INFO:silnlp.common.onboard_project:Processing onboarding request for main project 'AAI_2026_03_24'   
2026-03-24 11:01:54,855 - silnlp.common.onboard_project - INFO - Onboarding main project 'AAI_2026_03_24'
INFO:silnlp.common.onboard_project:Onboarding main project 'AAI_2026_03_24'
2026-03-24 11:01:54,936 - silnlp.common.onboard_project - INFO - Extracted corpus '/root/M/MT/scripture/anp-AAI_2026_03_24.txt' already exists. Skipping corpus extraction.
2026-03-24 11:01:54,998 - silnlp.common.onboard_project - INFO - Collecting verse counts for project 'AAI_2026_03_24'
  0%|                                                                          | 0/1 [00:00<?, ?it/s]2026-03-24 11:01:55,129 - silnlp.common.collect_verse_counts - INFO - Found verse counts for /root/M/MT/scripture/anp-AAI_2026_03_24.txt
100%|████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1390.68it/s]


2026-03-24 11:01:54,736 - silnlp.common.collect_verse_counts - INFO - No files smaller than 41KB were found.
2026-03-24 11:01:54,736 - silnlp.common.collect_verse_counts - INFO - All files were found.
2026-03-24 11:01:54,738 - silnlp.common.onboard_project - INFO - Running Wildebeest analysis on /root/M/MT/scripture/anp-AAI_2026_03_24.txt.
2026-03-24 11:01:55,601 - silnlp.common.onboard_project - INFO - Calculating tokenization stats for project 'AAI_2026_03_24'
2026-03-24 11:01:59,445 - silnlp.nmt.config - INFO - Preprocessing anp-AAI_2026_03_24 -> anp-AAI_2026_03_24
2026-03-24 11:01:59,685 - silnlp.nmt.config - INFO - train size: 0, val size: 0, test size: 0,
2026-03-24 11:01:59,686 - silnlp.nmt.config - WARNING - Glosses could not be included. No source or target language matches any of the supported gloss language codes: fr, en, id, es, pt.
2026-03-24 11:01:59,686 - silnlp.nmt.config - INFO - terms train size: 0
2026-03-24 11:01:59,686 - silnlp.nmt.config - INFO - Calculating tokenization statistics
Traceback (most recent call last):
  File "/root/miniconda3/envs/silnlp/lib/python3.10/runpy.py", line 196, in _run_module_as_main      
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/silnlp/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/silnlp/silnlp/common/onboard_project.py", line 775, in <module>
    main()
  File "/root/silnlp/silnlp/common/onboard_project.py", line 771, in main
    onboarding_request.process_onboarding_request()
  File "/root/silnlp/silnlp/common/onboard_project.py", line 439, in process_onboarding_request      
    self.main_project.calculate_tokenization_stats(
  File "/root/silnlp/silnlp/common/onboard_project.py", line 209, in calculate_tokenization_stats    
    config.preprocess(stats=True, force_align=True)
  File "/root/silnlp/silnlp/nmt/config.py", line 261, in preprocess
    self._build_corpora(tokenizer, stats, force_align)
  File "/root/silnlp/silnlp/nmt/config.py", line 326, in _build_corpora
    self._calculate_tokenization_stats()
  File "/root/silnlp/silnlp/nmt/config.py", line 406, in _calculate_tokenization_stats
    tokens_verse_df = distribution_df(top_header, src_tokens_per_verse, trg_tokens_per_verse)        
  File "/root/silnlp/silnlp/nmt/config.py", line 389, in distribution_df
    min(src_data),
ValueError: min() arg is an empty sequence

[TaperChipmunk32]

@bhartmoore For multiple projects, do you want to be able to specify multiple main projects? Because you can onboard multiple projects at once using --ref-projects to list reference projects for the main project to be onboarded. For example, if AII and AIH are related, you could run a command like poetry run python -m silnlp.common.onboard_project AAI --ref-projects AIH --copy-from /mnt/c/Users/bmoor/Downloads --extract-corpora --collect-verse-counts --datestamp --wildebeest --stats to use AIH as a reference project for AII.

The only difference in a reference project and main project is for --stats and --align, where the main project is set as the target language and the reference projects are sources, with the results being stored on the bucket with the main project.

[bhartmoore]

@TaperChipmunk32 Thanks, yes, we want to specify multiple main projects that are not related, as was the case here. We're often working a few different requests at one time. It sounds like that may not be workable with the new --ref-projects argument. It's not strictly necessary; we can run separate commands. Maybe there could be an option (if we don't set --ref--projects, for example) to extract multiple projects. The workflow here was that folks were connecting lots of new projects at the workshop in India and I just needed to check whether they were valid projects (rather than blanks created to hold drafts) before activating them.

[TaperChipmunk32]

@TaperChipmunk32 Thanks, yes, we want to specify multiple main projects that are not related, as was the case here. We're often working a few different requests at one time. It sounds like that may not be workable with the new --ref-projects argument. It's not strictly necessary; we can run separate commands. Maybe there could be an option (if we don't set --ref--projects, for example) to extract multiple projects. The workflow here was that folks were connecting lots of new projects at the workshop in India and I just needed to check whether they were valid projects (rather than blanks created to hold drafts) before activating them.

It is workable, I've made progress on getting it working, but I just wanted to make sure I was understanding it correctly. Hopefully I'll get the PR submitted today.

[TaperChipmunk32]

@bhartmoore The PR has just been merged. You should now be able to list multiple main projects in the same way it worked before.