data Details

[MrRace]

Could you please provide additional sample data for the following data, as well as how these data were generated?

DATASET_PATH = [
    ("/mnt/e/ai/dataset/pt/zh", 20 * 10000),
    ("/mnt/e/ai/dataset/pt/zh_r18_pixiv", 20 * 10000),
    ("/mnt/e/ai/dataset/pt/en", 30 * 10000),
    ("/mnt/e/ai/dataset/pt/en_r18_visual_novels", 10 * 10000),
    ("/mnt/e/ai/dataset/pt/ja", 40 * 10000),
    ("/mnt/e/ai/dataset/pt/ja_r18", 32.5 * 10000),
    ("/mnt/e/ai/dataset/pt/ja_r18_rpg", 7.5 * 10000),
    ("/mnt/e/ai/dataset/pt/ko", 20 * 10000),
    ("/mnt/e/ai/dataset/pt/ko_web", 20 * 10000),
    # ("/mnt/e/ai/dataset/pt/zh_cc100", 800 * 10000),
    # ("/mnt/e/ai/dataset/pt/zh_cc100_tw", 400 * 10000),
    # ("/mnt/e/ai/dataset/pt/en_cc100", 800 * 10000),
    # ("/mnt/e/ai/dataset/pt/ja_cc100_izumi_lab", 1200 * 10000),
    # ("/mnt/e/ai/dataset/pt/ko_cc100", 800 * 10000),
]

[neavo]

It's just plain text, one sentence in one line:
text 1
text 2
... ...

[MrRace]

It's just plain text, one sentence in one line: text 1 text 2 ... ...

@neavo Thank you for your prompt reply!
Do you have scripts for generating these TXT datasets? For example, convert an article(maybe from Wikipedia) into a pure text format suitable for pre-training? Additionally, it seems that the script 00_pt.py does not involve dictionary expansion. It appears to only perform continued pre-training based on the modern_bert_multilingual_nodecay model to obtain keyword_gacha_base_multilingual. This process does not expand the dictionary of the initial model.

[neavo]

It's just plain text, one sentence in one line: text 1 text 2 ... ...

@neavo Thank you for your prompt reply! Do you have scripts for generating these TXT datasets? For example, convert an article(maybe from Wikipedia) into a pure text format suitable for pre-training? Additionally, it seems that the script 00_pt.py does not involve dictionary expansion. It appears to only perform continued pre-training based on the modern_bert_multilingual_nodecay model to obtain keyword_gacha_base_multilingual. This process does not expand the dictionary of the initial model.

• I directly used datasets that others have already pre-cleaned, for example: cc100
• If you want to start training from raw, uncleaned data, then I think you might need to perform many data cleaning steps to ensure data quality.

• 02_tokenizer.py is a script for training your own tokenizer from scratch.
  However, because I ultimately chose to directly use the vocabulary of Qwen2.5, I haven't actually used this script.You can refer to the logic within it.
• Once you have generated your own tokenizer (whether trained from scratch or modified from another tokenizer), then overwrite the original tokenizer file, and then adjust the values of fields like vocab_size and *_bos_token_id in config.json. You have essentially completed the expansion (or replacement) of the vocabulary.
• Indeed, 00_pt.py in the repo, as you described, is for continued pre-training.
  You can modify the SCRATCH = True parameter at the beginning to train from scratch.
  Because this is a script I used myself, it has a lot of hardcoded logic and may not be very suitable for your needs, but you can still refer to its logic.

Feel free to ask if you have more questions! :)