training dataset link?

[sleepingcat4]

where can I find the training corpus uploaded? I read the paper and it seems pre-training corpus was a mixture of DCLM-Baseline and SlimPajama from Cerebras. Unfortunately that does not provide the full picture since recipe can vary.

is there any robust source of where can the pre-training corpus be found? I need it to use this model on my paper.

[sleepingcat4]

@piuzha bumping!

[piuzha]

Thanks, we are working on this and will release the dataset soon.

[sleepingcat4]

@piuzha thanks! but I actually need to know the datasets that were used in what proportion in the pre-training process since there's a time sensitive deadline for my paper.

Moxin beat other models at the same par on benchmarks that's why we dropped our previous model choice. But if we don't know what's the actual proportion and what pre-training corpus was used we have to drop Moxin from our list.

Can you at least specify, what datasets were used and how much of them were in the pre-training? Like if DCLM-Baseline was used then what version and whether entire or portion of it was used for pre-training? and same goes for its other datasets.

[piuzha]

We use the whole slimpajama and about1.4T tokens from DCLM-baseline-1.0.

[sleepingcat4]

@piuzha can you exactly tell me what part of DCLM-Baseline is being used? Cuz otherwise I have to scan through the entire-corpus.

[piuzha]

We use the DCLM data processing pipeline to further process the DCLM-baseline and filter/get high quality data. It is hard to tell exactly which part of the dataset is being used in words.