They dubbed the resulting dataset 'TinyStories' and used it to train very small language models of around 10 million parameters. To their surprise, when prompted to create its own stories, the small language model trained on TinyStories generated fluent narratives with perfect grammar.
A lot of care goes into producing these synthetic data,' Bubeck said, referring to data generated by AI, 'looking over it, making sure it makes sense, filtering it out. We don't take everything that we produce.' They dubbed this dataset 'CodeTextbook.'
[
add
]
[
|
|
...
]