Implement WikiText-103 processed dataset publication utility (#40)

[nsrawat0333]

Summary

Resolves Issue #40 by implementing a comprehensive solution to publish and distribute processed WikiText-103 dataset, addressing a 4+ year old community request from @cp-pc.

Problem

In 2020, @cp-pc requested: "Will it be convenient to publish the processed WikiText103 data set"

The issue remained unresolved because:

• No automated way to obtain processed WikiText-103 dataset
• Manual download and processing was complex and error-prone
• No standardized vocabulary or validation
• Researchers had to recreate processing pipelines individually

Solution

Created a one-command solution that:

• ✅ Downloads raw WikiText-103 dataset automatically
• ✅ Processes text into tokenized format with vocabulary
• ✅ Validates dataset integrity against published statistics
• ✅ Packages everything for immediate research use

Usage

# One command gets everything!
python scripts/create_processed_wikitext103_dataset.py \
    --output_dir /tmp/data \
    --operation all

# Results in ready-to-use dataset:
# /tmp/data/wikitext-103-processed/
# ├── train.txt          (103M tokens)
# ├── valid.txt          (218K tokens) 
# ├── test.txt           (246K tokens)
# ├── vocab.csv          (267K vocabulary)
# └── dataset_info.json  (statistics)

[polarbe]

Dooray! Failure Notice Failure Notice Your message sent to ***@***.***) has failed to be delivered. Please refer to the below for details. * Recipient :  ***@***.***) * Sent time :  2025-08-11T06:17:45 * Subject :  [google-deepmind/deepmind-research] Implement WikiText-103 processed dataset publication utility (#40) (PR #648) * Remote host said :  Your mail was denied from the receiver. This message was sent from a notification-only address that cannot accept incoming email. For more information, please contact ***@***.*** © Dooray!.