Skip to content

Commit 5d8c7a3

Browse files
fabianpOptaxDev
authored and
OptaxDev
committed
[DOC] Add to the gallery an example on a small language model
This example demonstrates how to train a small-scale transformer-based language model (inspired by NanoGPT) on the Tiny Shakespeare dataset. The core idea is to train a model that can predict the next character in a sequence of text based on the characters that came before it. PiperOrigin-RevId: 618788255
1 parent b555b4f commit 5d8c7a3

File tree

4 files changed

+752
-0
lines changed

4 files changed

+752
-0
lines changed

docs/conf.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,7 @@ def new_process_docstring(app, what, name, obj, options, lines):
227227
nb_execution_allow_errors = False
228228
nb_execution_excludepatterns = [
229229
# slow examples
230+
'nanolm.ipynb',
230231
'cifar10_resnet.ipynb',
231232
'adversarial_training.ipynb',
232233
'reduce_on_plateau.ipynb',
@@ -296,6 +297,7 @@ def linkcode_resolve(domain, info):
296297

297298
intersphinx_mapping = {
298299
'jax': ('https://jax.readthedocs.io/en/latest/', None),
300+
'flax': ('https://flax.readthedocs.io/en/latest/', None),
299301
}
300302

301303
source_suffix = ['.rst', '.md', '.ipynb']

docs/gallery.rst

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,23 @@
150150
<div class="sphx-glr-thumbnail-title">Adversarial training of CNN on MNIST.</div>
151151
</div>
152152

153+
154+
.. raw:: html
155+
156+
<div class="sphx-glr-thumbcontainer" tooltip="Character-level Transformer on Tiny Shakespeare.">
157+
158+
.. only:: html
159+
160+
.. image:: /images/examples/tiny_shakespeare.png
161+
:alt: Character-level Transformer on Tiny Shakespeare
162+
163+
:doc:`_collections/examples/nanolm`
164+
165+
.. raw:: html
166+
167+
<div class="sphx-glr-thumbnail-title">Character-level Transformer on Tiny Shakespeare.</div>
168+
</div>
169+
153170
.. raw:: html
154171

155172
</div>
111 KB
Loading

0 commit comments

Comments
 (0)