Running four AI models fully offline on a Nextion ONX3248G035
A while back I unboxed a Nextion touchscreen dev board and vibe-coded a WiFi auditor onto it. That board has been sitting on my desk ever since, and one question kept nagging: this thing has 8 MB of PSRAM and a dual-core ESP32-S3. Could it run an actual language model? Not talk to one over WiFi — run one.
Turns out: yes. Four of them, in fact. No internet, no server, no phone, no PC. Everything in this video is generated by the chip on the board.
What is actually running
The board boots into a menu. Tap a model, it loads and starts generating. All of these were trained from scratch on an M2 Max in well under an hour.
| Malayalam | English | Arithmetic | Icons | |
|---|---|---|---|---|
| Parameters | 7.6M | 7.6M | 1.0M | 577K |
| Flash | 3.95 MB | 3.95 MB | 0.41 MB | 0.58 MB |
| Speed | 13.0 tok/s | 14.9 tok/s | 20.7 tok/s | 3.4 s/image |
First things first: this is not a Nextion HMI
The ONX3248G035 is sold under Nextion branding, and the name is misleading in a way that matters. There is no serial HMI protocol here, no .tft project, nothing to upload over UART. It is a bare ESP32-S3 with an ST7796 SPI panel wired straight to it, a CST826 touch controller on I2C, and a PCF8574 expander holding the LCD reset and the SD chip-select. You drive the panel yourself.
Which is better, and is exactly why it can run any of this. Worth flagging all the same, because the branding implies the opposite: if you buy one expecting to send it HMI commands over serial, nothing is listening.
The good news is that you do not have to work any of it out. The board ships with vendor reference code, and it names everything — the panel init sequence, the SPI pins, the I2C addresses, which expander bit holds the LCD reset, which GPIO drives the backlight. Start there and the wiring is a five-minute read. Start anywhere else and you will spend a day rediscovering it, as I can report from experience.
One more trap: the console is a CH340K on UART0, not the S3 native USB. Build with CDCOnBoot=cdc and the board runs perfectly and prints absolutely nothing, which looks identical to a boot loop.
How do you fit a language model in 512 KB?
You do not. An ESP32-S3 has 512 KB of SRAM and normally the whole model must be reachable from there, which caps you at toys.
The trick is Per-Layer Embeddings, Google’s design from the Gemma models. Most of the parameters go into a table the model reads from rather than computes on, so they can live in slow flash and only a few rows get pulled per token:
SRAM (fast, tiny) the thinking core, used on every token PSRAM (medium) the output head and working memory FLASH (huge, slow) the parameter table, a few rows per token
The consequence is worth stating plainly: the table buys vocabulary and knowledge, not intelligence. The reasoning lives in a core that is 558K parameters by design, and no amount of flash changes that.
Teaching it Malayalam
The original project writes English. I wanted my mother tongue on there, and that turned into the most interesting engineering of the whole build.
The tokenizer bug that breaks every Indian language
The stock GPT-2 pre-tokenizer splits words using the rule \p{L}+ — “one or more letters”. In Malayalam, Devanagari, Tamil and Bengali, vowel signs and the virama are combining marks, not letters. So that rule chops every single word apart at each mark, and the tokenizer can never learn a real subword unit.
| Pre-tokenizer | Merges learned | Chars/token |
|---|---|---|
stock \p{L}+ | saturates at 20,800 | 1.37 |
mark-aware [\p{L}\p{M}]+ | full 32,768 | 5.06 |
A phrase like കുട്ടികൾക്ക് വീട്ടിലേക്ക് പോയി goes from 26 pieces down to 3. And here is the nasty part: round-trip decoding is byte-exact either way. Nothing crashes. Nothing looks wrong. It shows up only as a model that refuses to learn.
Fixed, Malayalam actually compresses better than English — 5.06 against 4.25 chars per token — because agglutinative morphology gives BPE longer units to grab.
Rendering the script without a font engine
An MCU graphics library draws one codepoint at a time, left to right. That is simply wrong for Malayalam, where the written unit is a grapheme cluster whose vowel signs can sit above, below, or before the consonant, and where consonants fuse into conjuncts. Doing that properly needs text shaping, which is not something you run on an ESP32.
So I moved the shaping to the host. HarfBuzz shapes each cluster with a real font, FreeType rasterises it, and only finished 1-bit bitmaps go into flash. The device just matches bytes and blits. 4,149 clusters cover 100% of the corpus in 649 KB.
Four bugs there all had the same signature — a silently deleted or wrong letter, never a crash. My favourite: a Unicode extended grapheme cluster breaks after the virama, so conjuncts came out decomposed with a stray chandrakkala. The bytes were perfectly correct; the writing was not.
The corpus decided everything
First attempt was a 300 MB general web crawl. Result: perplexity 196.6, greedy decoding collapsing into a repetition loop, and sampling that drifted into adult content that was sitting in the crawl. Unusable.
A simplicity-constrained story corpus took it to 85.1. Topping it up with 40k machine-translated TinyStories got 21.1, and longer training landed at 17.57. For a 558K core the corpus has to be simple by construction — a web crawl is the exact opposite.
The English model
The upstream 28.9M-parameter model reproduces almost exactly on my tree — they report perplexity 11.41 and 102.9 ms/step, I measure 11.40 and 102.5 ms. That match is what made every later comparison trustworthy.
It is 14.91 MB though, which fills the model partition on its own. So the English model in the menu is retrained at a smaller vocabulary. That is not a downgrade: 10.20 perplexity against 11.40, at 14.9 tok/s against 9.47.
Can a 558K core actually reason?
Stories are impossible to score. So I built something with a right answer: chain-of-thought addition, where the model writes out every carry step, graded by exact string match.
- 100% on held-out operand pairs at lengths it was trained on, every carry correct.
- 0% at lengths it never saw.
It executes a learned algorithm perfectly and cannot generalise its control flow. That is a useful negative result — it means scaling the core is not the lever for that particular failure.
And the first version of this benchmark was measuring my own bug, not the model. Accuracy read 86.5% / 0% / 0%, which looked exactly like a capacity ceiling. It was not: operand lengths were sampled independently and then de-duplicated, leaving 95% of the corpus 3-digit. Balanced, it went flat to 100%. A result that confirms your hypothesis is the one to re-check hardest.
Making it draw
The fourth model is not a language model at all. It draws a 16x16 icon straight onto the panel, one pixel at a time, and it took three failed attempts to get there. Anime faces gave colour smears. Pixel sprites gave vague creatures. Both got deleted.
Along the way the sprite corpus taught me something I now check for reflexively: it was 81% background, which let a model score a better perplexity than the final working model by predicting “background” for all 1024 pixels and generating a blank white frame. No loss number can tell you that.
The bug I blamed on the architecture
What finally worked was 16x16 icons with a 4-colour palette. But my first icons still came out looking like texture rather than objects, and I was fairly sure the architecture was at fault.
It was not. The tight crop that had rescued the sprites was ruining the icons: it resizes a non-square glyph to fill a square tile, so a narrow glyph became an edge-to-edge slab with its proportions destroyed and no margin left. Measured on the tile border, only 48.4% of border pixels were blank — more than half of all icons had ink running off the frame.
The model was being taught that icons have no border, and it learned that faithfully. The same fix that saves one dataset can silently destroy another.
Then I changed the architecture anyway
Here is the thing about generating images with a transformer: in raster order, a pixel and the pixel directly above it are 16 positions apart. The model has to learn vertical adjacency through 1-D attention. That is the wrong prior for a picture.
A masked convolution has it built in. So I wrote a second inference runtime — a PixelCNN — and trained both at a matched parameter budget on the identical corpus. Then I scored them on structure rather than loss:
| Metric | Real icons | PixelCNN | Transformer |
|---|---|---|---|
| Blank tile border | 1.000 | 0.987 | 0.749 |
| Isolated ink pixels | 0.000 | 0.000 | 0.010 |
| Connected regions | 1.289 | 1.266 | 3.453 |
| Off-palette pixels | 0.000 | 0.000 | 0.004 |
| Val perplexity | — | 1.476 | 1.35 |
The PixelCNN matches the real corpus on every structural metric. The transformer misses all of them, drawing 2.7x too many disconnected regions — meaning its strokes never close.
And the transformer wins on perplexity. Loss ranked the two models backwards. Teacher-forced perplexity measures how well a model predicts the next pixel given a true prefix; sampling asks it to survive 256 steps of its own mistakes. For images those two things come apart completely.
And it is 3.7x faster
This is the counter-intuitive bit. Naively a PixelCNN re-runs the entire network for every pixel — 37.6 GMAC per image, minutes of work. But every convolution is causal, which means an activation is final once computed: no later pixel can ever change it.
So generation never re-runs the stack. It walks the layers at one position, and a whole image costs what a single forward pass costs — 147 MMAC instead of 37.6 GMAC. Nothing grows with position the way a KV cache does, and the working set collapses to 80 KB that fits in internal SRAM.
On the board: 3.42 seconds per image, against 12.8 seconds for the transformer. Faster and better.
The thread running through all of it
Nearly every real bug in this project was silent. Not a crash, not a stack trace — a number that looked perfectly reasonable, or output that looked plausible:
| Symptom | Actual cause |
|---|---|
| Model will not learn Malayalam | Tokenizer splitting at combining marks |
| Conjuncts render with stray marks | Grapheme clusters break after the virama |
| Words missing letters | Buffer length 16 vs a real 21 bytes |
| Reasoning ceiling at 3 digits | Corpus 95% 3-digit after dedup |
| Great perplexity, blank images | Corpus 81% background |
| Icons look like texture | Crop stretched away the margins |
| Better perplexity, worse pictures | Teacher forcing is not sampling |
| Right-sized file, divide by zero | DMA cannot read memory-mapped flash |
| Board runs, prints nothing | Wrong USB console setting for a CH340K |
The habit that caught every one of them was the same: pick a metric that can actually fail, and look at the artifact instead of the loss. The icon scorer deliberately reports no perplexity at all. The arithmetic grader scores exact strings rather than judging. And where a check could not be automated, I rendered the output and looked at it — which is the only reason the crop bug ever surfaced.
മലയാളത്തിൽ — ഇത് എന്താണ്?
ഇന്റർനെറ്റോ സെർവറോ ഫോണോ കമ്പ്യൂട്ടറോ ഇല്ലാതെ, കൈവെള്ളയിൽ ഒതുങ്ങുന്ന ഈ കുഞ്ഞു ബോർഡ് സ്വയം മലയാളത്തിൽ കഥകൾ എഴുതുന്നു — സെക്കൻഡിൽ ഏകദേശം പത്ത് വാക്ക്.
വീഡിയോയിൽ കാണുന്നത്:
- മലയാളം കഥ എഴുതുന്നു
- ഇംഗ്ലീഷ് കഥ എഴുതുന്നു
- കൂട്ടൽ കണക്ക് — ഓരോ ഘട്ടവും കാണിച്ചുകൊണ്ട്
- 16x16 ഐക്കൺ ചിത്രം സ്വയം വരയ്ക്കുന്നു
എങ്ങനെ സാധിക്കുന്നു? സാധാരണ AI മോഡലുകൾക്ക് വലിയ മെമ്മറി വേണം; ഈ ചെറിയ ചിപ്പിന് അതില്ല. അതുകൊണ്ട് മോഡലിന്റെ വലിയ ഭാഗം വേഗം കുറഞ്ഞ സ്റ്റോറേജിൽ സൂക്ഷിക്കുന്നു, ഓരോ വാക്ക് എഴുതുമ്പോഴും ആവശ്യമുള്ള കുറച്ച് ഭാഗം മാത്രം വായിക്കുന്നു. അങ്ങനെ വലിയ മോഡൽ ചെറിയ ചിപ്പിൽ ഒതുങ്ങുന്നു.
ഏറ്റവും ബുദ്ധിമുട്ടിയത് എന്താണ്? മലയാളം അക്ഷരങ്ങൾ ശരിയായി എഴുതിക്കാണിക്കുന്നത്. കൂട്ടക്ഷരങ്ങളും സ്വരചിഹ്നങ്ങളും ശരിയായ രീതിയിൽ ചേർത്ത് വരയ്ക്കാൻ സാധാരണ ചെറിയ ചിപ്പുകൾക്ക് കഴിയില്ല. അതിനാൽ ഓരോ അക്ഷരക്കൂട്ടത്തിന്റെയും ചിത്രം കമ്പ്യൂട്ടറിൽ മുൻകൂട്ടി തയ്യാറാക്കി ബോർഡിൽ സൂക്ഷിച്ചു.
ഇത് എന്ത് ചെയ്യില്ല? ഇത് ChatGPT അല്ല. ചോദ്യങ്ങൾക്ക് ഉത്തരം പറയില്ല, വിവരങ്ങൾ അറിയില്ല. ലളിതമായ കൊച്ചു കഥകൾ എഴുതാൻ മാത്രം പഠിച്ച ഒരു കുഞ്ഞു മോഡലാണ്. പക്ഷേ അത് പൂർണ്ണമായും ആ ബോർഡിനുള്ളിൽ തന്നെ നടക്കുന്നു — അതാണ് ഇതിന്റെ പ്രത്യേകത.
Everything is on GitHub
Training, quantization, the firmware, the Malayalam pipeline, the icon model — all of it, along with a long write-up of what went wrong at each step:
🔗 github.com/moheshmohan/esp32-ai
The weights are not in the repo, but every training command is, and the whole set retrains in under an hour on a laptop.
This is a fork of slvDev/esp32-ai, whose Per-Layer-Embedding architecture, training code and portable C runtime made all of this possible. Full credit there — the port to this board, the Malayalam work, the reasoning benchmark and the image model are what I added on top.
Comments
Post a Comment