NOVA

NOVA is a small creature that lives on your desk. It has eyes, expressions, a repertoire of wordless sounds of its own, a name, a memory of what has happened between the two of you, and it wakes up when you call it by name. No wheels, no motors, and since 18 September no conversation either: it isn’t a robot or a voice assistant, it’s an object with a face that reacts to what happens around it.

It’s also my first hardware project, and that was the main goal before anything else.

Why it exists

Three goals, in order:

1. Learn embedded. First hardware project.
2. Build something that can’t be cloned. A character, not a commodity. Hardware doesn’t differentiate, because anyone can order the same parts; the character and the finish are the only moat.
3. Try to go viral. Sell ten units with a waiting list and grow from there.

The cost is already spent and doesn’t depend on a video working. What was really bought is learning embedded by building something I actually want.

The video everything hangs from

The original script had four lines:

You: “I built a ChatGPT pet.”
The creature: “You haven’t talked to me in 3 days.”
You: “What?”
The creature: “I know.”

What matters about that script is that the three-days line is not a language-model response. It’s a timer. The whole architecture comes from there.

Since NOVA stopped talking (D16), the script uses that decision in its favour: it’s a thing that doesn’t talk, until it does.

You: “I built a desk pet. It doesn’t talk.”
(NOVA gurgles, blinks, looks at you.)
You: “See? It just makes sounds.”
The creature: “You haven’t talked to me in three days.”
You: “What?”
The creature: “I know.”

Those two lines are two audio files and a timer. What falls is the headline: “I built a ChatGPT pet” no longer describes the product and there’s no settled replacement. The candidate, since the activities exist (D18), is the pet that works alongside you.

Where it sits

The AI companion market exists and is bigger than it looks, but it lives in the premium range:

NOVA sits at €69–99, a gap almost nobody occupies. And there’s one fact in that table that weighed on the project’s biggest decision: Moflin doesn’t say a single word.

Decisions, with their why

Each one is written down so it doesn’t get re-argued in three weeks. Several have been revised by later decisions; they keep their original reasoning and a note on what changed.

D1. NOVA over the other 33 ideas

The spreadsheet with the 34 candidates had a scale bug: the margin column ran from 0 to 100 and everything else from 0 to 10. Result: margin carried a real 47% of the total score instead of the declared 10%, and the ranking was sorted by gross margin. The second column had the opposite problem, with technical fit weighing 78%.

With the scale fixed, NOVA climbs from 13th to 8th on market and to 2nd on first-project criteria. It’s the only thing on the list scoring 10/10/10 on differentiation, virality and technical fit at once; the sheet penalised it for margin and capital, which are irrelevant when you sell ten units to validate.

A signal that came out along the way: several ideas in the top 10 had an Etsy category search as their reference. When your benchmark is a marketplace page, the product is already a commodity.

D2. Integrated board, not loose components

Buy a board that already has everything and write only firmware. In favour: working the first weekend instead of three weeks fighting I2S on a breadboard. Cost: fewer free GPIOs and less control over the layout. When to change: from unit ten, with a custom PCB.

D3. No battery in v0

The ESP32-S3 with WiFi active draws 150–250 mA; 1,000 mAh gives four or five hours, that is, an object that dies just when you need it. It’s a desk object: it lives plugged in. It also avoids the batteries directive and the LiPo risk. The board bought has battery management and a connector, so the option stays open.

D5. v0 has no voice

The decision that removed the most risk. v0 had eyes, expressions, three buttons, the state machine with decay and the proactive messages as text on screen. No speech recognition, no language model, no synthesis. Zero inference cost while validating, zero audio-latency problem and, above all, the genuinely hard part, the character and the timing, solved before adding complexity.

Revoked in practice by D14: the reasoning still holds, but that private v0 was never built.

D6. Each user brings their own API key

The cost that kills isn’t the bill of materials, it’s inference: ten minutes of conversation a day is €2–6 a month per live unit. With a one-off sale at €89 you have about eighteen months of runway and then you lose money on every active customer. This has killed half the AI gadgets of recent years. The user’s own key means zero recurring cost and zero responsibility over data.

Revised by D16: the device no longer does inference, so there’s no key to ask for in order for NOVA to work. The recurring cost wasn’t shifted to the user: it disappeared. The key stays, at most, as an app option for anyone who wants text features with a model.

D7. State machine, not language model

The most important architecture decision in the project. “An LLM with memory” produces a friendly chatbot. “A Tamagotchi that can talk” produces a creature. The state lives outside the model; the model only verbalises it, and sometimes not even that. D16 is the natural conclusion of this decision: it removes the verbaliser and leaves the heart intact.

D8. Time is injected

The state machine doesn’t depend on the wall clock. It receives its time source, and in tests I control it: six months of neglect are simulated in a unit test, with no board. The only case that needs real time is the final rehearsal before filming, to verify the device survives three days powered on with WiFi reconnection and RTC without hanging. Since D18 the same rule applies to the activities that run on the board.

D9. No camera

Three reasons, in order of weight:

1. Physical. A parallel DVP camera needs 10–12 pins. There are three free GPIOs and no camera connector. It’s not hard: it doesn’t fit.
2. Product. The moment NOVA has a real eye it stops being an endearing object and becomes something people cover with a sticker.
3. Cost. Vision tokens, and GDPR.

The substitute is presence via millimetre-wave radar and sound direction via the microphone array (see perception).

D10. The personality is prototyped in the browser

Blinking, saccades, breathing, gestures, springs and timing personalities are developed on a 466×466 HTML canvas, the same size and shape as the real display. The save-refresh cycle versus compile-flash-40 s is ten times faster. What gets ported afterwards splits in two: the data (presets, constants, gesture sequences) is exported from the bench; the drawing is rewritten from canvas to LVGL with the same logic.

The bench isn’t part of the product: it’s the sketchbook. Since 19 September it’s also the gesture and sound bench, and since the 20th it exports the C header the firmware compiles itself, so the two copies can’t drift apart again.

D11. Java and Spring Boot for the backend

Over Python and Express. Where I’m strongest, and in a project with no salary friction is what kills. Besides, java.time.Clock is literally the injectable time source D8 asks for: the class exists for this. The domain stays dependency-free, so it survives framework version changes.

Spring Boot 4, not 3: Jakarta EE 11, Java 17 or later and everything deprecated in 3.x removed. Practical consequence: the 2024 tutorials don’t compile.

D12. It wakes up by voice, not buttons

The product wakes up when you say its name. No button in the experience the world sees.

A wake word like “Alexa” comes pre-trained; “NOVA” exists in no catalogue. Espressif’s official path (WakeNet) requires, for a custom word, a corpus recorded by more than 500 people, including at least 100 children, in a soundproof room: company scale. The path chosen is microWakeWord, from the Home Assistant ecosystem: it runs entirely on the device, nothing is sent anywhere until the word is detected, and it’s trained on synthetic audio generated by TTS. The board is designed for this: its datasheet describes the two-microphone array explicitly for voice wake-up.

Still in force after D16, with one change in what happens after the word: it no longer opens a conversation, it triggers a command from a closed list. That makes recognition easier, not harder: instead of transcribing any sentence, it has to tell a handful of orders apart.

D13. Presence is deduplicated; buttons aren’t

When wiring the WebSocket, two consecutive presence: false messages generated two identical rows in the history. The state didn’t break, but the history did. The distinction was fixed in code:

• Presence is a state signal. “Someone is still there” repeated a hundred times isn’t new information. It’s only persisted when the value changes.
• A button is a discrete action. Pressing ❤️ twice is two facts; the second counts less (diminishing returns), but it counts. Never deduplicated.

It matters right away because the presence radar can flicker at the edge of its range and would generate hundreds of junk rows a minute.

D14. Skip the private v0 with buttons

Decided on 15 September, with the WiFi freshly written. It revokes the operational part of D5: the reasoning of validating cheaply before investing in voice still holds in the abstract, but the risk is accepted and it isn’t executed. The PWR and BOOT buttons aren’t read, no push-buttons get soldered, there’s no prior two-week cohabitation.

What’s lost, said explicitly: the cheap test of “does the character hook?”. If something doesn’t work later, there’ll be no way to know whether it’s the personality or the audio, because both variables arrive at once. Accepted knowingly. What doesn’t change: EmotionInput, press() and diminishing returns are still needed; only who calls them changes.

D15. Voice: what was tried and what was learned

On 17 September the voice phase opened with a whole session of local TTS. XTTS-v2 is voice cloning, not character voice: with a recording of mine as reference, NOVA tried to sound like me. Kokoro and Piper are blazing fast but no Spanish voice fit. Chatterbox Multilingual V3 did convince on quality, and took about 136 seconds to generate a short sentence on an M3 Pro.

The finding wasn’t about speed but category: Chatterbox is autoregressive, and that’s both what gives it the expressiveness and what makes it slow. Piper and Kokoro are another family, non-autoregressive, which is why they run two orders of magnitude faster and sound flatter. No setting closes that gap. And a product that needs an M3 Pro to run slowly has no answer for where that runs with ten live units.

The hybrid route that came out of it (pre-generated proactive lines, conversation via API) was only half executed, and not along the planned path. The voice that truly convinced came from ElevenLabs, and its usage policy forbids using its audio as input to another model: cloning into a local TTS was closed by contract, not by technique. Hume Octave directs intonation better than anything, but it’s trained for realistic human speech: you ask for a robot and get a person playing a robot. In the end, the 25 proactive lines were generated with ElevenLabs plus an effect of my own in Python (pyworld and pedalboard) to give them a small-robot timbre: pitch at 280 Hz, formants ×1.28, flattened intonation. The voice ended up as a seven-number preset over a base voice, which is exactly what would make “choose your NOVA’s voice” viable in an app.

The other half, free conversation via API, was revoked by D16 without a line being written.

D16. NOVA doesn’t talk

The biggest design change since D7, decided on 18 September. NOVA stops being a creature that converses and becomes a creature that gestures, makes sounds and obeys specific commands. The long stuff (conversation, questions, text) moves to a web app, and later a mobile one.

The problem in the voice session wasn’t technical. It was a design contradiction: a voice that sounds human forces you to judge it as human speech. Any odd accent, any emphasis out of place, and the illusion collapses. A non-verbal sound has no such bar: nobody compares a beep to anything. It’s why R2-D2 and WALL-E convey more than most talking robots, and why Moflin, which doesn’t say a word, sold out.

Three consequences that reinforce the decision:

1. Sounds have no language. 26 audio files were about to be duplicated into English, with French and German behind them.
2. The recurring cost and the API-key friction disappear, which was the product’s real ceiling: 95% of buyers won’t go and get a key.
3. It’s the natural conclusion of D7. Remove the verbaliser and leave the heart intact.

The voice isn’t removed: it becomes scarce. NOVA gurgles for weeks without saying a word, and one day, after three days of silence, it says a single sentence. The contrast is the effect: an object that never talks and suddenly talks gets remembered. Three or four of the 25 generated lines survive: “You haven’t talked to me in three days”, “I know”, and at most one or two more, to be decided by living with it. The rest are archived, regenerable in minutes.

What falls off the plan: bidirectional audio streaming with Opus, STT and TTS via API, the sub-1.2 s latency target, the always-open microphone and, with it, a good part of the GDPR problem. What stays intact: the whole state machine, the eyes, the WebSocket, the history and the on-device wake word.

The risk accepted: the moat moves. It’s no longer the text of the lines, it’s how it sounds and how it moves. That’s sound design and animation, less familiar ground than writing lines.

D17. It babbles: the voice is a creature’s, not a person’s

Decided on 19 September, in the bench. D16 removed speech; D17 says what replaces it. NOVA babbles: a synthetic small-creature voice, a high sawtooth through two formants, chopped into syllables, with vibrato and each syllable slightly out of tune, that has intonation but no language. What the Minions in Despicable Me do: you understand everything and it says nothing.

It doesn’t have D16’s bar, because babble isn’t compared to anything, but unlike a beep it sounds like someone. The first test in the bench ended with a request to put it in more gestures. And it’s arithmetic: one oscillator, two second-order filters, white noise and envelopes. It fits on the ESP32-S3 without a single audio file, which was proven on the board on 20 September.

The rules that come out of it:

1. Beep is interface, babble is creature, and they mix on purpose. Only three gestures remain beeps: the wake-word acknowledgement, the tap on the table and the countdown. They’re NOVA as a device, and they should sound different from NOVA as a critter.
2. The mouth isn’t part of the identity. NOVA is still two eyes; in every preset the mouth is at zero. It appears only when a gesture asks for it (yawn, sneeze, speaking, petting), because there the eyes alone couldn’t tell a yawn from a startle.
3. The whole face moves. Three body channels on the same spring as everything else. Nodding bounces, shaking sways, thinking tilts, “I’m listening” cocks the head like a dog. It’s what reads from the other side of the desk, without looking at the eyelids.
4. Asleep is derived, not stored: eyelids closed and no gesture running. With that, the “z”s per breathing cycle come out on their own and the automatic micro-gestures get blocked.

D18. Activities run on the board

Decided on 19 September when defining the useful features: timer, pomodoro-style focus, alarm, guided breathing, “what’s playing?” and “yes or no?”. Three decisions in one:

1. A new layer is needed. A timer isn’t an emotion or a gesture: it’s a timed state that lives on top of the creature without replacing it. NOVA is still bored or happy while it counts; the activity lends it the pose, the gaze and the rhythm while it lasts. And it runs on the board, with an injected clock: an alarm that depends on WiFi isn’t an alarm. The backend records that it happened; the board rules over time.
2. Input is fixed word plus touch plus app, never transcription. Alarm and timer need numbers, and that’s open vocabulary; microWakeWord recognises fixed words. So: one-word commands with a default value (“timer” is ten minutes), adjustment on the screen itself (swipe ±5 min, tap to confirm) and the app for the fine stuff.
3. “What’s playing?” yes, with AudD, and with rules. It’s the only feature that reintroduces audio leaving the device and a paid service. Accepted because it costs $0.005 a query, with the key in the backend and a monthly cap per device (200 queries is a dollar). Audio only after the explicit command, ten seconds, not stored, and the listening gesture is the visible notice that it’s recording. And before identifying, reacting: beat detection on the board, no cloud, so it bobs to the rhythm.

What comes out of this for the round screen: a progress ring around the edge of the disc, shared by timer, pomodoro, alarm and breathing, plus the digits under the eyes. The creature stays inside the ring, working. Set aside for now: calendar (OAuth, friction) and presence-based breaks (they need the radar).

Hardware

Waveshare ESP32-S3-Touch-AMOLED-1.75-B, bought on 3 September 2026 for €44.93 shipped. What matters from the datasheet:

On arrival, the points noted as doubts were checked: the 8 Ω 2 W speaker comes in the box with a plug-in connector, the eight-pin header comes soldered (so no soldering iron needed to start) and the USB-C cable had data lines, because /dev/tty.usbmodem1101 showed up first time. On 20 September the speaker sounded for the first time, over I2S, with the synthesis running on the chip.

The main constraint is that there are only three free GPIOs (16, 17 and 18 on the header) and one UART. Since D12 and D14 it weighs less: with no buttons in the product or the prototype, the pin budget is left for the presence radar, which goes over that UART. Escape routes if more were needed: the I2C expander the board carries, the PWR and BOOT buttons, and the touch screen itself. That’s the price of the integrated board: I/O freedom in exchange for speed.

Explicitly deferred to the product phase: a custom printed case (the board comes with a protective one), battery, the presence radar (good idea, bad purchase right now) and the camera (ruled out).

Perception without a camera

Mouse-following in the eye bench stands in for a single thing: “someone is there”. On the board that’s covered by four sensors, three of them already soldered: the touch screen (it looks where you touch it), the IMU (it knows when you pick it up or tap the table) and the microphone array. The reactions to all three are already designed as gestures; the IMU and touch drivers are work for the alarm stage.

Sound direction

Reacting to sound is what animals do; for a creature it’s more powerful than following a face. The right component is ESP-APA-DOA, which estimates the delay between the two microphones with GCC-PHAT and derives the angle. With the spacing that fits in a 1.75” diameter the resolution is coarse: left, centre, right and little more. And that’s plenty: the illusion is in the timing of the saccade, not the precision.

Presence by radar

The LD2410 module (24 GHz radar, €4–7) isn’t a PIR. A PIR detects heat changes, that is, movement: if you sit down to read and stay still for three minutes, it concludes nobody is there and NOVA would fall asleep with you in front of it. The radar detects moving and static targets separately, the static ones through micro-movement (breathing, posture shifts), with distance and a 5–6 m range in gates of about 75 cm.

What matters isn’t the three behaviours that come free with distance. It’s that it lets you tell apart two kinds of neglect that aren’t the same:

You were away on a trip for a week. Not your fault.

You sat right in front of it four hours a day and said nothing. That’s something else.

“You haven’t talked to me in 3 days” is a good joke. “You’ve been sitting right there” is better, and it’s only possible if the device knows you were there. That turns the sensor from ornament into part of the product’s heart.

Details already noted so an afternoon isn’t lost: it goes over UART and the header has exactly one free; the default baud rate is 256000, not 115200; it sees through plastic, so it goes inside the case with no window, but not through metal; it gets confused by fans and curtains, and the time goes on tuning sensitivities, not on code.

And a rule that came out of writing the firmware–backend contract: being connected to WiFi is not the same as someone being there. The board doesn’t send presence on purpose, because it lives plugged in and would start the “ignored while present” counter falsely. Presence is only sent once a real source exists: radar, touch or IMU.

Architecture

Device (ESP32-S3) / HTML test client
   |  eyes · mouth · body · speaker · mic · touch · IMU
   v  WebSocket  ws://.../ws/creature?device=xyz
Backend (Java), tested against the hardware
   ├── CreatureWebSocketHandler    JSON to domain calls
   ├── CreatureSessionRegistry     live creatures, per device
   ├── CreatureTriggerScheduler    every 30 s: anything to say?
   ├── Creature (pure domain)      the heart
   ├── CreatureEventStore          event history in Postgres
   └── ProactiveAudioLibrary       the 3 or 4 lines, over HTTP

The central principle

The state lives outside the language model. The model only verbalises it.

The state is four values: affection, energy, mood and last interaction. The mechanics: decay over time, inputs that inject values, and proactive triggers, that is, the device reacts without you talking to it because a timer fired.

After D16 the principle holds in its extreme form: outside the model, and with no model. The state is expressed with face, sound and, very occasionally, one of the three or four recorded lines. The line “I know” is only funny if it’s true.

Non-verbal language

Since D16 this is the main design work, the equivalent of what the eye timing was.

Intonation communicates; timbre only identifies. A rising sound is a question. A falling one, disappointment. Two short quick ones, joy. One long one fading out, sleep. Four or five rules, and if they’re always respected, whoever lives with NOVA learns them without noticing in two days. The pitch contour is the same for a beep and for a syllable; what changes with D17 is that now there’s someone behind it.

Three implementation decisions:

1. Sounds are synthesised, not recorded. A beep is arithmetic, not a file: it takes zero flash and gives infinite variation, which is what stops it sounding like a machine repeating the same WAV.
2. Sound is always attached to an animation. A chirp with the eyes squeezing is a creature. The same chirp on its own is a phone notification.
3. It’s prototyped in the eye bench itself, not in a separate sound bench. With WebAudio inside the bench, each gesture carries its sound on the same data line, and the previous rule holds by construction: if the gesture is interrupted, the sound cuts.

The gestures, in code

Sequences, not poses: the character is in what it does between expressions. Each gesture is a list of frames with their instant in milliseconds and, in each one, any combination of pose parameters, gaze (with the option of locking it), “z”s to release, and at the end a preset to finish in. Each gesture carries its sound as a requirement: a gesture without sound is half designed.

The pose has twelve parameters, all on the same spring: six for eyelids and pupil, three for the mouth and three for the body (offset and tilt). The bench exports the 32 gestures, their 149 frames and the rhythms as a C header the firmware includes as is; nothing is copied by hand.

Asleep isn’t a gesture: it’s derived. While it lasts, one “z” per breathing cycle and no automatic micro-gesture. The snore plays only once, on the first cycle after falling asleep, and then stays quiet until next time: repeated every cycle it got tiresome.

A deep bug found along the way: a gesture that interrupted another inherited its pose. Petting followed by a tap left the petting’s mouth and eyes forever, because when a gesture started it stored the current target as the rest pose, which was the previous gesture’s pose. Same family: changing preset or moving a slider during a gesture was undone when it ended. The rule that stuck: the rest pose changes through a single path, which also updates the base of the gesture in progress; a gesture that interrupts another restores that base first; and the sound cuts with the gesture, or an orphaned purr is left on top of the tap.

The activities, in code

An activity is an object with start, step and, if it reacts to touches, touch; with real constants (the ones exported to the firmware) and compressed ones for the bench (ten minutes is forty seconds). It’s logic, not data: it’s rewritten in C with the same reasoning, like the drawing.

What an activity can lend the face, all cleaned up when it ends: a held pose, the gaze, the rhythm (the timer switches to ALERT in the last 20%), guided breathing, silence, no micro-gestures, and the ring, the digits, the caption and a text.

Two untimed modes: silence (shush, and from then on visual only: the sound is cut in a single place, not in each gesture) and music (double take and then nodding to the tempo: drops sharply on the beat, comes back slowly, tilts alternately).

The two poses. During an activity there are two poses and they mustn’t be mixed: the one the activity holds (focus looking at its desk) and the true rest, which is the mood. The first version mixed them: cancelling the alarm halfway through the snooze growl left it asleep, and changing the mood during the timer was lost at the end, because it went back to the mood from ten minutes ago. Rule: the external (presets, sliders, affection; on the board, the backend) always updates the rest even when it isn’t visible; the activity’s internals never touch the rest and die with it. Found with a state harness under a controlled clock, not by eye: 28 states checked frame by frame, nine cross-interruption scenarios, and the sound counted with an oscillator counter (in silence and in focus none gets created).

The domain, in code

Module com.nova.creature.domain: pure Java, zero dependencies, not a single Spring annotation. Spring wraps it from outside; the personality logic doesn’t know a framework exists and the tests run in milliseconds.

Five decisions that hold:

1. Continuous exponential decay, not stepped. Three days in one jump give exactly the same as 72 one-hour jumps, and there’s a test that checks it. Without that property, a device that disconnects for an afternoon comes back with the wrong state, and that bug is hellish to find on the board.
2. Mood reverts to a baseline set by affection. The long bond fixes the resting state; events perturb it. It’s what makes the creature coherent with itself.
3. Diminishing returns on inputs. A saturation variable that decays. Without it, hammering ❤️ maxes out affection and the mechanic feels fake in ten seconds. Confirmed in real use from the browser: from 55 to 75 to 91.7 with the first presses, then movements in tenths.
4. Energy isn’t persisted: it’s derived from local time. A pure function of the clock, minimum at 4:00 and maximum at 16:00. And it overrides mood: in the small hours the timing personality is TIRED without exception, even with mood at its peak.
5. The Snapshot returns no geometry. Not an eyelid angle, not a curve: it returns what it feels and at what pace. The shape lives in the firmware, and that allows redesigning the whole face without touching the personality.

Ethics, as constants with tests

The “charming or coercive” question stops being an intention and becomes three numbers in Tuning, each with its test:

It isn’t “says it once and never again”, which would be worse: if you leave for two months it would mention it on day three and go quiet forever. It’s at most once a day, without escalating. If someone ever loosens those three numbers, let it be knowing what they’re loosening.

Writing the contract with the firmware surfaced a trigger that breaks that guarantee: LONELY has no cooldown of its own, only the general two-hour one. With affection below the loneliness threshold it could fire twelve times a day indefinitely. It’s noted and pending; the fix is the same pattern as NEGLECTED.

A reference timeline

boot         affection= 55.0 mood=  +0.0  NEUTRAL CALM
❤️           affection= 75.0 mood= +30.0  HAPPY   CALM
+2 h         affection= 74.5 mood= +19.9  NEUTRAL CALM
😡           affection= 72.5 mood= -21.3  HURT    CALM
+8 h         affection= 70.7 mood=  +6.4  NEUTRAL TIRED   <- the anger passed on its own
+3 days      affection= 56.4 mood=  +2.5  NEUTRAL TIRED   -> fires NEGLECTED
+2 weeks     affection= 28.9 mood=  -8.4  BORED   TIRED
❤️ returns   affection= 48.9 mood= +21.6  HAPPY   TIRED   <- asymmetric recovery

Pending tuning by living with it, not by theorising: should two weeks of silence leave it at 28? Should a single ❤️ take it from 28 to 48?

NEGLECTED is collected when you come back

Decided on 18 September while solving a voice problem, and kept with D16 because it affects the domain, not the audio.

The problem: at exactly three days the most likely thing is that nobody is there, precisely because you’ve been away three days. And since the firmware doesn’t send presence on purpose, the backend can’t know. The line would go out to an empty room. The video script already said it without saying it: you speak first, and NOVA answers with the three-days line. It’s not a timer that fires on its own, it’s a timer that’s collected when you come back.

How it ends up: the scheduler stops evaluating NEGLECTED. When an interaction arrives, before updating the last one, the elapsed time is calculated; if it exceeds the threshold, the interaction returns the notice together with that duration. If the interaction were applied first, the time would become zero and the line would no longer make sense. The domain returns only the fact and the duration; choosing whether the numbered variant plays (only between 72 and 96 hours) or an unnumbered one is the audio-selection layer’s job.

No “pending” state needs storing: it’s derived, just as energy is derived from the hour. And it has a good side effect: this way NEGLECTED can’t fire twice in 24 hours by construction, because the very interaction that delivers it resets the counter.

The WebSocket and the contract

The com.nova.creature.ws package is the border between Spring and the pure domain, just as persistence is for JPA. Three pieces: the handler that translates JSON messages into domain calls and returns the Snapshot; the session registry; and the scheduler that every 30 s in development (minutes in production) asks each connected creature whether it has something to say on its own.

The history is only rebuilt once, on connection, by replaying the stored events. From then on the creature lives in memory. When the last session closes it’s forgotten, because each event was already persisted as it happened; the next connection rebuilds from scratch.

Concurrency is real, not theoretical: Creature isn’t thread-safe, and it doesn’t need to be, but the thread processing an input and the scheduler’s thread can touch the same creature at once. Every access goes through synchronized (creature).

The protocol between the two pieces lives in a separate document, the firmware–backend contract, with each claim pointing at the code that guarantees it. I wrote it on 15 September, before flashing, and even then it brought to light things the code didn’t say: that the LOVE expression is unreachable over the network (no path in the domain returns it, because affection is a slow bond, not a face), that without inputs the mood can only sit at its baseline and therefore the only visible expressions in the first cut would be NEUTRAL, ASLEEP and BORED, and the cooldown LONELY is missing. Version 2, from 20 September, adds what stage 2a needed:

• Several sessions per device. The board and the test client connect with the same identifier and share the creature. Before, the last to connect displaced the previous one, and the proactive lines went to the browser instead of the board.
• state broadcast to every session after each input, so the board reacts instantly to what’s pressed in the client, without waiting for its next sync.
• gesture, which the backend forwards as is to every session without touching the creature or the history, and force, which pins expression and rhythm so the eight faces by four rhythms can be walked through on the board without having to push affection up to each one. It’s the only way to see LOVE.
• speak now does something on the board: it launches the speaking gesture, with the mouth pulsing with the syllables.

Testing six months of neglect in milliseconds

All the personality logic depends on the passage of time. Because the time source is injected, the tests simulate three days, a week or six months of silence without waiting for anything. Adding the history surfaced an interesting failure: the replay clock was born by default in 1970 until someone set it to a date. It was fixed by removing from the constructor the possibility of being born without knowing when. Objects that can be born in an invalid state end up being born in an invalid state.

The firmware, in code

Project nova-firmware, on ESP-IDF v6.1. Four dependencies and not one hand-written driver:

# main/idf_component.yml
dependencies:
  waveshare/esp32_s3_touch_amoled_1_75: { version: "*" }
  lvgl/lvgl: { version: "9.4.*", public: true }
  espressif/esp_websocket_client: { version: "^1" }
  espressif/cjson: { version: "*" }

The finding that changed phase 1: there’s an official managed component for the board, and it solves everything with one call. bsp_display_start() initialises the display over QSPI, the touch over I2C and starts the LVGL task; bsp_audio_init() does the same for the codec and the speaker. That was 80% of the dreaded bring-up work, and it doesn’t exist.

How an eye is drawn. Not with LVGL widgets but by writing the canvas buffer directly, in horizontal runs (outside, antialias ramp, solid, halo, pupil), and evaluating floating point only in the two to six transition pixels of each row, not in the canvas’s 23,000. The eyelid is subtracted from the body by writing the alpha channel by hand, with partial-coverage antialiasing: it’s the exact equivalent of the bench’s destination-out, not an approximation. The canvas is RGB565A8: 16-bit colour with alpha kept separately, half the PSRAM of ARGB8888 and no format conversion on blit.

Curved eyelids (16 September) seemed to call for polygons, and LVGL 9 has no polygon primitive. The way out was realising that the bench’s quadratic Bézier, with both ends at the same height, is a pure parabola: it’s solved as a closed formula per column, with no tessellation. And a deliberate decision: the bench’s span factor (2.6) wasn’t copied. With 2.6 the arch parameter is almost all vertical offset and barely any curvature, because the bench needs it that way so the eyelid keeps covering after being rotated; ported literally, the laugh still came out as a flat band. It was lowered to 1.05 so there’d be a real arch, verified by rendering both options before flashing, not by eye on the board.

Mouth and body (20 September). The mouth is a third canvas, two filled cubics per scanline, and it’s only rasterised when a gesture asks for it. The body shifts the whole canvases, as breathing already did; the tilt doesn’t rotate bitmaps (rotating two canvases in software every frame would sink the FPS): one eye goes up and the other down, and the eyelids skew, which is how the rotations compose in the bench.

Performance: 61 real FPS, measured on the board. The 49 FPS benchmark from 14 September was Waveshare’s demo; the face itself had never been measured. When a glow effect was requested on the 17th it turned out to be running at 21 FPS, with no headroom at all. The path, measured with esp_timer at each step: 17 ms per eye at baseline, of which 11.7 were three lv_draw_rect with radius masks; rasterising by hand, 8 ms and 34 FPS; from pixel by pixel to runs per row, 3.6 ms and 51 FPS; invalidating only the canvas rectangle with real content, 61 FPS. The ceiling is the 16 ms tick itself (62.5 FPS), not the drawing. Before finding the real cause, three plausible hypotheses were ruled out with measurement: that the LVGL blit was expensive, that the PSRAM was slow, and that it was the instruction cache. And each version of the rasteriser was compared pixel by pixel in Python against the previous one before flashing: zero differing pixels across 192 combinations.

The glow was dropped. Implemented, tested on the board, and out: it dropped from 61 to 20 FPS because the dirty area of the two eyes merges into a much larger rectangle, and without the scanlines that accompany it in the bench it looked fake. It stays behind a flag at zero in case it’s ever revisited by another route.

Audio (20 September). A dedicated task synthesises D17’s full babble (sawtooth with vibrato, two formant biquads in parallel, breath noise, consonant hit, per-syllable envelope, snore tremolo) and writes it to the ES8311 codec in blocks of 128 samples at 22,050 Hz. Measured with esp_timer: about 18 µs per block at rest, 0.3% of a 5.8 ms block; a 57.7% peak during a WiFi reconnection. The eight beep contours and babble with chosen vowels are ported; each gesture fires its sound on start and the new one cuts the previous.

Concurrency on the board. The network module never touches LVGL: it posts to length-one queues with overwrite semantics, and the eye tick drains them inside the LVGL task. They’re two queues, state and gesture, not one: if both arrive between two ticks, with a single one the second would overwrite the first. Sound goes through a third. It’s the same reasoning as the backend’s synchronized, and it leaves nova_eyes.c unaware that the network exists.

Verifying without a board and without looking. The bench is verified in headless Chrome with an instrumented copy that patches performance.now and requestAnimationFrame with a controlled clock, to capture the exact instant of each gesture and fire the sounds while dumping errors to the DOM. Chrome’s --virtual-time-budget didn’t work: twelve captures at different budgets all came out on the first frame.

Economics

With AMOLED at €89 the margin is thin, but the product is visibly better.

The cost missing from almost every analysis is the recurring one: €2–6 per unit per month of inference. It’s the variable that separates what scales from what becomes a liability. D6 shifted it to the user; D16 made it disappear, because the device no longer does inference. The only paid service left is music identification, at $0.005 a query and with a monthly cap: the product absorbs it.

Another metric almost nobody looks at: cloning speed. For an NFC-card-type product, three days. For NOVA, months, and even then they can’t copy the character.

Traps: the ones avoided and the ones that actually happened

In the firmware

• -Wno-error=attributes in main/CMakeLists.txt is mandatory, not cosmetic. This one did happen. While simplifying Waveshare’s example CMakeLists I removed that flag; with CONFIG_LV_ATTRIBUTE_FAST_MEM_USE_IRAM on, LVGL marks functions for IRAM and GCC 15 (ESP-IDF v6.x) detects conflicting sections in lv_color.c and aborts the build. Corollary: Waveshare’s examples carry odd flags for a reason, and you don’t clean them up without understanding what they do.
• The core json component was removed in ESP-IDF v6.0. cJSON.h keeps working the same, but it has to be pulled in as espressif/cjson in the manifest; without that, cJSON.h: No such file when compiling the network client.
• Kconfig.projbuild isn’t auto-generated and doesn’t warn if missing. If the project’s menu doesn’t show up in idf.py menuconfig, the first thing to check is that the file physically exists in main/, and after creating it you need idf.py reconfigure.
• Changing partitions.csv requires idf.py erase-flash before the first flash with the new table, or the bootloader may read an otadata full of garbage from the previous one. The original table, with an 8 MB factory app, didn’t allow OTA: two slots plus otadata didn’t fit in 16 MB. It was changed while it was cheap, with the board still on the desk and flashing over cable.
• A DMA error that looked benign and wasn’t. Failed to allocate priv TX buffer showed up as an occasional spike on 16 September. The real cause: the LVGL buffers lived in PSRAM and the display driver doesn’t read from there directly, so for each strip it allocated an intermediate buffer in internal RAM, copied it and freed it; under WiFi traffic that RAM fragmented. Fixed at the root by moving the strips to fixed internal RAM (22 KB reserved before WiFi starts): zero copies, zero errors across four full boots.
• The WiFi tasks live on core 0. It actually happened on 20 September: the audio task, created with no affinity, competed with the WiFi association and with LVGL on core 0; the idle thread didn’t run for five seconds and the watchdog fired with LVGL stuck drawing an eye. Fix: the audio task pinned to core 1, and the per-sample sinf, cosf and powf gone (vibrato and tremolo now rotate a unit vector). The cost at rest dropped from 30 to 18 µs per block and the reconnection peak from 276% (overrun) to 57.7%. If the eyes stutter on connecting, first check which new task has landed on core 0.
• Turning the volume up by the numbers didn’t work. The simulation said the loudest sound reached 0.23 of full scale and the hums came out five to ten times quieter; the master gain was raised and a per-sound one added. On the board it sounded horrible and was reverted entirely. Digital peak isn’t the limit: the 5 V amplifier and the 8 Ω speaker in its enclosure distort well before. And esp_codec_dev_set_out_vol(70) isn’t 70%: the default curve maps 0 to −50 dB and 100 to 0 dB, so 70 was −15 dB by accident.
• CONFIG_ESPTOOLPY_FLASHSIZE_16MB. The default hello_world assumes 2 MB and warns about the mismatch; without that line 14 of the 16 MB are wasted.
• USB-C cable with no data lines. Half of all cables are charge-only: the board powers on, the computer doesn’t see it and you lose an hour thinking it’s dead. Avoided: the cable used did have them.
• ESP-IDF v6.1 isn’t validated by Waveshare (only 5.5.5 and 6.0.2). The risky option was tried first and it compiles; no need to downgrade.
• 8 MB PSRAM or don’t start. The “R8” in the part number. Verified before buying; a listing that was actually a WiFi-less RP2350 in disguise was spotted too.

In the backend

• Spring Boot 4 uses Jackson 3: databind moved to tools.jackson (asText() becomes asString()), but the annotations didn’t: @JsonIgnoreProperties and friends stay in com.fasterxml.jackson.annotation, and tools.jackson.annotation doesn’t exist.
• Diminishing returns are strong on purpose. With a 20-minute stimulation half-life, a burst of clicks damps fast (÷11 at the tenth, ÷21 at the twentieth). Not a bug: it’s in Tuning, and it was confirmed in real use.

In the voice

• The local TTS that sounds good is too slow, and the fast one doesn’t sound the way it should. It’s not a matter of settings but of model architecture (see D15).
• ElevenLabs’ usage policy forbids using its audio as input to another model. It closes cloning one of its voices into a local TTS by contract, not by quality. And the free plan grants no commercial rights over the audio.
• Hume Octave doesn’t do synthetic timbres. It’s trained for realistic human speech: however much robot you ask for, you get a person playing a robot.
• The 170 Hz snore sounded like a snake. The second formant of the “z” was at 4,200 Hz on purpose; heard for real it was a hiss. It was lowered until the fundamental fell inside the first formant, measured with a Python simulation of the same algorithm, not by ear, and along the way a real bug came out: the per-syllable detune was recalculated on every sample instead of once per syllable. The timbre was left as is, on purpose: after two rounds it still didn’t sound like a snore and it isn’t a priority sound.

In the bench

• A gesture that interrupts another inherited its pose, and an activity mixed its held pose with the rest. Both lessons are in the architecture section; both were found by a harness, not by an eye.
• Headless Chrome’s --virtual-time-budget doesn’t reliably advance requestAnimationFrame animations. What works is patching the clock.

For the custom PCB

A 470–1000 µF capacitor near the amplifier (the surge when the speaker starts sags the rail and the ESP32 resets; it’s bug number one in first audio projects), 5.1 k resistors on the USB-C CC1/CC2, no AMS1117, and muting the amplifier when it’s not playing to avoid the “pop”.

Roadmap

The two-week cohabitation phase with buttons was removed by D14; it stays struck through in the working document, not deleted, so there’s a record of what was decided not to do and why. The reasoning that justified it (80% of these projects die because their own creator stops talking to them on day four) is still true. That risk is accepted without having measured it.

The stages of phase 2

The order is one of dependencies, not of wishes. Each stage leaves something verifiable before the next, and none is opened without closing the previous one in the working document.

Stage 2a wasn’t in the plan of the 19th: I asked for it on the 20th, before adding wake word and activities, so that everything the bench already did would run on the board and I could check it by hand. Verified that night: all 32 gestures with mouth, body and sound, mostly at 59 FPS with peaks of 61 and occasional dips to 52–55 on the body gestures, which move the whole canvases. Accepted as is for now.

What’s left of phase 2 after the stages: gaze towards the voice with the microphone array, the “NOVA” wake word, deciding where emotions come from now that there’s no voice to classify, NEGLECTED delivered on the next interaction, and the three or four lines in Spanish and English.

Open questions

• Where do emotions come from without conversation? D14 assumed sentiment classification on the voice, which no longer exists. Candidates: recognised commands, petting on the touch screen, picking it up, presence. It has to be decided explicitly, not by default.
• What’s the video’s headline? “I built a ChatGPT pet” no longer describes the product. Candidate since D18: the pet that works alongside you.
• Does microWakeWord work for the commands too, or is separate recognition needed? How does it fare in Spanish? And how many models can the board sustain at 60 FPS? D18 assumes ten or twelve fixed words; it isn’t measured.
• Does the alarm ring in silent mode? In the bench, silence is silence, but an alarm that doesn’t ring isn’t an alarm.
• In focus, are touches ignored entirely, or does a pet count as pause?
• What happens to affection during a pomodoro? Ignoring it shouldn’t cost, and completing one should perhaps add.
• ALERT in the last 20% of the timer or in a fixed last two minutes? With ten minutes it’s the same; with an hour, twelve minutes of tension seems a lot.
• A mouth in some preset too? Today only in gestures, on purpose. Laugh and Happy are the candidates.
• How many voice lines survive exactly? Three or four, decided by living with it.
• One giant eye or two eyes inside the circle? There’s a board with two round displays that opens a third direction.
• Are transcripts stored? After D16 the question narrows: on the device there’s nothing to transcribe, so it only applies to what’s typed in the app. The “what’s playing?” audio isn’t stored; only the result.
• Final name? NOVA is the working one.

What’s already done

• 20 September. Stage 1 closed: the speaker sounds over I2S with D17’s synthesis ported to C and measured; the watchdog fired once because of the unpinned audio task and was fixed. Stage 2a closed the same night: the bench exports the firmware header, mouth, body and gesture sequencer on the board, eight beep contours plus babble, gesture and force messages, several sessions per device, contract at v2. Verified on the board.
• 19 September. D17: the eye bench becomes the gesture bench, with WebAudio. Babble, mouth, body, “z”s and derived sleep; 20 gestures, then 32 with the activities. Inherited-pose bug fixed. In the afternoon, D18 and stage 0: the whole activity layer in the bench, with ring and digits, verified with a 28-state harness that found the two-poses bug. Phase 2 plan in seven stages.
• 18 September. The 25 lines generated with ElevenLabs and a custom effect, after dropping Chatterbox over ElevenLabs’ policy and Hume over timbre. com.nova.creature.audio package built: library, no-repeat selection by shuffled bag, served over HTTP. Decided that NEGLECTED is delivered on the next interaction. And D16: NOVA doesn’t talk.
• 17 September. In the morning, from 21 to 61 FPS with measurement at each step, the DMA error fixed at the root and the glow tried and dropped. In the afternoon, the local TTS session (XTTS, Kokoro, Piper, Chatterbox), D15 and the 25 lines written in Spanish, warm but restrained in tone and without a drop of reproach.
• 16 September. First successful flash end to end: the board connects, opens the WebSocket and the face obeys the real state. Validated by pressing ❤️ from the client with the same identifier as the board. HAPPY and LAUGH confirmed reachable. Partition table with OTA. In the afternoon, curved eyelids as a closed parabola, specular highlight with drag, iris halo and eyeball parallax. Phase 1 closed.
• 15 September. WiFi and WebSocket client written (credentials from NVS with Kconfig fallback, reconnection with backoff, sync every five seconds, no presence on purpose); rhythms move from macros to a runtime table; thread-safe queue between the network and the eyes. Firmware–backend contract written, which surfaced LOVE being unreachable and the cooldown LONELY lacks. D14: skip the v0 with buttons.
• 14 September. Board received and verified. Factory demo checked, managed component discovered, ESP-IDF v6.1 compiles the examples, LVGL benchmark at 49 FPS. nova-firmware project created. The eyes blink on the real board. Build broken and fixed by -Wno-error=attributes.
• 6 September. Tests converted to JUnit 5 (the previous main() was invisible to the build: a false green); event history in Postgres with replay reconstruction verified; WebSocket built and tested; ESP-IDF v6.1 installed and hello_world compiled for ESP32-S3.
• 5 September. Fundamental change: wake-up by voice, not buttons. Roadmap reordered.
• 4 September. Eye bench v1 and v2 on canvas; state machine written and green; camera ruled out; perception documented.
• 3 September. Idea selected out of 34; board bought; decided on the integrated board, state machine over LLM and v0 without voice.