This capstone puts every part of the AI path together into a desk assistant you can talk to. Press the space bar and ask a question. Your browser's speech recognition turns your voice into text and sends it to the ESP32 over the USB cable. The ESP32 asks Claude, with a system prompt that makes it a helpful desk assistant called Pixel, and remembers the conversation. The answer appears on the LCD face, the computer speaks it out loud, and the robot head nods along. Three lights on its forehead show what it is doing: green while listening, yellow while thinking, red while speaking.
What you'll learn
- How to plan a system with several AI parts: speech recognition, a language model and speech synthesis
- How to split the work between a browser and a microcontroller
- How to design a simple message protocol between them
- How to give a device a personality with a system prompt
What you need
- ESP32 board, expansion board and USB cable (ESP32 IoT Starter Kit)
- 1602 LCD display, 10 kΩ potentiometer and 1 resistor, 220 Ω (ESP32 IoT Starter Kit)
- Servo motor SG90 with horns and screws (ESP32 IoT Starter Kit)
- Green, yellow and red LEDs and 3 resistors, 220 Ω (ESP32 IoT Starter Kit)
- A computer with a microphone and Google Chrome or Microsoft Edge
- A Claude API key from an adult's Anthropic account
- A 2.4 GHz Wi-Fi network
- The printed parts from the 3D printing box below (robot head), or the cardboard version described there
- Breadboard and jumper wires (ESP32 IoT Starter Kit)
Wiring
| From | To |
|---|---|
| LCD VSS, RW and K | GND |
| LCD VDD | 5V |
| LCD V0 | Middle pin (W) of the potentiometer |
| LCD RS | ESP32 GPIO 19 |
| LCD E | ESP32 GPIO 23 |
| LCD D4, D5, D6, D7 | ESP32 GPIO 18, 17, 16, 4 |
| LCD D0 to D3 | Not connected |
| LCD A (backlight +) | 220 Ω resistor, then 5V |
| Potentiometer outer pins | 5V and GND |
| Servo signal (orange) | ESP32 GPIO 13 |
| Servo power (red) and GND (brown) | 5V and GND |
| ESP32 GPIO 25 | 220 Ω resistor, then the green LED (listening) |
| ESP32 GPIO 26 | 220 Ω resistor, then the yellow LED (thinking) |
| ESP32 GPIO 27 | 220 Ω resistor, then the red LED (speaking) |
| All LED short legs | GND |
Build it in stages
- Face. Wire the LCD and upload the sketch. It shows "Hello! I am Pixel."
-
Brain. Type
Q:What is a robot?in the Serial Monitor (115200 baud, Newline). The yellow light comes on, then the answer appears on the LCD and in the Serial Monitor. - Body. Add the servo and LEDs. The head nods while it "speaks".
- Voice. Close the Serial Monitor, open the voice page and talk to it.
Get an API key
- Claude's API is a paid service, and an account must belong to an adult (18+). Ask a parent or teacher to create an account at console.anthropic.com, add a small amount of credit and set a low monthly spend limit under Billing.
- Under API keys, create a key and copy it. It starts with
sk-ant-. It works like a password and a credit card in one: never share it, never post your sketch online with the key in it, and delete the key in the Console when the project is finished. - Each question costs well under one cent with the settings in this sketch. Check the Console's usage page now and then.
The ESP32 sketch
In the Arduino IDE, open Library Manager and install the ArduinoJson library by Benoit Blanchon and ESP32Servo by Kevin Harrington.
Replace YOUR_WIFI_NAME and YOUR_WIFI_PASSWORD with your own network details (2.4 GHz only).
#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>
#include <LiquidCrystal.h>
#include <ESP32Servo.h>
const char* WIFI_NAME = "YOUR_WIFI_NAME";
const char* WIFI_PASSWORD = "YOUR_WIFI_PASSWORD";
const char* API_KEY = "YOUR_ANTHROPIC_API_KEY"; // keep this secret!
const char* MODEL = "claude-opus-5-5";
const char* SYSTEM = "You are Pixel, a friendly robot desk assistant for a young maker aged 10 to 15. "
"Your answer is spoken aloud and shown on a 16x2 screen, so answer in at most three short sentences. "
"Use plain English letters and punctuation only: no emoji, symbols, lists or formatting. "
"Be warm, encouraging and accurate, and keep everything suitable for children. "
"If you don't know something, say so.";
const int SERVO_PIN = 13, LISTEN_PIN = 25, THINK_PIN = 26, SPEAK_PIN = 27;
LiquidCrystal lcd(19, 23, 18, 17, 16, 4);
Servo neck;
const int MEMORY = 8;
String roles[MEMORY], texts[MEMORY];
int remembered = 0;
unsigned long speakUntil = 0;
String pageText = "";
int pageStart = 0;
unsigned long nextPage = 0;
// Only plain letters fit on the LCD: replace anything else with a space
String plain(const String& s) {
String out = "";
for (unsigned int i = 0; i < s.length(); i++) {
char ch = s[i];
out += (ch >= 32 && ch < 127) ? ch : ' ';
}
return out;
}
// One LCD line of up to 16 characters starting at 'from', breaking at a space
int fitLine(const String& s, int from, String& line) {
int end = min((int)s.length(), from + 16);
if (end < (int)s.length()) {
int space = s.lastIndexOf(' ', end);
if (space > from) end = space;
}
line = s.substring(from, end);
while (end < (int)s.length() && s[end] == ' ') end++;
return end;
}
void showPage() {
String one, two;
int next = fitLine(pageText, pageStart, one);
int after = fitLine(pageText, next, two);
lcd.clear();
lcd.print(one);
lcd.setCursor(0, 1);
lcd.print(two);
pageStart = after >= (int)pageText.length() ? 0 : after; // start again at the end
}
void showOnLcd(const String& text) {
pageText = plain(text);
pageStart = 0;
nextPage = 0;
}
void turnPages() {
if (pageText.length() == 0 || millis() < nextPage) return;
nextPage = millis() + 3500;
showPage();
}
// Ask Claude. roles[] are "user" or "assistant", texts[] the messages, n how many.
String askClaude(const char* system, const String roles[], const String texts[], int n) {
JsonDocument request;
request["model"] = MODEL;
request["max_tokens"] = 2000; // room for its thinking and a short answer
request["output_config"]["effort"] = "low"; // quick, simple answers
request["fallbacks"] = "default"; // if this model declines, try another one
request["system"] = system;
JsonArray messages = request["messages"].to<JsonArray>();
for (int i = 0; i < n; i++) {
JsonObject m = messages.add<JsonObject>();
m["role"] = roles[i];
m["content"] = texts[i];
}
String body;
serializeJson(request, body);
WiFiClientSecure client;
client.setInsecure();
HTTPClient http;
http.setTimeout(60000); // thinking can take a few seconds
http.begin(client, "https://api.anthropic.com/v1/messages");
http.addHeader("content-type", "application/json");
http.addHeader("x-api-key", API_KEY);
http.addHeader("anthropic-version", "2023-06-01");
http.addHeader("anthropic-beta", "server-side-fallback-2026-07-01");
int status = http.POST(body);
String response = http.getString();
http.end();
if (status != 200) {
Serial.printf("Claude API error %d: %s\n", status, response.c_str());
if (status == 401) return "Please check my API key.";
if (status == 429 || status == 529) return "I'm busy, ask me again in a minute.";
return "Sorry, I could not reach Claude.";
}
JsonDocument filter; // keep only what we need from the answer
filter["stop_reason"] = true;
filter["content"][0]["type"] = true;
filter["content"][0]["text"] = true;
JsonDocument answer;
if (deserializeJson(answer, response, DeserializationOption::Filter(filter))) return "Sorry, I got a strange answer.";
if (answer["stop_reason"] == "refusal") return "Sorry, I can't help with that one.";
String text = "";
for (JsonObject block : answer["content"].as<JsonArray>()) {
if (block["type"] == "text") text += block["text"].as<String>(); // skip the other kinds of block
}
text.trim();
return text;
}
void status(int listen, int think, int speak) {
digitalWrite(LISTEN_PIN, listen);
digitalWrite(THINK_PIN, think);
digitalWrite(SPEAK_PIN, speak);
}
void remember(const String& role, const String& text) {
if (remembered == MEMORY) {
for (int i = 2; i < MEMORY; i++) { roles[i - 2] = roles[i]; texts[i - 2] = texts[i]; }
remembered -= 2;
}
roles[remembered] = role;
texts[remembered] = text;
remembered++;
}
void answer(const String& question) {
status(0, 1, 0);
lcd.clear();
lcd.print("Hmm, let me");
lcd.setCursor(0, 1);
lcd.print("think...");
Serial.println("S:thinking");
remember("user", question);
String reply = askClaude(SYSTEM, roles, texts, remembered);
remember("assistant", reply);
reply.replace("\n", " ");
Serial.println("A:" + reply); // the page speaks this line
showOnLcd(reply);
status(0, 0, 1);
speakUntil = millis() + 400 + reply.length() * 70; // about as long as speaking takes
}
void setup() {
Serial.begin(115200);
pinMode(LISTEN_PIN, OUTPUT);
pinMode(THINK_PIN, OUTPUT);
pinMode(SPEAK_PIN, OUTPUT);
neck.attach(SERVO_PIN);
neck.write(90);
lcd.begin(16, 2);
lcd.print("Connecting...");
WiFi.begin(WIFI_NAME, WIFI_PASSWORD);
while (WiFi.status() != WL_CONNECTED) delay(500);
showOnLcd("Hello! I am Pixel. Ask me anything.");
Serial.println("S:ready");
}
void loop() {
turnPages();
if (millis() < speakUntil) { // nod while speaking
neck.write(90 + 12 * sin(millis() / 180.0));
} else if (speakUntil != 0) {
speakUntil = 0;
neck.write(90);
status(0, 0, 0);
}
if (!Serial.available()) return;
String line = Serial.readStringUntil('\n');
line.trim();
if (line == "L:1") status(1, 0, 0); // the page is listening
else if (line == "L:0") status(0, 0, 0);
else if (line.startsWith("Q:") && line.length() > 2) answer(line.substring(2));
}The voice page
Copy the code below into a text editor (Notepad on Windows, TextEdit in plain-text mode on a Mac), save it as desk-assistant.html on your computer and open it in Google Chrome or Microsoft Edge. Other browsers don't support Web Serial yet. Close the Arduino IDE's Serial Monitor first: only one program can use the port.
<!doctype html>
<html><head><meta charset="utf-8"><title>Pixel</title>
<style>
body{font-family:sans-serif;max-width:640px;margin:20px auto;padding:0 16px}
button{padding:14px 20px;margin:6px 6px 6px 0;font-size:18px;border:0;border-radius:12px;background:#1B3A7B;color:#fff}
#talk.on{background:#23A07A}.me{text-align:right;color:#1B3A7B}.pixel{background:#F1F5FC;border-radius:12px;padding:10px}
</style></head><body>
<h1>Talk to Pixel</h1>
<button onclick="connect()">Connect ESP32</button><button id="talk" onclick="listen()">🎤 Talk (space bar)</button>
<p id="state">Not connected.</p><div id="chat"></div>
<script>
let writer, reader, buffer = '';
async function connect() {
const port = await navigator.serial.requestPort();
await port.open({ baudRate: 115200 });
writer = port.writable.getWriter();
const decoder = new TextDecoderStream();
port.readable.pipeTo(decoder.writable);
reader = decoder.readable.getReader();
document.getElementById('state').textContent = 'Connected. Press the space bar and speak.';
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += value;
let i;
while ((i = buffer.indexOf('\n')) >= 0) {
const line = buffer.slice(0, i).trim();
buffer = buffer.slice(i + 1);
if (line === 'S:thinking') document.getElementById('state').textContent = 'Pixel is thinking...';
if (line.startsWith('A:')) reply(line.slice(2));
}
}
}
function send(text) {
if (writer) writer.write(new TextEncoder().encode(text + '\n'));
}
function add(cls, text) {
const p = document.createElement('p');
p.className = cls;
p.textContent = text;
document.getElementById('chat').prepend(p);
}
function reply(text) {
add('pixel', text);
document.getElementById('state').textContent = 'Press the space bar to talk.';
const voice = new SpeechSynthesisUtterance(text);
voice.rate = 1.05;
speechSynthesis.speak(voice);
}
function listen() {
const Recognition = window.SpeechRecognition || window.webkitSpeechRecognition;
if (!Recognition) { alert('Please use Chrome or Edge.'); return; }
speechSynthesis.cancel(); // stop talking when you start
const r = new Recognition();
r.lang = 'en-GB';
const talk = document.getElementById('talk');
talk.classList.add('on');
send('L:1');
r.onresult = (e) => {
const text = e.results[0][0].transcript;
add('me', text);
send('Q:' + text.replace(/\n/g, ' '));
};
r.onend = () => { talk.classList.remove('on'); send('L:0'); };
r.start();
}
document.addEventListener('keydown', (e) => {
if (e.code === 'Space' && !e.repeat) { e.preventDefault(); listen(); }
});
</script></body></html>client.setInsecure() keeps the connection encrypted but skips checking the server's certificate. That keeps the sketch short and is fine for a learning project. For a device you leave running for years, load the server's root certificate with client.setCACert() instead.
Talk to Pixel
- Upload the sketch, close the Serial Monitor and open the page in Chrome or Edge.
- Click Connect ESP32. Press the space bar, allow the microphone, and ask "What can you do?". The green light shows Pixel is listening.
- The yellow light means it is thinking. Then the answer appears on the LCD face and in the page, the computer speaks it, and the head nods.
- Ask a follow-up, like "Tell me more". Pixel remembers the last four questions.
Privacy: in Chrome, speech recognition sends the sound of your voice to Google, and your questions go to Anthropic to be answered. Don't say or ask anything private.
How it works
Three AI systems work together here, each on the machine that suits it best. Speech recognition runs in the browser, which has the microphone. The language model runs at Anthropic, reached by the ESP32 over Wi-Fi, so the API key stays on the ESP32 and never appears in the web page. Speech synthesis runs in the browser again, which has the speakers.
The browser and the ESP32 talk with a tiny protocol: every message is one line, and its first letters say what it is. Q: is a question, A: an answer, L:1 and L:0 switch the listening light, S: a status. Anything else, like debugging messages, is ignored by the page. Simple, clear message formats are how most devices talk to each other.
askClaude() sends a request to Claude's Messages API: the model's name, a system prompt that tells Claude who it is and how to answer, and the conversation so far. It is written as JSON with the ArduinoJson library and sent over HTTPS with the API key in a header. A few seconds later the answer comes back, also as JSON. Its content is a list of blocks; the sketch keeps only the text blocks, the actual answer. effort: "low" asks for quick, simple answers, which suits short questions and keeps them cheap.
Claude doesn't remember anything between requests. To hold a conversation, the sketch sends the last few questions and answers every time. That is also why the cost grows if you send a long history.
If Claude declines a request, the answer's stop_reason is refusal, and the sketch says so politely. fallbacks: "default" lets the service try a different model first.
The head nods with a sine wave: sin(millis() / 180.0) swings smoothly between -1 and 1, so the servo rocks 12 degrees each way for about as long as the answer takes to speak.
Try this next
- Give Pixel a different personality, or let it speak another language.
- Let Pixel control the model room from lesson 7: if the question is a command, it sends the right line to the room.
- Show a little face on the LCD while it listens, using custom characters.
Troubleshooting
-
"Please check my API key": copy the key again from the Console; it must start with
sk-ant-and have no spaces. - "I could not reach Claude": check the Serial Monitor for the error. Make sure the account has credit and the spend limit isn't reached.
- The LCD lights up but shows nothing, or only black boxes: turn the potentiometer slowly until the text appears.
-
The page says nothing: check the computer's sound, and that the line in the Serial Monitor started with
A:. - The board restarts when the servo moves: servos draw bursts of current. Use a good USB port or a powered hub, and check that the servo's red wire is on 5V.
- A printed part is too tight or too loose: printers differ by a few tenths of a millimetre. Sand or file a tight hole, and fix a loose part with a drop of hot glue or a strip of tape.
