Smart speakers understand what you say and switch things on for you. You can build the same idea. Your browser has speech recognition built in: an AI model that turns your voice into text. In this lesson a web page listens to you, works out what you want, and sends a command to the ESP32 over the USB cable. The ESP32 runs a little model room with a ceiling lamp, a coloured mood light and a curtain on a servo. Say "lamp on", "make it purple", "open the curtain" or "good night", and the room obeys, and the page answers you out loud.
What you'll learn
- How speech recognition turns sound into text
- How to find a command and its details in a sentence (intent and slots)
- How a web page and the ESP32 share a simple command language
- How speech synthesis gives the computer a voice
What you need
- ESP32 board, expansion board and USB cable (ESP32 IoT Starter Kit)
- White LED and 1 resistor, 220 Ω (ESP32 IoT Starter Kit)
- RGB LED and 3 resistors, 220 Ω (ESP32 IoT Starter Kit)
- Servo motor SG90 with horns and screws (ESP32 IoT Starter Kit)
- A computer with a microphone and Google Chrome or Microsoft Edge
- The printed parts from the 3D printing box below (model room), or the cardboard version described there
- Breadboard and jumper wires (ESP32 IoT Starter Kit)
Wiring
| From | To |
|---|---|
| ESP32 GPIO 13 | 220 Ω resistor, then the white LED's long leg |
| ESP32 GPIO 25 | 220 Ω resistor, then the RGB LED's red leg |
| ESP32 GPIO 26 | 220 Ω resistor, then the green leg |
| ESP32 GPIO 27 | 220 Ω resistor, then the blue leg |
| White LED short leg and RGB common (longest) leg | GND |
| Servo signal (orange) | ESP32 GPIO 18 |
| Servo power (red) and GND (brown) | 5V and GND |
The ESP32 sketch
In the Arduino IDE, open Library Manager and install the ESP32Servo library by Kevin Harrington.
The ESP32 understands short commands like LAMP 1 or COLOUR 255 0 128. You can test it in the Serial Monitor at 115200 baud before using your voice.
#include <ESP32Servo.h>
const int LAMP_PIN = 13, RED_PIN = 25, GREEN_PIN = 26, BLUE_PIN = 27, SERVO_PIN = 18;
const int CLOSED_ANGLE = 10, OPEN_ANGLE = 150;
Servo curtain;
bool party = false;
unsigned long lastParty = 0;
void colour(int r, int g, int b) {
analogWrite(RED_PIN, r);
analogWrite(GREEN_PIN, g);
analogWrite(BLUE_PIN, b);
}
void setup() {
Serial.begin(115200);
pinMode(LAMP_PIN, OUTPUT);
pinMode(RED_PIN, OUTPUT);
pinMode(GREEN_PIN, OUTPUT);
pinMode(BLUE_PIN, OUTPUT);
curtain.attach(SERVO_PIN);
curtain.write(CLOSED_ANGLE);
Serial.println("READY");
}
void loop() {
if (party && millis() - lastParty > 300) { // party mode: random colours
lastParty = millis();
colour(random(256), random(256), random(256));
}
if (!Serial.available()) return;
String line = Serial.readStringUntil('\n');
line.trim();
int a = 0, b = 0, c = 0;
if (sscanf(line.c_str(), "LAMP %d", &a) == 1) {
digitalWrite(LAMP_PIN, a);
Serial.println(a ? "OK The lamp is on." : "OK The lamp is off.");
} else if (sscanf(line.c_str(), "COLOUR %d %d %d", &a, &b, &c) == 3) {
party = false;
colour(a, b, c);
Serial.println("OK Done.");
} else if (sscanf(line.c_str(), "CURTAIN %d", &a) == 1) {
curtain.write(map(constrain(a, 0, 100), 0, 100, CLOSED_ANGLE, OPEN_ANGLE));
Serial.println(a > 0 ? "OK Opening the curtain." : "OK Closing the curtain.");
} else if (line == "PARTY") {
party = true;
Serial.println("OK Let's party!");
} else if (line == "NIGHT") {
party = false;
digitalWrite(LAMP_PIN, LOW);
colour(0, 0, 0);
curtain.write(CLOSED_ANGLE);
Serial.println("OK Good night. Sleep well!");
}
}The voice page
Copy the code below into a text editor (Notepad on Windows, TextEdit in plain-text mode on a Mac), save it as voice-room.html on your computer and open it in Google Chrome or Microsoft Edge. Other browsers don't support Web Serial yet. Close the Arduino IDE's Serial Monitor first: only one program can use the port.
<!doctype html>
<html><head><meta charset="utf-8"><title>Voice room</title>
<style>
body{font-family:sans-serif;max-width:640px;margin:20px auto;padding:0 16px;text-align:center}
button{padding:14px 20px;margin:6px;font-size:18px;border:0;border-radius:12px;background:#1B3A7B;color:#fff}
#mic.on{background:#DE544A}#heard{font-size:26px;min-height:36px}#said{color:#1B3A7B;font-size:20px}
</style></head><body>
<h1>Voice room</h1>
<button onclick="connect()">Connect ESP32</button><button id="mic" onclick="listen()">🎤 Speak</button>
<p id="heard">Click Speak and say a command.</p><p id="said"></p>
<p>Try: "lamp on", "lamp off", "make it blue", "open the curtain", "close the curtain", "party time", "good night"</p>
<script>
const COLOURS = { red: [255, 0, 0], green: [0, 255, 0], blue: [0, 0, 255], yellow: [255, 160, 0], orange: [255, 70, 0],
purple: [140, 0, 255], pink: [255, 40, 120], white: [255, 255, 255] };
let writer, reader, buffer = '';
async function connect() {
const port = await navigator.serial.requestPort();
await port.open({ baudRate: 115200 });
writer = port.writable.getWriter();
const decoder = new TextDecoderStream();
port.readable.pipeTo(decoder.writable);
reader = decoder.readable.getReader();
while (true) { // answers from the ESP32
const { value, done } = await reader.read();
if (done) break;
buffer += value;
let i;
while ((i = buffer.indexOf('\n')) >= 0) {
const line = buffer.slice(0, i).trim();
buffer = buffer.slice(i + 1);
if (line.startsWith('OK ')) say(line.slice(3));
}
}
}
function send(text) {
if (writer) writer.write(new TextEncoder().encode(text + '\n'));
}
function say(text) { // speech synthesis: the computer talks
document.getElementById('said').textContent = text;
speechSynthesis.speak(new SpeechSynthesisUtterance(text));
}
// Understand a sentence: find the intent (what to do) and the details
function understand(text) {
const t = text.toLowerCase();
if (t.includes('good night')) return 'NIGHT';
if (t.includes('party')) return 'PARTY';
if (t.includes('curtain') || t.includes('window')) {
if (t.includes('half')) return 'CURTAIN 50';
if (t.includes('close') || t.includes('shut')) return 'CURTAIN 0';
if (t.includes('open')) return 'CURTAIN 100';
}
for (const name in COLOURS) {
if (t.includes(name)) return 'COLOUR ' + COLOURS[name].join(' ');
}
if (t.includes('lamp') || t.includes('light')) {
if (t.includes('off')) return 'LAMP 0';
if (t.includes('on')) return 'LAMP 1';
}
return null;
}
function listen() {
const Recognition = window.SpeechRecognition || window.webkitSpeechRecognition;
if (!Recognition) { say('Sorry, this browser has no speech recognition.'); return; }
const r = new Recognition();
r.lang = 'en-GB';
r.interimResults = false;
const mic = document.getElementById('mic');
mic.classList.add('on');
r.onresult = (e) => {
const text = e.results[0][0].transcript;
const confidence = Math.round(e.results[0][0].confidence * 100);
document.getElementById('heard').textContent = '"' + text + '" (' + confidence + ' % sure)';
const command = understand(text);
if (command) send(command);
else say('Sorry, I did not understand that.');
};
r.onend = () => mic.classList.remove('on');
r.start();
}
</script></body></html>Talk to your room
- Upload the sketch, close the Serial Monitor and open the page in Chrome or Edge.
- Click Connect ESP32 and pick its port.
- Click Speak, allow the microphone, and say "lamp on". The page shows what it heard, the lamp lights and the computer says "The lamp is on".
- Try the other commands. Change
r.langtoen-USor another language if it understands you better.
Privacy: in Chrome, speech recognition sends the sound of your voice to Google's servers to turn it into text. Only speak while the microphone button is red, and don't say anything private.
How it works
Speech recognition is a big AI model that has learned from huge amounts of recorded speech how sounds turn into words. It returns the most likely sentence and how sure it is.
Understanding the sentence is a second job. Assistants find the intent, what you want to happen, and the slots, the details: in "make it purple", the intent is "change colour" and the slot is "purple". The page does this with simple keyword rules in understand(). That is easy to follow, but it only knows the phrases you wrote rules for. The capstone uses a large language model instead, which understands almost any way of saying things.
Web Serial lets a web page talk to a board on a USB cable, after you pick the port and allow it. The page sends one line of text per message, and the ESP32 answers with lines of its own. A page opened from a file on your own computer counts as safe, so the browser also lets it use the camera and the microphone, which it wouldn't for a page served by the ESP32 over plain http.
Speech synthesis works the other way round: speechSynthesis.speak() turns the ESP32's answer into a voice.
Try this next
- Add "dim the lamp" with
analogWrite()and a brightness slot like "dim to 30". - Add a wake word: keep listening, and only act on sentences that start with "room".
- Teach it a second language.
Troubleshooting
- "No speech recognition": use Chrome or Edge on a computer.
-
It hears the wrong words: speak clearly, close to the microphone, and try another
r.lang. - It hears you but nothing happens: click Connect ESP32 again; the connection is lost when the ESP32 restarts.
- The board restarts when the servo moves: servos draw bursts of current. Use a good USB port or a powered hub, and check that the servo's red wire is on 5V.
- A printed part is too tight or too loose: printers differ by a few tenths of a millimetre. Sand or file a tight hole, and fix a loose part with a drop of hot glue or a strip of tape.
