"Hey, Siri", "OK Google", "Alexa": smart speakers listen all the time for one special sound, and they recognise it on the device itself, without sending everything you say to the internet. In this capstone you build a small version of that idea. The XIAO ESP32S3 Sense has a tiny digital microphone. The sketch waits for a sound, records it for about a third of a second, and splits it into pitches with a mathematical trick called the FFT. A clap, a whistle and a finger snap look very different that way. You teach it from its own web page: tap "Teach Clap", clap, and repeat a few times for each sound. From then on, every clap switches the red of an RGB LED, every whistle the green and every snap the blue. And on the page you can watch each sound's "fingerprint".
What you'll learn
- How a digital (PDM) microphone delivers sound as numbers, 16 000 times a second
- How an FFT splits a sound into pitches, and why a window function helps
- How to detect the start of a sound by comparing it with the background noise
- How to turn a sound into a small fingerprint of 24 numbers that doesn't depend on how loud it was
- How k-nearest neighbours learns sounds from a few examples, and when it should say "not sure"
What you need
- A Seeed Studio XIAO ESP32S3 Sense: the small ESP32-S3 board with the camera and microphone board that clips on top. Use a USB-C cable. It is not in the kit.
- RGB LED (common cathode) and 3 × 220 Ω resistors (ESP32 IoT Starter Kit)
- Breadboard and jumper wires (ESP32 IoT Starter Kit), and 4 female-to-male jumper wires for the XIAO
- A phone, tablet or laptop with Wi-Fi
About the XIAO ESP32S3 Sense: it is a thumb-sized ESP32-S3 board with 8 MB of extra memory (PSRAM). The Sense version has a second little board that clips onto the back: it carries the camera, a digital microphone and a microSD slot. Press it on firmly until it clicks. The pins along the sides are named D0 to D10; the sketch uses their GPIO numbers (D0 = GPIO 1, D1 = GPIO 2, D2 = GPIO 3). Stick the antenna onto the board's small round socket: without it the Wi-Fi range is very short.
Set up the XIAO ESP32S3 in the Arduino IDE
- In the Arduino IDE, open Boards Manager and install or update esp32 by Espressif Systems to version 3 or newer. The camera, microphone, Wi-Fi and LittleFS support all come with it.
- Choose Tools → Board → esp32 → XIAO_ESP32S3.
- Set Tools → PSRAM to OPI PSRAM. This is important: without it the extra memory stays switched off and the sketch stops with a message.
- Plug in the board and choose its port under Tools → Port.
- Partition Scheme: leave the default, Default with spiffs (3MB APP/1.5MB SPIFFS). The "spiffs" part is where the board saves the sounds it learned.
- If an upload fails or the port disappears, hold the tiny B (BOOT) button, plug in the cable, let go, and upload again. Press the R (reset) button afterwards to start the sketch.
Wiring
The XIAO is usually sold without pins: if yours has none, solder on the two 7-pin headers that come with it (ask an adult to help), or buy the version with pins already soldered. The Sense board on the back gets in the way of a breadboard, so connect the XIAO with female-to-male jumper wires: the female ends go onto the XIAO's pins, the male ends into the breadboard, as in the picture.
The RGB LED has four legs; the longest one is the common cathode (−):
Build it step by step
Plug in all the parts first, then add the wires in this order. In each picture the new wires are bright and the ones from earlier steps are faded. Click a picture to see it larger.
| From | To |
|---|---|
| D0 (GPIO 1) | 220 Ω resistor → RGB LED red leg |
| D1 (GPIO 2) | 220 Ω resistor → RGB LED green leg |
| D2 (GPIO 3) | 220 Ω resistor → RGB LED blue leg |
| RGB LED longest leg (−) | XIAO GND |
If your RGB LED is a module with the resistors already on it, connect its R, G and B pins straight to D0, D1 and D2, and its − pin to GND.
The sketch
No extra libraries are needed: the microphone driver (ESP_I2S) comes with the esp32 boards package. Change AP_PASSWORD to your own password of at least 8 characters. You can also change NAMES to other short sounds, such as a knock or a "shh".
#include <ESP_I2S.h>
#include <WiFi.h>
#include <WebServer.h>
#include <LittleFS.h>
// ---------- settings ----------
const char* AP_NAME = "SoundLearner"; // the board makes its own Wi-Fi network
const char* AP_PASSWORD = "change-me-123"; // at least 8 characters
const char* NAMES[3] = {"Clap", "Whistle", "Snap"}; // the three sounds it learns
const int RGB_PINS[3] = {1, 2, 3}; // D0 red, D1 green, D2 blue of the RGB LED
const float TRIGGER = 6.0; // a sound starts when it is 6 times louder than the background
const float MIN_LEVEL = 2000; // ... and louder than this (see "Level" on the page)
const float NOT_SURE = 4.0; // farther than this from every example: "Not sure"
const int K = 3;
const int PER_SOUND = 20; // examples per sound
const int MAX_EXAMPLES = 3 * PER_SOUND;
// ---------- sound settings ----------
const int RATE = 16000; // 16 000 samples a second
const int N = 256; // one frame = 256 samples = 16 ms
const int FRAMES = 24; // one sound = 24 frames = 0.38 s
const int BANDS = 16, PARTS = 8; // 16 pitch bands, the loudness in 8 time slices
const int FEATURES = BANDS + PARTS;
I2SClass mic;
WebServer server(80);
int16_t samples[N];
float re[N], im[N];
float background = 0, level = 0;
int bandStart[BANDS + 1];
float examples[MAX_EXAMPLES][FEATURES];
uint8_t labels[MAX_EXAMPLES];
int exampleCount = 0;
int teachNext = -1; // the next sound is an example of this one
bool ledOn[3] = {false, false, false};
float lastF[FEATURES];
int lastResult = -2, lastVotes = 0, soundCount = 0;
float lastDist = 0;
const char PAGE[] = R"rawliteral(<!doctype html><html><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1"><title>Sound learner</title>
<style>
body{margin:0;font-family:Arial,sans-serif;background:#F6F8FC;color:#1B3A7B;max-width:520px}
header{background:#1B3A7B;color:#fff;padding:12px 16px;font-size:20px;font-weight:bold}
.card{background:#fff;margin:12px;padding:12px 14px;border-radius:16px;box-shadow:0 2px 8px rgba(27,58,123,.08)}
#res{font-size:28px;font-weight:bold;text-align:center} #res small{display:block;font-size:14px;font-weight:normal;opacity:.7}
canvas{width:100%;height:150px} .lbl{font-size:13px;opacity:.7}
.meter{height:14px;background:#E3E8F2;border-radius:7px;position:relative;overflow:hidden}
#lv{height:100%;background:#1E9E5A;width:0} #th{position:absolute;top:0;bottom:0;width:3px;background:#DE544A}
.row{display:flex;gap:8px;margin:10px 12px}
button{flex:1;font-size:16px;font-weight:bold;padding:14px 4px;border:0;border-radius:14px;background:#1B3A7B;color:#fff}
button.on{background:#DE544A} button.light{background:#E3E8F2;color:#1B3A7B}
</style></head><body>
<header>Sound learner</header>
<div class="card"><div id="res">Make a sound<small></small></div></div>
<div class="card"><div class="lbl">The last sound: 16 pitch bands (low to high), then its loudness over time</div><canvas id="c"></canvas>
<div class="lbl">Level now (red line: start of a sound)</div><div class="meter"><div id="lv"></div><div id="th"></div></div></div>
<div class="row" id="teach"></div>
<div class="row"><button class="light" onclick="act('/save')">Save to flash</button><button class="light" onclick="act('/clear')">Forget all</button></div>
<script>
var names = [];
function act(u) { fetch(u).then(function () { update(); }); }
function draw(f) {
var c = document.getElementById('c'), w = c.width = c.clientWidth * 2, h = c.height = c.clientHeight * 2, g = c.getContext('2d');
g.clearRect(0, 0, w, h);
if (!f) return;
var bw = w / f.length;
f.forEach(function (v, i) {
var y = i < 16 ? (v + 3) / 6 : v;
y = Math.max(0.02, Math.min(1, y));
g.fillStyle = i < 16 ? '#2F6FDB' : '#1E9E5A';
g.fillRect(i * bw + 3, h - y * (h - 10), bw - 6, y * (h - 10));
});
}
function update() {
fetch('/status').then(function (r) { return r.json(); }).then(function (d) {
if (!names.length) {
names = d.names;
document.getElementById('teach').innerHTML = names.map(function (n, i) { return '<button id="b' + i + '" onclick="act(\'/teach?c=' + i + '\')">Teach ' + n + ' <span id="n' + i + '"></span></button>'; }).join('');
}
d.counts.forEach(function (c, i) {
document.getElementById('n' + i).textContent = '(' + c + ')';
document.getElementById('b' + i).className = d.teach == i ? 'on' : '';
});
var r = document.getElementById('res');
r.firstChild.textContent = d.teach >= 0 ? 'Now make a ' + names[d.teach].toLowerCase() + '...' : d.result == -2 ? 'Make a sound' : d.result == -1 ? 'Not sure' : names[d.result];
r.lastChild.textContent = d.result > -2 && d.total > 0 ? 'Sound ' + d.sounds + ' · ' + d.votes + ' of ' + d.k + ' votes · distance ' + d.dist.toFixed(2) : d.total == 0 ? 'Teach me first: tap a Teach button, then make that sound.' : '';
document.getElementById('lv').style.width = Math.min(100, 100 * Math.log10(1 + d.level) / 6) + '%';
document.getElementById('th').style.left = Math.min(100, 100 * Math.log10(1 + d.trigger) / 6) + '%';
draw(d.features);
});
}
update(); setInterval(update, 300);
</script></body></html>)rawliteral";
// ---------- FFT: splits 256 samples into 128 pitches ----------
void fft(float* x, float* y, int n) {
for (int i = 1, j = 0; i < n; i++) { // reorder the samples
int bit = n >> 1;
for (; j & bit; bit >>= 1) j ^= bit;
j ^= bit;
if (i < j) { float t = x[i]; x[i] = x[j]; x[j] = t; t = y[i]; y[i] = y[j]; y[j] = t; }
}
for (int len = 2; len <= n; len <<= 1) { // combine ever bigger groups
float a = -2 * PI / len, wr = cos(a), wi = sin(a);
for (int i = 0; i < n; i += len) {
float cr = 1, ci = 0;
for (int k = 0; k < len / 2; k++) {
int p = i + k, q = p + len / 2;
float tr = x[q] * cr - y[q] * ci, ti = x[q] * ci + y[q] * cr;
x[q] = x[p] - tr; y[q] = y[p] - ti;
x[p] += tr; y[p] += ti;
float nr = cr * wr - ci * wi;
ci = cr * wi + ci * wr;
cr = nr;
}
}
}
}
// Reads one frame from the microphone. Returns its loudness, and adds its pitch bands to bands[] if it isn't NULL.
float readFrame(float* bands) {
mic.readBytes((char*)samples, sizeof(samples));
float mean = 0;
for (int i = 0; i < N; i++) mean += samples[i];
mean /= N;
float energy = 0;
for (int i = 0; i < N; i++) {
float s = samples[i] - mean; // remove the offset
energy += s * s;
re[i] = s * (0.5 - 0.5 * cos(2 * PI * i / (N - 1))); // Hann window: fade the edges of the frame
im[i] = 0;
}
energy /= N;
if (bands) {
fft(re, im, N);
for (int b = 0; b < BANDS; b++)
for (int k = bandStart[b]; k < bandStart[b + 1]; k++) bands[b] += re[k] * re[k] + im[k] * im[k];
}
return energy;
}
// Records one whole sound and turns it into 24 numbers: its pitch shape and its loudness over time.
void recordSound(float firstEnergy, float* f) {
float bands[BANDS] = {0}, parts[PARTS] = {0};
fft(re, im, N); // the first frame is still in re[] and im[]: use it too
for (int b = 0; b < BANDS; b++)
for (int k = bandStart[b]; k < bandStart[b + 1]; k++) bands[b] += re[k] * re[k] + im[k] * im[k];
parts[0] = firstEnergy;
for (int fr = 1; fr < FRAMES; fr++) parts[fr * PARTS / FRAMES] += readFrame(bands);
float avg = 0;
for (int b = 0; b < BANDS; b++) { f[b] = log10(bands[b] + 1); avg += f[b]; }
avg /= BANDS;
for (int b = 0; b < BANDS; b++) f[b] -= avg; // pitch SHAPE, not how loud
float top = 1;
for (int p = 0; p < PARTS; p++) top = max(top, parts[p]);
for (int p = 0; p < PARTS; p++) f[BANDS + p] = parts[p] / top; // 1 = the loudest moment
}
float distance(const float* a, const float* b) {
float d = 0;
for (int i = 0; i < FEATURES; i++) d += (a[i] - b[i]) * (a[i] - b[i]) * (i < BANDS ? 1 : 2); // timing counts double
return sqrt(d);
}
void classify(const float* f) {
float bestD[K];
int bestL[K];
for (int k = 0; k < K; k++) { bestD[k] = 1e9; bestL[k] = -1; }
for (int i = 0; i < exampleCount; i++) {
float d = distance(f, examples[i]);
for (int k = 0; k < K; k++)
if (d < bestD[k]) {
for (int j = K - 1; j > k; j--) { bestD[j] = bestD[j - 1]; bestL[j] = bestL[j - 1]; }
bestD[k] = d; bestL[k] = labels[i];
break;
}
}
int count[3] = {0, 0, 0};
for (int k = 0; k < K; k++) if (bestL[k] >= 0) count[bestL[k]]++;
int best = bestL[0]; // a tie goes to the nearest example
for (int c = 0; c < 3; c++) if (count[c] > count[best]) best = c;
lastDist = bestD[0];
lastVotes = count[best];
lastResult = (exampleCount == 0 || lastDist > NOT_SURE) ? -1 : best;
}
void handleSound(float firstEnergy) {
recordSound(firstEnergy, lastF);
soundCount++;
digitalWrite(LED_BUILTIN, LOW); // the XIAO's own orange LED blinks for every sound
if (teachNext >= 0) { // learn it
int have = 0, oldest = -1;
for (int i = 0; i < exampleCount; i++)
if (labels[i] == teachNext) { if (oldest < 0) oldest = i; have++; }
if (have >= PER_SOUND) { // full for this sound: drop its oldest example
memmove(examples[oldest], examples[oldest + 1], (exampleCount - oldest - 1) * sizeof(examples[0]));
memmove(labels + oldest, labels + oldest + 1, exampleCount - oldest - 1);
exampleCount--;
}
memcpy(examples[exampleCount], lastF, sizeof(lastF));
labels[exampleCount++] = teachNext;
lastResult = teachNext; lastVotes = K; lastDist = 0;
Serial.printf("Learned a %s\n", NAMES[teachNext]);
teachNext = -1;
} else { // recognise it and switch its colour
classify(lastF);
if (lastResult >= 0) {
ledOn[lastResult] = !ledOn[lastResult];
digitalWrite(RGB_PINS[lastResult], ledOn[lastResult]);
}
if (exampleCount == 0) Serial.println("Heard a sound: teach me first");
else Serial.printf("Heard: %s (distance %.2f)\n", lastResult >= 0 ? NAMES[lastResult] : "not sure", lastDist);
}
delay(60);
digitalWrite(LED_BUILTIN, HIGH);
for (int i = 0; i < 12; i++) readFrame(NULL); // skip 0.2 s of echo
}
// ---------- saving and the web page ----------
void saveExamples() {
File f = LittleFS.open("/sounds.bin", "w");
if (!f) return;
f.write((uint8_t*)&exampleCount, sizeof(exampleCount));
f.write(labels, exampleCount);
f.write((uint8_t*)examples, exampleCount * sizeof(examples[0]));
f.close();
}
void loadExamples() {
File f = LittleFS.open("/sounds.bin", "r");
if (!f) { Serial.println("0 examples loaded from flash"); return; }
int n = 0;
if (f.read((uint8_t*)&n, sizeof(n)) == sizeof(n) && n > 0 && n <= MAX_EXAMPLES) {
f.read(labels, n);
if (f.read((uint8_t*)examples, n * sizeof(examples[0])) == n * sizeof(examples[0])) exampleCount = n;
}
f.close();
Serial.printf("%d examples loaded from flash\n", exampleCount);
}
void sendStatus() {
int counts[3] = {0, 0, 0};
for (int i = 0; i < exampleCount; i++) counts[labels[i]]++;
String json = "{\"names\":[\"" + String(NAMES[0]) + "\",\"" + NAMES[1] + "\",\"" + NAMES[2] + "\"],\"counts\":[" +
counts[0] + "," + counts[1] + "," + counts[2] + "],\"total\":" + exampleCount + ",\"teach\":" + teachNext +
",\"result\":" + lastResult + ",\"votes\":" + lastVotes + ",\"k\":" + K + ",\"dist\":" + String(lastDist, 2) +
",\"sounds\":" + soundCount + ",\"level\":" + String(level, 0) + ",\"trigger\":" + String(max(background * TRIGGER, MIN_LEVEL), 0) +
",\"features\":";
if (soundCount == 0) json += "null";
else {
json += "[";
for (int i = 0; i < FEATURES; i++) json += (i ? "," : "") + String(lastF[i], 2);
json += "]";
}
server.send(200, "application/json", json + "}");
}
void setup() {
Serial.begin(115200);
pinMode(LED_BUILTIN, OUTPUT);
digitalWrite(LED_BUILTIN, HIGH); // HIGH = off on the XIAO
for (int c = 0; c < 3; c++) pinMode(RGB_PINS[c], OUTPUT);
for (int b = 0; b <= BANDS; b++) // 125 Hz ... 8 kHz, spaced like musical notes, at least 1 pitch wide
bandStart[b] = max(b ? bandStart[b - 1] + 1 : 2, (int)round(2 * pow(64.0, (float)b / BANDS)));
bandStart[BANDS] = N / 2;
mic.setPinsPdmRx(42, 41); // the Sense board's microphone: clock 42, data 41
if (!mic.begin(I2S_MODE_PDM_RX, RATE, I2S_DATA_BIT_WIDTH_16BIT, I2S_SLOT_MODE_MONO)) {
Serial.println("Microphone not found: press the Sense board on firmly");
while (true) delay(1000);
}
LittleFS.begin(true);
loadExamples();
WiFi.softAP(AP_NAME, AP_PASSWORD);
server.on("/", []() { server.send(200, "text/html", PAGE); });
server.on("/status", sendStatus);
server.on("/teach", []() { int c = server.arg("c").toInt(); teachNext = (c >= 0 && c < 3) ? c : -1; sendStatus(); });
server.on("/save", []() { saveExamples(); sendStatus(); });
server.on("/clear", []() { exampleCount = 0; LittleFS.remove("/sounds.bin"); sendStatus(); });
server.begin();
Serial.printf("Join the Wi-Fi \"%s\", then open http://", AP_NAME);
Serial.println(WiFi.softAPIP());
for (int i = 0; i < 30; i++) background += readFrame(NULL) / 30; // learn how loud the room is
}
void loop() {
server.handleClient();
level = readFrame(NULL);
if (level > background * TRIGGER && level > MIN_LEVEL) handleSound(level);
else background = background * 0.98 + level * 0.02; // follow slow changes in the room
}
Try it
- Upload the sketch and open the Serial Monitor at 115200 baud. Be quiet for the first second after a restart: the board is measuring how loud the room is.
- On your phone, join the SoundLearner Wi-Fi with your password (choose to stay connected if it says there is no internet) and open
http://192.168.4.1. The level bar moves when you talk. - Tap Teach Clap: the button turns red and the page says "Now make a clap...". Clap once, about 50 cm from the board. The XIAO's small orange LED blinks, the counter on the button goes to (1) and the chart shows the clap's fingerprint. Repeat until you have 5 to 8 claps.
- Do the same for Whistle (a short whistle, about half a second) and Snap (a finger snap). Look at the blue bars: a whistle has one tall bar, a clap and a snap spread over many bands, and the green loudness bars show that a whistle lasts longer.
- Now just make the sounds. A clap switches the red of the RGB LED on or off, a whistle the green, a snap the blue. Mix them for other colours: red and green make yellow.
- Talk, cough or knock on the table: if it isn't close to anything it learned, the page says Not sure and nothing switches. If it confuses two sounds, teach a few more examples of each.
- Tap Save to flash, so it still knows your sounds after a restart.
What you should see
How it works
Sound is the air pressure moving quickly up and down. The microphone on the Sense board measures that pressure and sends it as a fast stream of 1s and 0s called PDM; the ESP32-S3's I2S hardware turns this into ordinary numbers, 16 000 a second. The sketch reads them in frames of 256 samples, which is 16 ms each.
To notice a sound, the sketch keeps track of the background: the average loudness of the room, which it updates slowly. When a frame is 6 times louder than that (and louder than MIN_LEVEL), a sound has started. The sketch then records 24 frames, about 0.4 seconds, and stops listening for 0.2 seconds afterwards, so the echo from the walls doesn't count as a second sound.
Next it needs a fingerprint. The FFT (Fast Fourier Transform) takes the 256 samples of a frame and works out how much of each pitch is in them, from low to high, in 128 steps. Before that, each frame is faded in and out with a Hann window, because a sudden start and end would look like extra high pitches. The sketch then groups the 128 pitches into 16 bands that get wider towards the high notes, like the keys of a piano, and adds them up over the whole sound. A whistle puts almost everything into one band; a clap and a snap spread over many, and a snap has more high pitches than a clap. The sketch uses the logarithm and subtracts the average, so a loud clap and a soft clap get the same shape. Eight more numbers describe the loudness over time: a clap and a snap are over in a moment, and a whistle goes on. That makes 24 numbers in all.
Recognising then works like the object camera from the previous capstone: k-nearest neighbours. The sketch measures the distance between the new fingerprint and every example you taught, and the 3 nearest examples vote. The timing numbers count double, because they are very good at telling a whistle from the others. Big speech recognisers use the same first steps (frames, FFT, bands that get wider at the top), but then feed them into a neural network trained on thousands of hours of speech. Yours learns from about 20 examples in a minute, right on the board.
Try this next
- Add a fourth sound (a knock on the table, a "shh") by making the arrays one bigger and adding a name.
- Make it a clap switch for a lamp: use the relay module from the kit to switch a USB desk lamp (low voltage only, never mains) with a double clap.
- Show the full 128-pitch spectrum live on the page, so you can see your own voice change when you sing high and low.
- Send each recognised sound to the ESP-NOW hub, so a clap in one room switches something in another.
- Combine it with the camera that learns: let a whistle tell the camera to take a photo.
Troubleshooting
- "Microphone not found": press the Sense board firmly onto the back of the XIAO until it clicks, and check that you chose the XIAO_ESP32S3 board.
-
Nothing happens when you clap: look at the level bar on the page. If the green bar never reaches the red line, make
TRIGGERsmaller (for example 4) orMIN_LEVELsmaller. Make sure the microphone hole on the Sense board isn't covered. -
It reacts to talking and every little noise: make
TRIGGERbigger (for example 10), orMIN_LEVELbigger. - It mixes up two sounds: teach 3 to 5 more examples of each, at the same distance you will use later. Make the whistle short, about half a second.
-
It says "Not sure" for sounds it should know: make
NOT_SUREbigger (for example 6). If it switches LEDs for sounds it never learned, make it smaller. - One colour of the LED never lights up: check that LED leg's resistor and wire. If all three behave the wrong way round (on when they should be off), your RGB LED is common anode: connect its long leg to 3V3 instead of GND.
- The page doesn't open: check that the phone is still on the SoundLearner Wi-Fi (switch off mobile data for a moment), and that the antenna is stuck on.
- The port disappeared and uploads fail: hold the B button while plugging in the cable, let go, and upload again.
