Teach an ESP32-S3 camera to recognise objects, with AI on the board

Phones sort your photos by what is in them, shops count products on shelves with cameras, and recycling machines tell plastic from glass. All of them recognise objects. Most use big AI models that were trained on millions of pictures in a data centre. In this capstone you build a small recognising camera that learns on the board itself, from examples you show it, in a few minutes. The XIAO ESP32S3 Sense is a thumb-sized ESP32-S3 with a camera. It makes its own Wi-Fi network and shows its camera picture on your phone, with a button for each object. Show it an apple and tap "Apple" ten times, then a banana, then the empty table. From then on it tells them apart by itself, five times a second, and lights a different LED for each. Nothing is sent to the internet: the whole "AI" runs on a chip smaller than your fingernail.

What you'll learn

• How a picture turns into numbers (features) that a computer can compare
• How the k-nearest-neighbours method learns from examples and votes on a new picture
• Why the sketch divides out the brightness, and what "not sure" means for an AI
• How the ESP32-S3's PSRAM holds camera pictures and how to save what it learned in flash
• How to serve a live camera picture and teaching buttons from the board's own web page

What you need

• A Seeed Studio XIAO ESP32S3 Sense: the small ESP32-S3 board with the camera and microphone board that clips on top. Use a USB-C cable. It is not in the kit.
• 3 LEDs, for example red, yellow and green, and 3 × 220 Ω resistors (ESP32 IoT Starter Kit)
• Breadboard and jumper wires (ESP32 IoT Starter Kit), and 4 female-to-male jumper wires for the XIAO
• Three things to teach it, for example an apple, a banana and an empty table, and a lamp or a window for steady light
• A phone, tablet or laptop with Wi-Fi

About the XIAO ESP32S3 Sense: it is a thumb-sized ESP32-S3 board with 8 MB of extra memory (PSRAM). The Sense version has a second little board that clips onto the back: it carries the camera, a digital microphone and a microSD slot. Press it on firmly until it clicks. The pins along the sides are named D0 to D10; the sketch uses their GPIO numbers (D0 = GPIO 1, D1 = GPIO 2, D2 = GPIO 3). Stick the antenna onto the board's small round socket: without it the Wi-Fi range is very short.

Set up the XIAO ESP32S3 in the Arduino IDE

1. In the Arduino IDE, open Boards Manager and install or update esp32 by Espressif Systems to version 3 or newer. The camera, microphone, Wi-Fi and LittleFS support all come with it.
2. Choose Tools → Board → esp32 → XIAO_ESP32S3.
3. Set Tools → PSRAM to OPI PSRAM. This is important: without it the extra memory stays switched off and the sketch stops with a message.
4. Plug in the board and choose its port under Tools → Port.
5. Partition Scheme: leave the default, Default with spiffs (3MB APP/1.5MB SPIFFS). The "spiffs" part is where the board saves what it learned.
6. If an upload fails or the port disappears, hold the tiny B (BOOT) button, plug in the cable, let go, and upload again. Press the R (reset) button afterwards to start the sketch.

Wiring

The XIAO is usually sold without pins: if yours has none, solder on the two 7-pin headers that come with it (ask an adult to help), or buy the version with pins already soldered. The Sense board on the back gets in the way of a breadboard, so connect the XIAO with female-to-male jumper wires: the female ends go onto the XIAO's pins, the male ends into the breadboard, as in the picture.

Each LED gets its own 220 Ω resistor:

Build it step by step

Plug in all the parts first, then add the wires in this order. In each picture the new wires are bright and the ones from earlier steps are faded. Click a picture to see it larger.

The sketch

No extra libraries are needed: the camera support comes with the esp32 boards package. Change NAMES to the three things you want to teach (the third one works best as "Nothing", for an empty background), and AP_PASSWORD to your own password of at least 8 characters.

#include "esp_camera.h"
#include "img_converters.h"
#include <WiFi.h>
#include <WebServer.h>
#include <LittleFS.h>

// ---------- settings ----------
const char* AP_NAME = "ObjectLearner";              // the board makes its own Wi-Fi network
const char* AP_PASSWORD = "change-me-123";          // at least 8 characters
const char* NAMES[3] = {"Apple", "Banana", "Nothing"};   // what you will teach it: change these
const int LED_PINS[3] = {1, 2, 3};                  // D0, D1, D2: one LED for each object
const int K = 3;                                    // how many nearest examples vote
const uint32_t NOT_SURE = 90000;                    // farther than this from every example: "Not sure"
const int PER_OBJECT = 30;                          // examples per object
const int MAX_EXAMPLES = 3 * PER_OBJECT;

// The picture is cut into 8 x 6 squares. The average colour of each square (red, green, blue) is one feature.
const int GX = 8, GY = 6, FEATURES = GX * GY * 3;

uint8_t examples[MAX_EXAMPLES][FEATURES];           // everything the board has learned
uint8_t labels[MAX_EXAMPLES];
int exampleCount = 0;
int result = -1, votes = 0;                         // what it sees right now
uint32_t nearest = 0;
unsigned long lastGuess = 0;
WebServer server(80);

const char PAGE[] = R"rawliteral(<!doctype html><html><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1"><title>Object learner</title>
<style>
body{margin:0;font-family:Arial,sans-serif;background:#F6F8FC;color:#1B3A7B;max-width:520px}
header{background:#1B3A7B;color:#fff;padding:12px 16px;font-size:20px;font-weight:bold}
#cam{width:100%;aspect-ratio:4/3;background:#000;display:block}
#res{margin:12px;padding:14px;border-radius:16px;background:#fff;box-shadow:0 2px 8px rgba(27,58,123,.08);font-size:26px;font-weight:bold;text-align:center}
#res small{display:block;font-size:14px;font-weight:normal;opacity:.7;margin-top:4px}
.row{display:flex;gap:8px;margin:10px 12px}
button{flex:1;font-size:16px;font-weight:bold;padding:14px 4px;border:0;border-radius:14px;background:#1B3A7B;color:#fff}
button.light{background:#E3E8F2;color:#1B3A7B} button.red{background:#DE544A}
p{margin:8px 14px;font-size:14px;opacity:.75}
</style></head><body>
<header>Object learner</header>
<img id="cam" alt="">
<div id="res">-<small></small></div>
<div class="row" id="teach"></div>
<p>Hold an object in front of the camera and tap its button 10 times or more, moving it a little each time. Teach "Nothing" with an empty background.</p>
<div class="row"><button class="light" onclick="act('/save')">Save to flash</button><button class="red" id="clr" onclick="clearAll()">Forget all</button></div>
<script>
var cam = document.getElementById('cam'), names = [], clearTimer;
function next() { cam.src = '/frame?' + Date.now(); }
cam.onload = function () { setTimeout(next, 80); };
cam.onerror = function () { setTimeout(next, 1000); };
function act(url) { return fetch(url).then(status); }
function clearAll() {
  var b = document.getElementById('clr');
  if (b.textContent == 'Tap again') { b.textContent = 'Forget all'; act('/clear'); return; }
  b.textContent = 'Tap again'; clearTimeout(clearTimer);
  clearTimer = setTimeout(function () { b.textContent = 'Forget all'; }, 2500);
}
function status(r) {
  (r ? Promise.resolve(r) : fetch('/status')).then(function (x) { return x.json(); }).then(function (d) {
    if (!names.length) {
      names = d.names;
      document.getElementById('teach').innerHTML = names.map(function (n, i) { return '<button onclick="act(\'/teach?c=' + i + '\')">' + n + ' <span id="n' + i + '"></span></button>'; }).join('');
    }
    d.counts.forEach(function (c, i) { document.getElementById('n' + i).textContent = '(' + c + ')'; });
    var res = document.getElementById('res');
    res.firstChild.textContent = d.total == 0 ? 'Teach me something first' : d.result < 0 ? 'Not sure' : d.names[d.result];
    res.lastChild.textContent = d.total == 0 ? '' : d.votes + ' of ' + d.k + ' votes · distance ' + d.nearest + ' (not sure above ' + d.notSure + ')';
  });
}
next(); status(); setInterval(function () { status(); }, 500);
</script></body></html>)rawliteral";

// ---------- camera (XIAO ESP32S3 Sense pins) ----------
bool startCamera() {
  camera_config_t c = {};
  c.ledc_channel = LEDC_CHANNEL_0;
  c.ledc_timer = LEDC_TIMER_0;
  c.pin_d0 = 15; c.pin_d1 = 17; c.pin_d2 = 18; c.pin_d3 = 16;
  c.pin_d4 = 14; c.pin_d5 = 12; c.pin_d6 = 11; c.pin_d7 = 48;
  c.pin_xclk = 10; c.pin_pclk = 13; c.pin_vsync = 38; c.pin_href = 47;
  c.pin_sccb_sda = 40; c.pin_sccb_scl = 39;
  c.pin_pwdn = -1; c.pin_reset = -1;
  c.xclk_freq_hz = 20000000;
  c.pixel_format = PIXFORMAT_RGB565;                // raw colours, so the sketch can measure them
  c.frame_size = FRAMESIZE_QVGA;                    // 320 x 240
  c.fb_count = 2;
  c.fb_location = CAMERA_FB_IN_PSRAM;
  c.grab_mode = CAMERA_GRAB_LATEST;
  return esp_camera_init(&c) == ESP_OK;
}

// ---------- features: the average colour of each square, scaled by the brightness of the whole picture ----------
void getFeatures(camera_fb_t* fb, uint8_t* f) {
  int cw = fb->width / GX, ch = fb->height / GY;
  uint32_t sum[FEATURES] = {0}, all = 0;
  int n = 0;
  for (int gy = 0; gy < GY; gy++)
    for (int gx = 0; gx < GX; gx++) {
      uint32_t r = 0, g = 0, b = 0;
      n = 0;
      for (int y = gy * ch; y < (gy + 1) * ch; y += 2)
        for (int x = gx * cw; x < (gx + 1) * cw; x += 2) {   // every second pixel is enough
          const uint8_t* p = fb->buf + 2 * (y * fb->width + x);
          uint8_t hi = p[0], lo = p[1];                      // RGB565: rrrrrggg gggbbbbb
          r += hi & 0xF8;
          g += ((hi & 0x07) << 5) | ((lo & 0xE0) >> 3);
          b += (lo & 0x1F) << 3;
          n++;
        }
      int i = (gy * GX + gx) * 3;
      sum[i] = r / n; sum[i + 1] = g / n; sum[i + 2] = b / n;
      all += sum[i] + sum[i + 1] + sum[i + 2];
    }
  uint32_t mean = all / FEATURES + 1;                 // brighter room = bigger numbers everywhere: divide that out
  for (int i = 0; i < FEATURES; i++) f[i] = min(255UL, (unsigned long)(sum[i] * 128 / mean));
}

// ---------- k-nearest neighbours ----------
uint32_t distance(const uint8_t* a, const uint8_t* b) {
  uint32_t d = 0;
  for (int i = 0; i < FEATURES; i++) {
    int x = a[i] - b[i];
    d += x * x;
  }
  return d;
}

void classify(const uint8_t* f) {
  if (exampleCount == 0) { result = -1; votes = 0; nearest = 0; return; }
  uint32_t bestD[K];
  int bestL[K];
  for (int k = 0; k < K; k++) { bestD[k] = UINT32_MAX; bestL[k] = -1; }
  for (int i = 0; i < exampleCount; i++) {          // keep the K closest examples, closest first
    uint32_t d = distance(f, examples[i]);
    for (int k = 0; k < K; k++)
      if (d < bestD[k]) {
        for (int j = K - 1; j > k; j--) { bestD[j] = bestD[j - 1]; bestL[j] = bestL[j - 1]; }
        bestD[k] = d; bestL[k] = labels[i];
        break;
      }
  }
  int count[3] = {0, 0, 0};
  for (int k = 0; k < K; k++) if (bestL[k] >= 0) count[bestL[k]]++;
  int best = bestL[0];                             // a tie goes to the nearest example
  for (int c = 0; c < 3; c++) if (count[c] > count[best]) best = c;
  nearest = bestD[0];
  votes = count[best];
  result = nearest > NOT_SURE ? -1 : best;
}

void showLeds() {
  for (int c = 0; c < 3; c++) digitalWrite(LED_PINS[c], result == c ? HIGH : LOW);
}

// ---------- saving what it learned ----------
void saveExamples() {
  File f = LittleFS.open("/examples.bin", "w");
  if (!f) return;
  f.write((uint8_t*)&exampleCount, sizeof(exampleCount));
  f.write(labels, exampleCount);
  f.write((uint8_t*)examples, exampleCount * FEATURES);
  f.close();
}

void loadExamples() {
  File f = LittleFS.open("/examples.bin", "r");
  if (!f) { Serial.println("0 examples loaded from flash"); return; }
  int n = 0;
  if (f.read((uint8_t*)&n, sizeof(n)) == sizeof(n) && n > 0 && n <= MAX_EXAMPLES) {
    f.read(labels, n);
    if (f.read((uint8_t*)examples, n * FEATURES) == (size_t)(n * FEATURES)) exampleCount = n;
  }
  f.close();
  Serial.printf("%d examples loaded from flash\n", exampleCount);
}

// ---------- web pages ----------
void sendFrame() {                                  // the camera picture as a JPEG for the page
  camera_fb_t* fb = esp_camera_fb_get();
  if (!fb) { server.send(503, "text/plain", "no picture"); return; }
  uint8_t* jpg = NULL;
  size_t len = 0;
  bool ok = frame2jpg(fb, 70, &jpg, &len);
  esp_camera_fb_return(fb);
  if (!ok) { server.send(500, "text/plain", "jpeg failed"); return; }
  server.sendHeader("Cache-Control", "no-store");
  server.send_P(200, "image/jpeg", (const char*)jpg, len);
  free(jpg);
}

void sendStatus() {
  int counts[3] = {0, 0, 0};
  for (int i = 0; i < exampleCount; i++) counts[labels[i]]++;
  String json = "{\"names\":[\"" + String(NAMES[0]) + "\",\"" + NAMES[1] + "\",\"" + NAMES[2] + "\"],\"counts\":[" +
                counts[0] + "," + counts[1] + "," + counts[2] + "],\"total\":" + exampleCount + ",\"result\":" + result +
                ",\"votes\":" + votes + ",\"k\":" + K + ",\"nearest\":" + nearest + ",\"notSure\":" + NOT_SURE + "}";
  server.send(200, "application/json", json);
}

void teach() {
  int c = server.arg("c").toInt();
  camera_fb_t* fb = esp_camera_fb_get();
  if (fb && c >= 0 && c < 3) {
    int have = 0, oldest = -1;
    for (int i = 0; i < exampleCount; i++)
      if (labels[i] == c) { if (oldest < 0) oldest = i; have++; }
    if (have < PER_OBJECT) {                        // add a new example
      getFeatures(fb, examples[exampleCount]);
      labels[exampleCount++] = c;
    } else {                                        // full for this object: move the oldest to the end and refresh it
      memmove(examples[oldest], examples[oldest + 1], (exampleCount - oldest - 1) * FEATURES);
      memmove(labels + oldest, labels + oldest + 1, exampleCount - oldest - 1);
      getFeatures(fb, examples[exampleCount - 1]);
      labels[exampleCount - 1] = c;
    }
  }
  if (fb) esp_camera_fb_return(fb);
  sendStatus();
}

void setup() {
  Serial.begin(115200);
  for (int c = 0; c < 3; c++) pinMode(LED_PINS[c], OUTPUT);
  if (!psramFound()) {
    Serial.println("No PSRAM: set Tools > PSRAM to OPI PSRAM and upload again");
    while (true) delay(1000);
  }
  if (!startCamera()) {
    Serial.println("Camera not found: press the camera board on firmly");
    while (true) delay(1000);
  }
  LittleFS.begin(true);
  loadExamples();
  WiFi.softAP(AP_NAME, AP_PASSWORD);
  server.on("/", []() { server.send(200, "text/html", PAGE); });
  server.on("/frame", sendFrame);
  server.on("/status", sendStatus);
  server.on("/teach", teach);
  server.on("/save", []() { saveExamples(); sendStatus(); });
  server.on("/clear", []() { exampleCount = 0; LittleFS.remove("/examples.bin"); sendStatus(); });
  server.begin();
  Serial.printf("Join the Wi-Fi \"%s\", then open http://", AP_NAME);
  Serial.println(WiFi.softAPIP());
}

void loop() {
  server.handleClient();
  if (millis() - lastGuess >= 200) {                // look and guess 5 times a second
    lastGuess = millis();
    camera_fb_t* fb = esp_camera_fb_get();
    if (fb) {
      uint8_t f[FEATURES];
      getFeatures(fb, f);
      esp_camera_fb_return(fb);
      classify(f);
      showLeds();
    }
  }
}

Try it

1. Upload the sketch and open the Serial Monitor at 115200 baud. It says "0 examples loaded from flash" and which Wi-Fi to join.
2. On your phone, join the ObjectLearner Wi-Fi with your password (choose to stay connected if the phone says there is no internet), then open http://192.168.4.1. You see the camera picture, which updates several times a second.
3. Point the camera at the empty table and tap Nothing about 10 times, moving the camera a little between taps.
4. Hold the apple in front of the camera, about 20 cm away, and tap Apple 10 times. Turn it a little and move it around the picture between taps. Do the same with the banana.
5. Now show it the apple again: the page says "Apple", with "3 of 3 votes", and the first LED lights up. Swap to the banana, and to the empty table. Watch the distance number: small means very similar to an example.
6. Try to fool it: hold the apple very close, or show it something it has never seen. When nothing it knows is close enough, the page says Not sure and all LEDs go off. If it gets something wrong, teach a few more examples of the right object in exactly that position.
7. Tap Save to flash. Unplug the board and plug it in again: the Serial Monitor says "33 examples loaded from flash" (or however many you taught), and it recognises the objects straight away.

What you should see

How it works

To a computer, a camera picture is just a long list of numbers: 320 × 240 pixels, each with a red, green and blue value. Comparing two pictures pixel by pixel would be slow, and useless: move the apple by one centimetre and almost every pixel changes. So the sketch first makes the picture simpler. It cuts it into 8 × 6 squares and works out the average colour of each square. That gives 144 numbers, the features: a very blurry, tiny version of the picture. A red blob in the middle looks similar whether the apple is a bit to the left or right.

Light is the next problem: the same apple near a window gives much bigger numbers than in a dark corner. So the sketch divides every feature by the average brightness of the whole picture. What is left is mostly the colours and the shape, not how bright the room is.

Teaching simply stores the features of the picture together with its label: "these 144 numbers are an Apple". That is all the learning there is. To recognise a new picture, the sketch measures the distance to every stored example: it subtracts the features one by one, squares the differences and adds them up. Then the 3 nearest examples vote. This method is called k-nearest neighbours (k-NN), and with k = 3 one wrong example can never win on its own. If even the nearest example is farther away than NOT_SURE, the camera admits that it doesn't know. That is an important habit for any AI: a guess on something it has never seen is worse than "Not sure".

Why the ESP32-S3? A raw 320 × 240 picture takes 150 KB, and the sketch keeps two of them while it works. That is more than the chip's own memory can spare, so they go into the 8 MB PSRAM on the XIAO. The S3 is also fast enough to compare a picture with 90 examples five times a second. Big image AIs (neural networks) learn their own features from millions of pictures and recognise thousands of things, but they need a computer to train. k-NN learns in a second, right here, from your own examples, and you can see exactly why it made each decision.

Try this next

• Use more squares, for example 16 × 12, and see whether it gets better at small objects. (Remember that each example then takes 576 bytes.)
• Make it a sorting machine: add a servo, as in the robot arm, that points left for apples and right for bananas.
• Count things: every time the result changes from "Nothing" to "Apple", add one to a counter on the page.
• Teach it hand signs (a fist, an open hand, a thumbs up) instead of objects, and use them to control the LEDs.
• Compare with the colour detector that learns from the AI path: that one uses one colour sensor, this one uses 48 squares of colour.

Troubleshooting

• "No PSRAM" in the Serial Monitor: set Tools → PSRAM to OPI PSRAM and upload again.
• "Camera not found": the camera board isn't pressed on firmly. Unplug, press the Sense board onto the back of the XIAO until it clicks, and check that the camera's ribbon cable is in its socket.
• The page doesn't open or the picture is black: check that the phone is still on the ObjectLearner Wi-Fi (switch off mobile data for a moment) and that the antenna is stuck on. Type http://192.168.4.1 with http, not https.
• It always says the same thing: teach more examples of each object, from different angles, and make sure the "Nothing" examples really show the empty background.
• It says "Not sure" too often: make NOT_SURE bigger (for example 150000). If it guesses wrong things instead of saying "Not sure", make it smaller.
• It worked in the morning but not in the evening: the colour of the light changed (sunlight is blue-white, lamps are yellow). Teach a few examples in both kinds of light.
• The XIAO gets warm: that is normal: the camera and Wi-Fi run all the time.
• The port disappeared and uploads fail: hold the B button while plugging in the cable, let go, and upload again.