Imagine you can't see, and you want to know what is in front of you: is the door open, which way is the exit, what does the sign say, is that a chair or a bag? Apps on phones that describe pictures have already changed daily life for many blind people. In this capstone you build the same idea into a pair of talking camera glasses. A thumb-sized XIAO ESP32S3 Sense with a camera sits in the bridge of 3D-printed glasses, a button sits on the left arm and a small speaker on the right one. Press the button and the glasses tell you what is in front of you, obstacles first. Or hold the button and ask your own question out loud, such as "Is the door open?" or "What does the sign say?". The glasses connect to the internet through your phone's hotspot, so they work anywhere your phone does. The thinking is done by AI in the cloud: Claude looks at the picture, and OpenAI's speech models turn your voice into text and the answer back into speech. It is the most complex project on this path, and a good one for learning how to design technology that people may rely on.
What you'll learn
- How the ESP32-S3 takes a JPEG picture and records your voice into its 8 MB of PSRAM
- How to send a picture to an AI vision model (Claude) and get a short spoken answer back
- How speech-to-text and text-to-speech work through an API, and how a WAV file is built
- How to drive a digital I2S amplifier and speaker from the ESP32
- How to write a system prompt for an assistive device: obstacles first, honest when unsure, never "it is safe"
- What privacy and safety mean when a device sends pictures of other people to the cloud
What you need
- A Seeed Studio XIAO ESP32S3 Sense: the small ESP32-S3 board with the camera and microphone board that clips on top. Use a USB-C cable. It is not in the kit.
- MAX98357A I2S amplifier board (from Adafruit or a compatible copy) and a small 8 Ω speaker, about 28 mm across and 0.5 to 1 W. They are not in the kit.
- Push button, breadboard and jumper wires (ESP32 IoT Starter Kit), and thin flexible wire for the glasses
- A small USB power bank and a USB-C cable, ideally with an angled plug
- A phone that can share its internet as a hotspot
- API keys for Claude (Anthropic) and OpenAI, set up by an adult (see below)
- The printed parts from the 3D printing box below (glasses front and two arms), or the cardboard version described there
- A soldering iron, and an adult to help with it
Safety and privacy: this is a learning project, not a medical device or a mobility aid. Never use it instead of a white cane, a guide dog or a person guiding you, and never to decide whether it is safe to cross a road or use stairs. Each answer takes 10 to 20 seconds, the AI can miss things or get them wrong, and its distances are only rough guesses: one camera can't measure distance. Test it slowly, indoors, with your eyes open first and a friend next to you. The glasses send a photo to the cloud every time you press the button: tell people around you, don't point them at people who haven't agreed, and don't use them where photos aren't allowed (changing rooms, schools during lessons, some shops and museums). This design has not been tested with blind users: don't give it to anyone to rely on.
About the XIAO ESP32S3 Sense: it is a thumb-sized ESP32-S3 board with 8 MB of extra memory (PSRAM). The Sense version has a second little board that clips onto the back: it carries the camera, a digital microphone and a microSD slot. Press it on firmly until it clicks. The pins along the sides are named D0 to D10; the sketch uses their GPIO numbers (D0 = GPIO 1, D1 = GPIO 2, D2 = GPIO 3). Stick the antenna onto the board's small round socket: without it the Wi-Fi range is very short.
Set up the XIAO ESP32S3 in the Arduino IDE
- In the Arduino IDE, open Boards Manager and install or update esp32 by Espressif Systems to version 3 or newer. The camera, microphone, Wi-Fi and LittleFS support all come with it.
- Choose Tools → Board → esp32 → XIAO_ESP32S3.
- Set Tools → PSRAM to OPI PSRAM. This is important: without it the extra memory stays switched off and the sketch stops with a message.
- Plug in the board and choose its port under Tools → Port.
- If an upload fails or the port disappears, hold the tiny B (BOOT) button, plug in the cable, let go, and upload again. Press the R (reset) button afterwards to start the sketch.
Wiring
Build and test everything on the breadboard first; it goes into the glasses afterwards. The Sense board on the back of the XIAO gets in the way of a breadboard, so connect it with female-to-male jumper wires for now. The MAX98357A is a tiny digital amplifier: the ESP32 sends it the sound as numbers over three wires (the I2S bus), and it drives the speaker directly.

Build it step by step
Plug in all the parts first, then add the wires in this order. In each picture the new wires are bright and the ones from earlier steps are faded. Click a picture to see it larger.



| From | To |
|---|---|
| MAX98357A LRC | XIAO D2 (GPIO 3) |
| MAX98357A BCLK | XIAO D1 (GPIO 2) |
| MAX98357A DIN | XIAO D3 (GPIO 4) |
| MAX98357A Vin | XIAO 5V |
| MAX98357A GND | XIAO GND |
| MAX98357A GAIN and SD | Not connected |
| Speaker | The amplifier's + and − screw terminal (or solder pads) |
| Push button | One leg to XIAO D0 (GPIO 1), the other to GND |
The XIAO's 5V pin carries the power from the USB cable, so the amplifier gets 5 V from the power bank. The camera and the microphone are already on the Sense board: they need no wires.
Build this circuit in EEMENTOR Lab
Wire it yourself in your browser before you touch a real wire. No sign-up, nothing to install.
Build it yourself
- The ESP32 and this lesson’s parts are already placed. Wire them using the wiring table above. Pins with no wire show red.
- Not in the Lab yet: camera, speaker and amplifier. Build the rest of the circuit without it.
- Press Run, then try the inputs: click, hold or slide them under Selected.
- Wi-Fi and Bluetooth don’t run in the Lab yet, so use it to build the circuit and check the wiring before you upload the real sketch.
EEMENTOR Lab is free and in beta. Results are a guide; a real build can differ slightly. About the Lab
Set up the phone hotspot
- On your phone, switch on the personal hotspot (iPhone: Settings → Personal Hotspot) or mobile hotspot (Android: Settings → Network → Hotspot). Note its name and password.
- The ESP32 only works with 2.4 GHz Wi-Fi. On an iPhone, switch on Maximise Compatibility; on Android, set the hotspot band to 2.4 GHz.
- Some iPhones switch the hotspot off when nothing has been connected for a while. Open the hotspot screen before you switch on the glasses.
- Each question uses about 1 to 1.5 MB of mobile data: your voice and the picture going up, and above all the spoken answer coming down.
Get the two API keys
Two AI companies do the work. Claude (by Anthropic) looks at the picture and writes the answer: it can see, but it can't hear or speak. OpenAI turns your voice into text and reads the answer aloud. Both APIs are paid services that charge a small amount per question, and both accounts must belong to an adult (18+), so ask a parent or teacher to do these steps with you.
-
For Claude: create an account at console.anthropic.com, add a small amount of credit and set a low monthly spend limit under Billing. Under API keys, create a key and copy it. It starts with
sk-ant-. -
For OpenAI: create an account at platform.openai.com, add a small amount of credit under Billing and set a low monthly budget. Under API keys, create a new secret key and copy it. It starts with
sk-. - Make new keys just for these glasses. An API key works like a password and a credit card in one: never share it, never post your sketch online with a key in it, and delete the keys when the project is finished.
The sketch
In the Arduino IDE, open Library Manager and install the ArduinoJson library by Benoit Blanchon.
The camera, microphone and I2S support come with the esp32 boards package. Replace YOUR_HOTSPOT_NAME and YOUR_HOTSPOT_PASSWORD with your hotspot's details, and paste your two keys into CLAUDE_KEY and OPENAI_KEY.
// Talking camera glasses: press the button and ask a question. The glasses take a picture,
// an AI looks at it and the answer comes out of the small speaker on the arm.
// short press -> "What is in front of me?"
// hold -> ask your own question while you hold the button (up to 6 seconds)
// Needs: XIAO ESP32S3 Sense (camera + microphone), MAX98357A amplifier + speaker, a push button,
// and the internet through your phone's hotspot. Tools > PSRAM must be "OPI PSRAM".
#define ARDUINOJSON_STRING_LENGTH_SIZE 4 // allow texts longer than 64 KB (pictures and sound)
#include "esp_camera.h"
#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>
#include <ESP_I2S.h>
#include "mbedtls/base64.h"
// ---------- settings ----------
const char* WIFI_NAME = "YOUR_HOTSPOT_NAME"; // your phone's hotspot (2.4 GHz)
const char* WIFI_PASSWORD = "YOUR_HOTSPOT_PASSWORD";
const char* CLAUDE_KEY = "YOUR_ANTHROPIC_API_KEY"; // Claude looks at the picture: keep it secret!
const char* OPENAI_KEY = "YOUR_OPENAI_API_KEY"; // OpenAI hears and speaks: keep it secret!
const char* CLAUDE_MODEL = "claude-opus-5-5";
const char* HEAR_MODEL = "gpt-transcribe"; // speech -> text
const char* VOICE_MODEL = "gpt-audio-1.5"; // text -> speech
const char* VOICE = "marin";
const char* DEFAULT_QUESTION = "What is in front of me?";
const int VOLUME = 70; // 0 to 100
const bool CAMERA_UPSIDE_DOWN = false; // true if the picture comes out upside down
const int MAX_SECONDS = 6; // longest question
const int MIC_GAIN = 4; // the microphone is quiet: make it louder
const char* EYES = "You are the eyes of a blind or visually impaired person. "
"The photo comes from a small camera on their glasses and shows what is in front of them. "
"Answer their question in one to three short spoken sentences, at most 50 words. "
"Always mention obstacles and dangers first: steps, kerbs, holes, doors, poles, furniture, people, bikes, cars, "
"and anything at head height. Say where they are (left, right or straight ahead) and roughly how far, in steps, "
"and make clear that distances are only a guess. "
"If the photo is dark or blurred, or you are not sure, say so. "
"Never tell them that a road, crossing or path is safe: you cannot see everything. "
"Plain words only, for speaking aloud: no lists, no symbols, no emoji.";
const char* READ_ALOUD = "You are a reading voice for a blind person. Read the user's message aloud exactly as written, "
"in a calm, clear voice at a slightly quick pace. Do not answer it, add to it or change it.";
// ---------- pins ----------
const int BUTTON = 1; // D0: push button to GND
const int SPK_BCLK = 2, SPK_LRC = 3, SPK_DIN = 4; // D1, D2, D3: MAX98357A
const int LED = 21; // the XIAO's small orange LED (LOW = on)
const int RATE = 16000; // microphone: 16 000 samples a second
I2SClass mic(I2S_NUM_0); // the microphone needs I2S port 0
I2SClass speaker(I2S_NUM_1);
uint32_t speakerRate = 24000;
uint8_t* wav = NULL; // the recorded question, as a WAV file in PSRAM
const size_t WAV_MAX = 44 + MAX_SECONDS * RATE * 2;
char* photo = NULL; // the picture, as Base64 text in PSRAM
// ---------- sounds ----------
void beep(int freq, int ms) {
const int N = 240;
int16_t buf[N];
long total = (long)speakerRate * ms / 1000, done = 0;
while (done < total) {
int n = min((long)N, total - done);
for (int i = 0; i < n; i++) buf[i] = (int16_t)(4000 * VOLUME / 100 * sin(2 * PI * freq * (done + i) / speakerRate));
speaker.write((uint8_t*)buf, n * 2);
done += n;
}
memset(buf, 0, sizeof(buf)); // a moment of silence so the amplifier doesn't click
speaker.write((uint8_t*)buf, sizeof(buf));
}
void errorBeeps() { for (int i = 0; i < 3; i++) { beep(300, 150); delay(80); } }
// ---------- camera (XIAO ESP32S3 Sense pins) ----------
bool startCamera() {
camera_config_t c = {};
c.ledc_channel = LEDC_CHANNEL_0;
c.ledc_timer = LEDC_TIMER_0;
c.pin_d0 = 15; c.pin_d1 = 17; c.pin_d2 = 18; c.pin_d3 = 16;
c.pin_d4 = 14; c.pin_d5 = 12; c.pin_d6 = 11; c.pin_d7 = 48;
c.pin_xclk = 10; c.pin_pclk = 13; c.pin_vsync = 38; c.pin_href = 47;
c.pin_sccb_sda = 40; c.pin_sccb_scl = 39;
c.pin_pwdn = -1; c.pin_reset = -1;
c.xclk_freq_hz = 20000000;
c.pixel_format = PIXFORMAT_JPEG;
c.frame_size = FRAMESIZE_VGA; // 640 x 480 is plenty for the AI and quick to send
c.jpeg_quality = 12; // 0 = best, 63 = smallest
c.fb_count = 1;
c.fb_location = CAMERA_FB_IN_PSRAM;
c.grab_mode = CAMERA_GRAB_WHEN_EMPTY; // only takes pictures when asked: cooler and saves power
if (esp_camera_init(&c) != ESP_OK) return false;
sensor_t* s = esp_camera_sensor_get();
s->set_vflip(s, CAMERA_UPSIDE_DOWN);
s->set_hmirror(s, CAMERA_UPSIDE_DOWN);
return true;
}
// Take a fresh picture and keep it as Base64 text (JSON can only carry text)
bool takePhoto() {
free(photo);
photo = NULL;
camera_fb_t* fb = esp_camera_fb_get(); // this one may be old: throw it away
if (fb) esp_camera_fb_return(fb);
fb = esp_camera_fb_get();
if (!fb) return false;
size_t len = 0;
mbedtls_base64_encode(NULL, 0, &len, fb->buf, fb->len);
photo = (char*)ps_malloc(len);
if (photo) mbedtls_base64_encode((unsigned char*)photo, len, &len, fb->buf, fb->len);
esp_camera_fb_return(fb);
return photo != NULL;
}
// ---------- the button ----------
bool pressed() { // LOW for 20 ms in a row: a real press, not noise
for (int i = 0; i < 3; i++) { if (digitalRead(BUTTON) == HIGH) return false; delay(10); }
return true;
}
// ---------- recording while the button is held ----------
size_t recordQuestion() { // returns the number of sound bytes
size_t len = 0;
char junk[1024];
for (int i = 0; i < 4; i++) mic.readBytes(junk, sizeof(junk)); // throw away old sound (and the beep)
unsigned long released = 0;
while (44 + len + 1024 <= WAV_MAX) {
if (digitalRead(BUTTON) == HIGH) { // let go for 30 ms: the question is finished
if (!released) released = millis();
else if (millis() - released > 30) return len;
} else released = 0;
int16_t* s = (int16_t*)(wav + 44 + len);
size_t got = mic.readBytes((char*)s, 1024);
for (size_t i = 0; i < got / 2; i++) s[i] = constrain((int32_t)s[i] * MIC_GAIN, -32768, 32767);
len += got;
}
beep(500, 80); // 6 seconds are up: "stopped listening"
return len;
}
void writeWavHeader(uint8_t* h, uint32_t dataLen) {
uint32_t v;
memcpy(h, "RIFF", 4); v = dataLen + 36; memcpy(h + 4, &v, 4);
memcpy(h + 8, "WAVEfmt ", 8); v = 16; memcpy(h + 16, &v, 4);
uint16_t pcm = 1, ch = 1, bits = 16, align = 2;
memcpy(h + 20, &pcm, 2); memcpy(h + 22, &ch, 2);
v = RATE; memcpy(h + 24, &v, 4); v = RATE * 2; memcpy(h + 28, &v, 4);
memcpy(h + 32, &align, 2); memcpy(h + 34, &bits, 2);
memcpy(h + 36, "data", 4); memcpy(h + 40, &dataLen, 4);
}
// ---------- 1. speech -> text (OpenAI) ----------
int hearStatus = 0; // 200 = OK, anything else = the error code
String hear(size_t soundLen) {
writeWavHeader(wav, soundLen);
const char* B = "----eementorGlasses7MA4";
String head = String("--") + B + "\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\n" + HEAR_MODEL + "\r\n" +
"--" + B + "\r\nContent-Disposition: form-data; name=\"file\"; filename=\"question.wav\"\r\n" +
"Content-Type: audio/wav\r\n\r\n";
String tail = String("\r\n--") + B + "--\r\n";
size_t total = head.length() + 44 + soundLen + tail.length();
uint8_t* body = (uint8_t*)ps_malloc(total);
hearStatus = -1;
if (!body) return "";
memcpy(body, head.c_str(), head.length());
memcpy(body + head.length(), wav, 44 + soundLen);
memcpy(body + head.length() + 44 + soundLen, tail.c_str(), tail.length());
WiFiClientSecure client;
client.setInsecure();
HTTPClient http;
http.setTimeout(30000);
http.begin(client, "https://api.openai.com/v1/audio/transcriptions");
http.addHeader("Authorization", String("Bearer ") + OPENAI_KEY);
http.addHeader("Content-Type", String("multipart/form-data; boundary=") + B);
hearStatus = http.POST(body, total);
String response = http.getString();
http.end();
free(body);
if (hearStatus != 200) { Serial.printf("Hearing error %d: %s\n", hearStatus, response.c_str()); return ""; }
JsonDocument answer;
DeserializationError err = deserializeJson(answer, response);
if (err) { Serial.printf("Hearing: strange answer (%s)\n", err.c_str()); return ""; }
String text = answer["text"] | "";
text.trim();
return text;
}
// ---------- 2. picture + question -> answer (Claude) ----------
String look(const String& question) {
JsonDocument request;
request["model"] = CLAUDE_MODEL;
request["max_tokens"] = 1500; // room for a little thinking and a short answer
request["output_config"]["effort"] = "low"; // quick answers
request["fallbacks"] = "default"; // if this model declines, another one may answer
request["system"] = EYES;
JsonObject m = request["messages"].to<JsonArray>().add<JsonObject>();
m["role"] = "user";
JsonArray content = m["content"].to<JsonArray>();
JsonObject img = content.add<JsonObject>();
img["type"] = "image";
img["source"]["type"] = "base64";
img["source"]["media_type"] = "image/jpeg";
img["source"]["data"] = (const char*)photo;
JsonObject txt = content.add<JsonObject>();
txt["type"] = "text";
txt["text"] = question;
String body;
serializeJson(request, body);
request.clear();
WiFiClientSecure client;
client.setInsecure();
HTTPClient http;
http.setTimeout(60000);
http.begin(client, "https://api.anthropic.com/v1/messages");
http.addHeader("content-type", "application/json");
http.addHeader("x-api-key", CLAUDE_KEY);
http.addHeader("anthropic-version", "2026-10-21");
http.addHeader("anthropic-beta", "server-side-fallback-2026-07-01");
int status = http.POST(body);
String response = http.getString();
http.end();
if (status != 200) {
Serial.printf("Claude error %d: %s\n", status, response.c_str());
if (status == 401) return "Please check my Claude API key.";
if (status == 400 && response.indexOf("credit") >= 0) return "My Claude account has no credit left.";
if (status == 429) return "Too many questions. Please wait a minute.";
if (status == 529 || status == 503) return "The A I is busy. Please try again.";
if (status < 0) return "I have no internet connection.";
return "Sorry, I could not reach the A I.";
}
JsonDocument filter;
filter["stop_reason"] = true;
filter["content"][0]["type"] = true;
filter["content"][0]["text"] = true;
JsonDocument answer;
DeserializationError err = deserializeJson(answer, response, DeserializationOption::Filter(filter));
if (err) { Serial.printf("Claude: strange answer (%s)\n", err.c_str()); return "Sorry, I got a strange answer."; }
if (answer["stop_reason"] == "refusal") return "Sorry, I can't describe this one.";
String text = "";
for (JsonObject block : answer["content"].as<JsonArray>())
if (block["type"] == "text") text += block["text"].as<String>(); // skip the thinking blocks
text.trim();
return text.length() ? text : "Sorry, I got an empty answer.";
}
// ---------- 3. text -> speech (OpenAI) and play it ----------
bool speak(const String& text) {
JsonDocument request;
request["model"] = VOICE_MODEL;
request["modalities"][0] = "text";
request["modalities"][1] = "audio";
request["audio"]["voice"] = VOICE;
request["audio"]["format"] = "wav";
JsonArray msgs = request["messages"].to<JsonArray>();
JsonObject sys = msgs.add<JsonObject>();
sys["role"] = "system";
sys["content"] = READ_ALOUD;
JsonObject user = msgs.add<JsonObject>();
user["role"] = "user";
user["content"] = text;
String body;
serializeJson(request, body);
WiFiClientSecure client;
client.setInsecure();
HTTPClient http;
http.setTimeout(60000);
http.useHTTP10(true); // a plain stream, so the sketch can read the answer as it arrives
http.begin(client, "https://api.openai.com/v1/chat/completions");
http.addHeader("Content-Type", "application/json");
http.addHeader("Authorization", String("Bearer ") + OPENAI_KEY);
int status = http.POST(body);
if (status != 200) { Serial.printf("Voice error %d: %s\n", status, http.getString().substring(0, 300).c_str()); http.end(); return false; }
JsonDocument filter; // keep only the sound from the (big) answer
filter["choices"][0]["message"]["audio"]["data"] = true;
JsonDocument answer;
DeserializationError err = deserializeJson(answer, http.getStream(), DeserializationOption::Filter(filter));
http.end();
if (err) { Serial.printf("Voice: strange answer (%s)\n", err.c_str()); return false; }
const char* data = answer["choices"][0]["message"]["audio"]["data"] | "";
size_t inLen = strlen(data), outLen = 0;
if (inLen < 100) { Serial.println("Voice: no sound in the answer"); return false; }
uint8_t* sound = (uint8_t*)ps_malloc(inLen * 3 / 4 + 4);
if (!sound) return false;
int bad = mbedtls_base64_decode(sound, inLen * 3 / 4 + 4, &outLen, (const unsigned char*)data, inLen);
answer.clear();
if (bad || outLen < 44 || memcmp(sound, "RIFF", 4) || memcmp(sound + 8, "WAVE", 4)) { free(sound); Serial.println("Voice: not a WAV file"); return false; }
// find the sound inside the WAV file: the "fmt " chunk says how to play it, "data" holds the samples
uint32_t rate = 0;
uint16_t channels = 0, bits = 0;
size_t pos = 12, start = 0, len = 0;
while (pos + 8 <= outLen) {
uint32_t size;
memcpy(&size, sound + pos + 4, 4);
if (!memcmp(sound + pos, "fmt ", 4) && pos + 24 <= outLen) {
memcpy(&channels, sound + pos + 10, 2); memcpy(&rate, sound + pos + 12, 4); memcpy(&bits, sound + pos + 22, 2);
}
if (!memcmp(sound + pos, "data", 4)) {
start = pos + 8;
len = (size == 0 || size > outLen - start) ? outLen - start : size; // streamed WAVs may not know their size
break;
}
if (size > outLen - pos - 8) break;
pos += 8 + size + (size & 1);
}
if (!start || bits != 16 || channels < 1 || channels > 2 || rate < 8000 || rate > 48000) {
Serial.printf("Voice: can't play this sound (%u Hz, %u channels, %u bits)\n", rate, channels, bits);
free(sound);
return false;
}
if (rate != speakerRate && speaker.configureTX(rate, I2S_DATA_BIT_WIDTH_16BIT, I2S_SLOT_MODE_MONO)) speakerRate = rate;
int16_t* s = (int16_t*)(sound + start);
size_t n = len / 2 / channels;
for (size_t i = 0; i < n; i++) s[i] = (int32_t)s[i * channels] * VOLUME / 100; // left channel only, at our volume
speaker.write((uint8_t*)s, n * 2);
free(sound);
int16_t quiet[256] = {0};
speaker.write((uint8_t*)quiet, sizeof(quiet));
return true;
}
void say(const String& text) {
Serial.println("Answer: " + text);
if (!speak(text)) errorBeeps(); // no voice: check the OpenAI key and the Serial Monitor
}
// ---------- Wi-Fi ----------
bool online() {
if (WiFi.status() == WL_CONNECTED) return true;
WiFi.reconnect();
for (int i = 0; i < 40 && WiFi.status() != WL_CONNECTED; i++) delay(250);
return WiFi.status() == WL_CONNECTED;
}
void setup() {
Serial.begin(115200);
pinMode(BUTTON, INPUT_PULLUP);
pinMode(LED, OUTPUT);
digitalWrite(LED, HIGH);
if (!psramFound()) { Serial.println("No PSRAM: set Tools > PSRAM to OPI PSRAM and upload again"); while (true) delay(1000); }
wav = (uint8_t*)ps_malloc(WAV_MAX);
mic.setPinsPdmRx(42, 41); // the Sense board's microphone: clock 42, data 41
if (!mic.begin(I2S_MODE_PDM_RX, RATE, I2S_DATA_BIT_WIDTH_16BIT, I2S_SLOT_MODE_MONO)) Serial.println("Microphone not found");
speaker.setPins(SPK_BCLK, SPK_LRC, SPK_DIN);
if (!speaker.begin(I2S_MODE_STD, speakerRate, I2S_DATA_BIT_WIDTH_16BIT, I2S_SLOT_MODE_MONO)) Serial.println("Speaker (I2S) did not start");
if (!startCamera()) { Serial.println("Camera not found: press the camera board on firmly"); while (true) { errorBeeps(); delay(3000); } }
WiFi.mode(WIFI_STA);
WiFi.setSleep(false); // answers come back faster
WiFi.begin(WIFI_NAME, WIFI_PASSWORD);
Serial.print("Connecting to the hotspot");
for (int i = 0; i < 60 && WiFi.status() != WL_CONNECTED; i++) { delay(500); Serial.print("."); }
if (WiFi.status() == WL_CONNECTED) {
Serial.println(" connected");
beep(660, 100); beep(880, 100); beep(1320, 150); // ready
} else {
Serial.println(" not connected yet: is the hotspot on?");
errorBeeps();
}
}
void loop() {
if (!pressed()) { delay(10); return; }
unsigned long start = millis();
beep(1200, 60); // "I'm listening"
delay(80); // let the beep finish before recording
digitalWrite(LED, LOW);
size_t soundLen = recordQuestion(); // records until the button is let go
bool ownQuestion = millis() - start > 700;
digitalWrite(LED, HIGH);
bool havePhoto = takePhoto(); // the picture shows what is in front right now
while (digitalRead(BUTTON) == LOW) delay(10); // held longer than 6 seconds: wait for the release
if (!online()) { Serial.println("No Wi-Fi: is the hotspot on?"); errorBeeps(); return; }
if (!havePhoto) { say("Sorry, my camera is not working."); return; }
beep(700, 60); // "thinking"
digitalWrite(LED, LOW);
String question = DEFAULT_QUESTION;
if (ownQuestion) {
String heard = hear(soundLen);
if (hearStatus == 401) { digitalWrite(LED, HIGH); say("Please check my Open A I key."); return; }
if (hearStatus != 200) { digitalWrite(LED, HIGH); say("Sorry, I could not reach the speech service."); return; }
if (heard.length() >= 3) question = heard; // nothing said: ask the standard question instead
}
Serial.println("Question: " + question);
String answer = look(question);
digitalWrite(LED, HIGH);
say(answer);
}
client.setInsecure() keeps the connection encrypted but skips checking the server's certificate. That keeps the sketch short and is fine for a learning project. For a device you leave running for years, load the server's root certificate with client.setCACert() instead.
Try it
- Switch on the hotspot, upload the sketch and open the Serial Monitor at 115200 baud. After "connected" the speaker plays three rising beeps: the glasses are ready.
- Point the camera at a table with a few things on it and press the button briefly. You hear a short high beep, then a lower one ("thinking"). After a few seconds the glasses say what is in front of them, and the Serial Monitor shows the question and the answer.
- Now hold the button and ask a question while you hold it, for example "What colour is the cup?" or "How many books are there?". Let go when you have finished speaking. The Serial Monitor shows what it heard.
- Try harder questions: hold up a sign or a page and ask "What does it say?". Point it at a door and ask "Is the door open?". Cover the camera and press the button: it should say that it can't see well, not make something up.
- Check which way up the camera is: hold the button and ask "Is this picture upside down?". If the answer is yes, set
CAMERA_UPSIDE_DOWN = trueand upload again. Otherwise leave it, or the answers about left and right will be swapped. - Build it into the printed glasses, power it from the power bank and walk slowly around a room with your eyes open and a friend next to you. Stop in front of a chair, a door or a bag and press the button. How often is the answer right? How long does it take?
What you should see

How it works

When you press the button, the sketch beeps and starts recording. The microphone on the Sense board delivers 16 000 numbers a second; the sketch makes them a bit louder and stores them in the PSRAM, up to 6 seconds (192 KB). A short press (under 0.7 s) means "use the standard question": the recording is thrown away and the question is "What is in front of me?". That skips the speech-to-text step and saves a few seconds. The moment you let go, the camera takes a fresh JPEG picture, 640 × 480 pixels, about 30 to 50 KB, so it shows what was in front of you when you asked. JSON can only carry text, so the sketch turns the picture straight into Base64: letters and digits, a third longer than the picture itself.
For your own question, the sketch puts a 44-byte WAV header in front of the recording: a few numbers that say "16-bit sound, one channel, 16 000 samples a second". It sends the file to OpenAI's speech-to-text model gpt-transcribe, which sends back the words as text.
Next the sketch sends the picture, your question and a system prompt to Claude. The system prompt is the most important part of the design. It tells Claude who it is talking to and how to answer: one to three short sentences, obstacles and dangers first, with left, right or straight ahead and the distance in steps; say so when the picture is dark or blurred; and never say that a road or path is safe, because a single photo can't show everything. The distances are guesses from the size of things in the picture, and the prompt asks Claude to say so. Claude thinks briefly (effort is set to low, for quick answers) and sends back text.
Finally the sketch sends that text to OpenAI's voice model gpt-audio-1.5 with the instruction to read it aloud word for word. The answer comes back as a WAV file in Base64. The sketch decodes it, finds the sound in the file, makes it quieter or louder with VOLUME, and sends it over I2S to the MAX98357A, which drives the speaker. The whole trip (your voice up, text back, picture up, text back, text up, voice back) takes about 10 to 20 seconds over a phone hotspot; most of it is the spoken answer, about a megabyte, coming back. Compare that with the obstacle-warning glasses, which react in a twentieth of a second but only know "something is close". Real products for blind people combine both: fast, simple sensors for safety and slower AI descriptions for understanding.
Why not do all of it on the board? The ESP32-S3 can recognise a few things it was taught, as in the object-learning camera, but describing any scene in words and answering any question needs an AI model millions of times bigger than the chip's memory. So the ESP32-S3 does what it is good at: the camera, the microphone, the speaker, Wi-Fi and memory, and the cloud does the understanding.
Try this next
- Change the system prompt for other jobs: "read any text in the picture aloud", "describe the colours of my clothes", or "tell me what is on this plate". Test each with real objects.
- Add a second button for "read this text", so there is no need to say the question out loud.
- Add a vibration motor, driven through a transistor, that buzzes once when the answer mentions an obstacle, for noisy places where the speaker is hard to hear.
- Combine it with the obstacle-warning glasses: an HC-SR04 on the front for instant warnings, and the camera for questions.
- Ask someone who is blind or has low vision to try it, with a friend next to them, and ask what they would change: the voice, the speed, the button, the length of the answers. Designing with users is what makes assistive technology work.
Troubleshooting
- "No PSRAM" in the Serial Monitor: set Tools → PSRAM to OPI PSRAM and upload again.
- "Camera not found" and three low beeps every few seconds: press the Sense board onto the back of the XIAO until it clicks, and check that the camera's ribbon cable is in its socket.
- It never connects: the hotspot must be on and set to 2.4 GHz (Maximise Compatibility on an iPhone). Check the name and password: both are case-sensitive. Keep the phone within a few metres and stick the antenna onto the XIAO.
- No sound at all, not even the beeps: check the three I2S wires (LRC to D2, BCLK to D1, DIN to D3) and that Vin and GND are connected. Vin belongs on 5V (on 3V3 it still works, but quieter). The XIAO only has 5 V on that pin when it runs from USB.
- Beeps work, but three low beeps instead of an answer: look in the Serial Monitor. "No Wi-Fi" means the hotspot is off or out of range. "Voice error 401" means the OpenAI key is wrong, 429 means no credit left or too many questions.
-
It says "Please check my Claude API key": the Claude key is wrong. Copy it again, without spaces, into
CLAUDE_KEY. "My Claude account has no credit left" means an adult needs to add credit in the Claude Console. -
It answers "What is in front of me?" instead of your own question: it heard nothing. Hold the button the whole time you speak, speak close to the glasses, or make
MIC_GAINbigger (for example 8). If it says "I could not reach the speech service", look for "Hearing error" in the Serial Monitor. -
Left and right are the wrong way round: the picture may be upside down. Hold the button and ask "Is this picture upside down?"; only if the answer is yes, set
CAMERA_UPSIDE_DOWN = true. -
The answers are too quiet or too loud: change
VOLUME(0 to 100). A crackle at high volume means the power bank or cable can't deliver enough current: try another one. - The port disappeared and uploads fail: hold the B button while plugging in the cable, let go, and upload again.
- A printed part is too tight or too loose: printers differ by a few tenths of a millimetre. Sand or file a tight hole, and fix a loose part with a drop of hot glue or a strip of tape.