Build a talking board you control by tilting your head

Imagine you can think clearly, but you can't speak and can't move your hands. Telling someone "I'm thirsty" or "I'm in pain" would be really hard. Many people live like this: after a stroke, with ALS or cerebral palsy, or for a while after an accident. They use AAC devices (augmentative and alternative communication): talking boards that they control with whatever movement they still have, sometimes only their eyes or their head. In this capstone you build one. The MPU-6050 sits on a headband. Tilting the head to the right moves the highlight to the next phrase on a tablet or phone, tilting left goes back, and a nod held for one second makes the tablet speak the phrase out loud. "Help!" also sounds the buzzer, so someone in the next room hears it.

What you'll learn

• How to turn small head movements into commands, with dead zones, hold times and repeat
• How a web page can speak out loud with the browser's built-in speech (Web Speech API)
• How to send commands and live data over a WebSocket in both directions
• Why designing for people with disabilities means making the device adjustable

What you need

• ESP32 board, expansion board and USB cable (ESP32 IoT Starter Kit)
• GY-521 MPU-6050 module (45-in-1 Sensor Kit)
• 1 push button (ESP32 IoT Starter Kit)
• Passive buzzer (ESP32 IoT Starter Kit)
• A headband, a cap or an elastic band to hold the sensor
• 4 long jumper wires (join female-to-female wires to make them about 60 cm long)
• A tablet, phone or computer with Chrome, Edge or Safari, on the same Wi-Fi
• Breadboard and jumper wires (ESP32 IoT Starter Kit)

Design it together. A talking board is personal: the phrases, the movements, how big a tilt is comfortable and how long a nod should be all depend on the person who uses it. Build it with them, and if they have a speech therapist, ask for their advice. Never fix the sensor so tight that it hurts, and let the person take it off whenever they want.

Wiring

Build it step by step

Plug in all the parts first, then add the wires in this order. In each picture the new wires are bright and the ones from earlier steps are faded. Click a picture to see it larger.

Tape the MPU-6050 flat onto the side of the headband, above one ear, with the chip facing outwards and the Y arrow printed on the module pointing up to the sky. The angle maths only works this way round. Run the four long wires down to the breadboard. The button is for the helper: it sets "looking straight ahead".

The sketch

Replace YOUR_WIFI_NAME and YOUR_WIFI_PASSWORD with your own network details (2.4 GHz only).

In the Arduino IDE, open Library Manager and install WebSockets by Markus Sattler (in the Library Manager, search for "WebSockets" and pick the one by Markus Sattler).

#include <WiFi.h>
#include <WebServer.h>
#include <ESPmDNS.h>
#include <WebSocketsServer.h>
#include <Wire.h>

const char* WIFI_NAME = "YOUR_WIFI_NAME";
const char* WIFI_PASSWORD = "YOUR_WIFI_PASSWORD";

const float TILT = 15;                 // degrees of head tilt that count as a move
const unsigned long REPEAT_MS = 900;   // keep tilting to keep moving
const unsigned long NOD_MS = 1000;     // hold a nod this long to speak
const int SIDE_SIGN = 1;               // -1 if tilting right moves the highlight left
const int NOD_SIGN = 1;                // -1 if looking up counts as a nod

const int MPU = 0x68, BUTTON_PIN = 32, BUZZER_PIN = 33;

WebServer server(80);
WebSocketsServer sockets(81);
float pitch = 0, roll = 0, pitchZero = 0, rollZero = 0;
int zone = 0;                          // 0 middle, 1 right, 2 left, 3 nod
unsigned long zoneStart = 0, lastFire = 0, lastSend = 0, lastSample = 0;
bool fired = false, lastButton = HIGH;

const char PAGE[] PROGMEM = R"rawliteral(<!doctype html><html><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1"><title>Talking board</title>
<style>
body{margin:0;font-family:Arial,sans-serif;background:#F6F8FC;color:#1B3A7B}
#top{display:flex;justify-content:space-between;align-items:center;padding:10px 16px;gap:10px;flex-wrap:wrap}
#grid{display:grid;grid-template-columns:repeat(3,1fr);gap:12px;padding:0 16px}
.card{background:#fff;border:4px solid #E3E8F2;border-radius:18px;padding:18px 8px;text-align:center;font-size:26px;font-weight:bold}
.card span{display:block;font-size:48px}
.card.on{border-color:#DE544A;background:#FDECEA}
.card.said{background:#DE544A;color:#fff}
button{font-size:18px;padding:10px 16px;border-radius:12px;border:0;background:#1B3A7B;color:#fff}
#head{width:120px;height:60px;background:#fff;border:2px solid #E3E8F2;border-radius:10px;position:relative}
#dot{width:14px;height:14px;border-radius:7px;background:#DE544A;position:absolute;left:53px;top:23px}
#nod{height:6px;background:#DE544A;width:0}
#log{padding:10px 16px;color:#5B6B8C}
#start{position:fixed;inset:0;background:rgba(27,58,123,.92);color:#fff;display:flex;align-items:center;justify-content:center;font-size:30px;text-align:center}
</style></head><body>
<div id="start">Tap here to start<br>(this lets the page speak)</div>
<div id="top"><b id="info">Connecting...</b>
<div><div id="head"><div id="dot"></div></div><div id="nod"></div></div>
<button id="zero">Set straight ahead</button></div>
<div id="grid"></div><div id="log"></div>
<script>
var P = [['\u{1F44D}', 'Yes'], ['\u{1F44E}', 'No'], ['\u{1F4A7}', "I'm thirsty"], ['\u{1F37D}', "I'm hungry"], ['\u{1F915}', "I'm in pain"],
  ['\u{1F64B}', 'Please come here'], ['\u{1F634}', "I'm tired"], ['\u{2764}', 'Thank you'], ['\u{1F198}', 'Help!']];
var sel = 0, sock, said = [];
var grid = document.getElementById('grid'), info = document.getElementById('info');
P.forEach(function (p, i) {
  var d = document.createElement('div'); d.className = 'card'; d.id = 'c' + i;
  d.innerHTML = '<span>' + p[0] + '</span>' + p[1];
  d.onclick = function () { sel = i; draw(); say(); };
  grid.appendChild(d);
});
function draw() { P.forEach(function (p, i) { document.getElementById('c' + i).classList.toggle('on', i === sel); }); }
function send(m) { if (sock && sock.readyState === 1) sock.send(m); }
function say() {
  var text = P[sel][1], u = new SpeechSynthesisUtterance(text);
  u.rate = 0.9;
  speechSynthesis.cancel(); speechSynthesis.speak(u);
  if (text === 'Help!') send('help');
  var c = document.getElementById('c' + sel); c.classList.add('said');
  setTimeout(function () { c.classList.remove('said'); }, 1500);
  said.unshift(new Date().toLocaleTimeString().slice(0, 5) + '  ' + text); said = said.slice(0, 5);
  document.getElementById('log').textContent = 'Said: ' + said.join('   |   ');
}
function connect() {
  sock = new WebSocket('ws://' + location.hostname + ':81/');
  sock.onopen = function () { info.textContent = 'Tilt to choose, nod to speak'; };
  sock.onclose = function () { info.textContent = 'Reconnecting...'; setTimeout(connect, 1000); };
  sock.onmessage = function (e) {
    var m = e.data;
    if (m === 'next') { sel = (sel + 1) % P.length; draw(); }
    else if (m === 'prev') { sel = (sel + P.length - 1) % P.length; draw(); }
    else if (m === 'say') say();
    else if (m === 'zeroed') info.textContent = 'Straight ahead saved';
    else if (m.charAt(0) === 't') {
      var v = m.split(',');
      document.getElementById('dot').style.left = (53 + Math.max(-50, Math.min(50, v[1] * 1.6))) + 'px';
      document.getElementById('dot').style.top = (23 + Math.max(-22, Math.min(22, v[2] * 0.7))) + 'px';
      document.getElementById('nod').style.width = (v[3] * 1.2) + 'px';
    }
  };
}
document.getElementById('start').onclick = function () {
  this.style.display = 'none';
  speechSynthesis.speak(new SpeechSynthesisUtterance('Ready'));
};
document.getElementById('zero').onclick = function () { send('zero'); };
draw(); connect();
</script></body></html>)rawliteral";

void readAngles(float& p, float& r) {
  Wire.beginTransmission(MPU);
  Wire.write(0x3B);
  Wire.endTransmission(false);
  Wire.requestFrom(MPU, 6);
  int16_t v[3];
  for (int i = 0; i < 3; i++) {
    v[i] = Wire.read() << 8;
    v[i] |= Wire.read();
  }
  float x = v[0], y = v[1], z = v[2];
  p = atan2(-x, sqrt(y * y + z * z)) * 180 / PI;   // nodding forwards and back
  r = atan2(y, z) * 180 / PI;                        // tilting to the shoulders
}

void zero() {                          // look straight ahead and keep still
  float sp = 0, sr = 0, p, r;
  for (int i = 0; i < 50; i++) {
    readAngles(p, r);
    sp += p;
    sr += r;
    delay(10);
  }
  pitch = pitchZero = sp / 50;
  roll = rollZero = sr / 50;
  sockets.broadcastTXT("zeroed");
  tone(BUZZER_PIN, 1047, 80);
}

void onSocket(uint8_t client, WStype_t type, uint8_t* payload, size_t length) {
  if (type != WStype_TEXT) return;
  String msg = String((char*)payload).substring(0, length);
  if (msg == "zero") zero();
  else if (msg == "help") {            // loud enough for the next room
    for (int i = 0; i < 5; i++) {
      tone(BUZZER_PIN, 1760, 250);
      delay(320);
      tone(BUZZER_PIN, 1175, 250);
      delay(320);
    }
  }
}

void setup() {
  Serial.begin(115200);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  Wire.begin(21, 22);                  // SDA, SCL
  Wire.beginTransmission(MPU);
  Wire.write(0x6B);                    // wake up
  Wire.write(0);
  Wire.endTransmission();
  delay(100);
  WiFi.begin(WIFI_NAME, WIFI_PASSWORD);
  Serial.print("Connecting");
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  MDNS.begin("talkingboard");          // http://talkingboard.local
  server.on("/", []() { server.send_P(200, "text/html", PAGE); });
  server.begin();
  sockets.begin();
  sockets.onEvent(onSocket);
  zero();
  Serial.print("\nOpen http://talkingboard.local or http://");
  Serial.println(WiFi.localIP());
}

void loop() {
  server.handleClient();
  sockets.loop();

  bool button = digitalRead(BUTTON_PIN);
  if (lastButton == HIGH && button == LOW) zero();   // the helper's button
  lastButton = button;

  unsigned long now = millis();
  if (now - lastSample < 20) return;   // 50 readings a second
  lastSample = now;
  float p, r;
  readAngles(p, r);
  pitch = pitch * 0.8 + p * 0.2;       // smooth out small shakes of the head
  roll = roll * 0.8 + r * 0.2;
  float side = SIDE_SIGN * (roll - rollZero);
  float nod = NOD_SIGN * (pitch - pitchZero);

  int z = 0;                           // which zone is the head in?
  if (fabs(side) > TILT && fabs(side) >= fabs(nod)) z = side > 0 ? 1 : 2;
  else if (nod > TILT) z = 3;
  if (z != zone) {
    zone = z;
    zoneStart = now;
    fired = false;
  }

  if ((zone == 1 || zone == 2) && now - zoneStart > 250 && (!fired || now - lastFire > REPEAT_MS)) {
    fired = true;
    lastFire = now;
    sockets.broadcastTXT(zone == 1 ? "next" : "prev");
    tone(BUZZER_PIN, 1200, 30);        // a tiny click, so the user feels in control
  }
  if (zone == 3 && !fired && now - zoneStart > NOD_MS) {
    fired = true;                      // one phrase per nod: look up again first
    sockets.broadcastTXT("say");
    tone(BUZZER_PIN, 1600, 60);
  }

  if (now - lastSend >= 100) {         // live head position for the little dot
    lastSend = now;
    int progress = zone == 3 ? min(100, (int)((now - zoneStart) * 100 / NOD_MS)) : 0;
    char msg[32];
    snprintf(msg, sizeof(msg), "t,%.0f,%.0f,%d", side, nod, progress);
    sockets.broadcastTXT(msg);
  }
}

Try it

1. Upload the sketch and open the Serial Monitor at 115200 baud. Put on the headband and look straight ahead while it starts: that is "straight ahead".
2. Open the address from the Serial Monitor on a tablet, phone or computer on the same Wi-Fi: http://talkingboard.local, or the number it shows, such as http://192.168.1.57.
3. Tap the blue screen once: the page says "Ready". Browsers only allow a page to speak after someone has tapped it.
4. Tilt your head towards your right shoulder: the red frame moves to the next phrase. Keep it tilted to keep moving; tilt left to go back.
5. Choose "I'm thirsty" and nod forwards, holding it for a second. The bar under the little head fills up, and the tablet says "I'm thirsty".
6. If right and left are swapped, set SIDE_SIGN = -1; if looking up speaks instead of nodding, set NOD_SIGN = -1. If the person's head is not quite straight, press the button or "Set straight ahead".

What you should see

How it works

Held still, the accelerometer only feels gravity. The sketch turns the three numbers into two angles, as in the MPU-6050 lesson: roll for tilting towards a shoulder and pitch for nodding. Heads move slowly, so the accelerometer alone is good enough here, and a smoothing step (80 % old value, 20 % new) removes small shakes. Both angles are measured from "straight ahead", which is saved at the start and whenever the helper presses the button.

The angles are then sorted into zones: middle, right, left or nod. Within 15° of straight ahead nothing happens; that dead zone lets the person relax, look around a little and talk to people without choosing anything by accident. A tilt has to last 250 ms before it counts, and while it continues, the highlight moves again every 900 ms. A nod has to be held a whole second, and it speaks only once until the head comes back up. These hold times make it hard to "say" something by mistake, which matters much more in a talking device than speed.

The ESP32 sends next, prev and say over a WebSocket, plus the head position ten times a second for the little dot. The tablet sends commands back the other way: zero, and help, which plays the alarm on the buzzer. To speak, the page uses the browser's Web Speech API: new SpeechSynthesisUtterance("I'm thirsty") and speechSynthesis.speak(). The voice comes from the tablet itself, so no internet connection or extra parts are needed.

Professional AAC devices use the same ideas: scanning through choices, dwell or hold times to select, and settings that can be changed for each person. Some are controlled with the eyes, a single button pressed with the cheek, or a puff of breath.

Try this next

• Change the phrases to the ones the person needs most, in their own language. Add u.lang = "ar-SA"; (or "de-DE", "et-EE"…) in say() to choose a voice for that language.
• Make the grid 4 × 3 with 12 phrases, and add a "page 2" card that switches to a second set of phrases.
• Let the helper change TILT and NOD_MS from the page, with two sliders that send new values to the ESP32.
• Send an ntfy message to the family's phones when "Help!" is chosen, like in the fall detector.

Troubleshooting

• The page never says anything: tap the blue start screen first, and check that the tablet is not on silent. On iPhones, the ring/silent switch also mutes web pages.
• The highlight jumps around by itself: the headband is loose or the dead zone is too small. Fix the sensor firmly and raise TILT to 20.
• It is hard to reach the nod: lower TILT to 10, or NOD_MS to 700. Ask the person what feels comfortable.
• The dot doesn't move: check SDA on GPIO 21 and SCL on GPIO 22, and VCC on 3V3.
• talkingboard.local doesn't open: some phones don't understand .local names. Type the number from the Serial Monitor instead, with http:// in front.
• It never connects to Wi-Fi: check the network name and password (both are case-sensitive) and make sure the network is 2.4 GHz.