Train a magic wand that recognises your gestures with Edge Impulse

Real AI doesn't always live in the cloud. TinyML runs small neural networks right on a microcontroller like the ESP32, with no internet at all. In this lesson you build a magic wand. You record yourself drawing a circle, waving and flicking the wand, upload the movements to Edge Impulse, a free website for training TinyML models, and train a neural network there. Then you download it as an Arduino library and run it on the ESP32. Draw a circle in the air, and the star lights up blue; wave, and it turns green.

What you'll learn

• How to collect training data from a sensor
• How a neural network is trained to recognise patterns in movement
• What "inference" on a microcontroller means
• How to read a model's confidence and decide when to trust it

What you need

• ESP32 board, expansion board and USB cable (ESP32 IoT Starter Kit)
• GY-521 MPU-6050 module (45-in-1 Sensor Kit)
• RGB LED module (45-in-1 Sensor Kit)
• 3 resistors, 220 Ω (ESP32 IoT Starter Kit)
• Passive buzzer (ESP32 IoT Starter Kit)
• 8 long female-to-male jumper wires
• A computer with Node.js and the Edge Impulse CLI, and a free Edge Impulse account (an adult's help to sign up)
• The printed parts from the 3D printing box below (magic wand), or the cardboard version described there
• Breadboard and jumper wires (ESP32 IoT Starter Kit)

Wiring

Sensor modules from different makers can label and order their pins differently. Always follow the labels printed on your module.

Step 1: record your gestures

This first sketch only sends the wand's movement to the computer, 100 times a second, as three numbers per line. Edge Impulse's data forwarder passes them on to your project.

#include <Wire.h>

const int MPU = 0x68;

// Acceleration in m/s² on all three axes (1 g = 9.81 m/s²)
void readAccel(float& x, float& y, float& z) {
  Wire.beginTransmission(MPU);
  Wire.write(0x3B);
  Wire.endTransmission(false);
  Wire.requestFrom(MPU, 6);
  int16_t ax = Wire.read() << 8 | Wire.read();
  int16_t ay = Wire.read() << 8 | Wire.read();
  int16_t az = Wire.read() << 8 | Wire.read();
  x = ax / 16384.0 * 9.81;
  y = ay / 16384.0 * 9.81;
  z = az / 16384.0 * 9.81;
}

void setup() {
  Serial.begin(115200);

  Wire.begin(21, 22);
  Wire.beginTransmission(MPU);
  Wire.write(0x6B);                 // wake the MPU-6050 up
  Wire.write(0);
  Wire.endTransmission();
}

void loop() {
  static unsigned long next = micros();
  if ((long)(micros() - next) < 0) return;
  next += 10000;                     // exactly 100 samples a second
  float x, y, z;
  readAccel(x, y, z);
  Serial.print(x, 3);
  Serial.print(',');
  Serial.print(y, 3);
  Serial.print(',');
  Serial.println(z, 3);
}

1. Create a free account at Edge Impulse and a new project called Magic Wand.
2. Install Node.js on your computer and then the Edge Impulse CLI, following the install guide on Edge Impulse's documentation (the command is npm install -g edge-impulse-cli).
3. Upload the recorder sketch, close the Serial Monitor, and run edge-impulse-data-forwarder in a terminal. Log in, pick your project, and name the three axes accX,accY,accZ.
4. In Edge Impulse, open Data acquisition. Record 2-second samples with the label circle (draw a circle in the air), wave (left-right-left), flick (a quick up-down) and idle (holding the wand still or moving it slowly).
5. Record about 30 samples of each, at different speeds and sizes. Variety makes the model clever.

Step 2: train the model

1. Open Create impulse: set the window size to 2000 ms, add a Spectral Analysis processing block and a Classification learning block, and save.
2. Open Spectral features, click Save parameters and then Generate features. The feature explorer shows your gestures as clusters of dots: the further apart, the easier they are to tell apart.
3. Open Classifier and click Start training. After a minute you see the accuracy and a table of which gestures it mixes up.
4. Open Model testing to try it on samples it hasn't seen. Above 85 % is good. If it is lower, record more samples.
5. Open Deployment, choose Arduino library and click Build. In the Arduino IDE, use Sketch → Include Library → Add .ZIP Library to install the downloaded file.

Step 3: run it on the wand

Edge Impulse names the library after your project. If yours isn't called "Magic Wand", change the #include line to the name of the library you installed (see File → Examples). Change the labels in react() if you used other names.

#include <Wire.h>
#include <Magic_Wand_inferencing.h>   // the name of your Edge Impulse library

const int MPU = 0x68;
const int RED_PIN = 25, GREEN_PIN = 26, BLUE_PIN = 27, BUZZER_PIN = 33;
const float SURE = 0.80;             // only act when the model is at least 80 % sure

float buffer[EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE];

// Acceleration in m/s² on all three axes (1 g = 9.81 m/s²)
void readAccel(float& x, float& y, float& z) {
  Wire.beginTransmission(MPU);
  Wire.write(0x3B);
  Wire.endTransmission(false);
  Wire.requestFrom(MPU, 6);
  int16_t ax = Wire.read() << 8 | Wire.read();
  int16_t ay = Wire.read() << 8 | Wire.read();
  int16_t az = Wire.read() << 8 | Wire.read();
  x = ax / 16384.0 * 9.81;
  y = ay / 16384.0 * 9.81;
  z = az / 16384.0 * 9.81;
}

void colour(int r, int g, int b) {
  analogWrite(RED_PIN, r);
  analogWrite(GREEN_PIN, g);
  analogWrite(BLUE_PIN, b);
}

void react(String label) {
  if (label == "circle") { colour(0, 0, 255); tone(BUZZER_PIN, 880, 300); }
  else if (label == "wave") { colour(0, 255, 0); tone(BUZZER_PIN, 660, 300); }
  else if (label == "flick") { colour(255, 0, 255); tone(BUZZER_PIN, 1320, 150); }
  else return;                       // idle: nothing to do
  delay(1000);
  colour(0, 0, 0);
}

void setup() {
  Serial.begin(115200);
  pinMode(RED_PIN, OUTPUT);
  pinMode(GREEN_PIN, OUTPUT);
  pinMode(BLUE_PIN, OUTPUT);

  Wire.begin(21, 22);
  Wire.beginTransmission(MPU);
  Wire.write(0x6B);                 // wake the MPU-6050 up
  Wire.write(0);
  Wire.endTransmission();
  Serial.println("Wave your wand!");
}

void loop() {
  // 1. record one window of movement, at the same rate as the training data
  for (size_t i = 0; i < EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE; i += 3) {
    unsigned long next = micros() + EI_CLASSIFIER_INTERVAL_MS * 1000;
    readAccel(buffer[i], buffer[i + 1], buffer[i + 2]);
    while ((long)(micros() - next) < 0) {}
  }

  // 2. let the neural network look at it
  signal_t signal;
  numpy::signal_from_buffer(buffer, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, &signal);
  ei_impulse_result_t result = {0};
  if (run_classifier(&signal, &result, false) != EI_IMPULSE_OK) {
    Serial.println("The classifier failed");
    return;
  }

  // 3. find the most likely gesture
  int best = 0;
  for (size_t i = 0; i < EI_CLASSIFIER_LABEL_COUNT; i++) {
    Serial.printf("%s %.2f  ", result.classification[i].label, result.classification[i].value);
    if (result.classification[i].value > result.classification[best].value) best = i;
  }
  Serial.println();
  if (result.classification[best].value >= SURE) react(result.classification[best].label);
}

How it works

A gesture is a pattern in time: a circle makes the sideways and up-down acceleration rise and fall one after the other, a wave swings one axis back and forth. Edge Impulse's spectral analysis turns each 2-second window into a list of features, such as how strongly each axis moves and how fast it wiggles.

The neural network is many artificial neurons in layers, like the one in lesson 1, but with thousands of weights. During training, Edge Impulse shows it your labelled samples over and over and nudges every weight to make fewer mistakes. The finished network is small enough to fit in the ESP32's memory.

Running the trained network on new data is called inference. For every window, it gives a confidence for each label, adding up to 1. The wand only reacts when one gesture is at least 80 % likely, otherwise random movements would trigger spells. Everything happens on the ESP32, without the internet.

Try this next

• Add a fourth spell, like a zigzag, and retrain.
• Use the gestures as a remote control: a circle switches a lamp on with the relay.
• Let a friend use the wand. Does it still work? Record some of their samples too.

Troubleshooting

• The data forwarder finds no data: close the Arduino Serial Monitor; only one program can use the port.
• Compile error at the #include line: use the exact library name from File → Examples.
• It recognises nothing: hold the wand the same way as when you recorded, and check that idle has plenty of samples too.
• Everything is "circle": record more samples of the other gestures, and keep the numbers of samples per gesture similar.
• A printed part is too tight or too loose: printers differ by a few tenths of a millimetre. Sand or file a tight hole, and fix a loose part with a drop of hot glue or a strip of tape.