Can a computer see your hand? With Teachable Machine, a free tool from Google, you can train an image recognition model in your browser in a few minutes, just by showing it examples on your webcam. In this lesson you train it to recognise rock, paper and scissors. A web page on your computer watches your hand, counts down "rock, paper, scissors, shoot!", and sends your move to the ESP32 over the USB cable. The ESP32 picks its own move at random, points to it on the printed dial with a servo, and lights green if you win, red if it wins, or yellow for a draw.
What you'll learn
- How to collect images and train an image classifier with Teachable Machine
- What classes, training images and confidence are
- How a web page talks to the ESP32 over USB with Web Serial
- Why a model needs a "nothing" class
What you need
- ESP32 board, expansion board and USB cable (ESP32 IoT Starter Kit)
- Servo motor SG90 with horns and screws (ESP32 IoT Starter Kit)
- Green, yellow and red LEDs and 3 resistors, 220 Ω (ESP32 IoT Starter Kit)
- Passive buzzer (ESP32 IoT Starter Kit)
- A computer with a webcam and Google Chrome or Microsoft Edge
- The printed parts from the 3D printing box below (game dial and pointer), or the cardboard version described there
- Breadboard and jumper wires (ESP32 IoT Starter Kit)
Wiring
| From | To |
|---|---|
| Servo signal (orange) | ESP32 GPIO 18 |
| Servo power (red) and GND (brown) | 5V and GND |
| ESP32 GPIO 25 | 220 Ω resistor, then the green LED (you win) |
| ESP32 GPIO 33 | 220 Ω resistor, then the yellow LED (draw) |
| ESP32 GPIO 32 | 220 Ω resistor, then the red LED (AI wins) |
| All LED short legs | GND |
| Passive buzzer + | ESP32 GPIO 26 |
| Passive buzzer − | GND |
Train the model
- Open Teachable Machine and start a new Image project (standard image model).
- Rename the classes to Rock, Paper and Scissors, and add a fourth class called Nothing.
- For each class, click Webcam and hold the button to record about 100 images of your hand making that move. Move your hand around a bit, nearer and further. For Nothing, record the empty background and your face without a hand.
- Click Train Model, then try it in the preview on the right.
- Click Export Model, keep Upload (shareable link) and click Upload my model. Copy the link it shows, which starts with
https://teachablemachine.withgoogle.com/models/.
Training happens in your browser. When you upload, only the trained model is stored online, not your webcam pictures.
The ESP32 sketch
In the Arduino IDE, open Library Manager and install the ESP32Servo library by Kevin Harrington.
Find the three servo angles at which the pointer shows rock, paper and scissors on your dial, and put them into ANGLE.
#include <ESP32Servo.h>
const int SERVO_PIN = 18, GREEN_PIN = 25, YELLOW_PIN = 33, RED_PIN = 32, BUZZER_PIN = 26;
const int ANGLE[3] = {20, 90, 160}; // rock, paper, scissors on the dial
const char* NAMES[3] = {"rock", "paper", "scissors"};
Servo pointer;
int myScore = 0, yourScore = 0;
int moveFrom(char c) {
if (c == 'R') return 0;
if (c == 'P') return 1;
if (c == 'S') return 2;
return -1;
}
void lights(int g, int y, int r) {
digitalWrite(GREEN_PIN, g);
digitalWrite(YELLOW_PIN, y);
digitalWrite(RED_PIN, r);
}
void play(int you) {
int me = esp_random() % 3; // a truly random choice
pointer.write(ANGLE[me]);
int result = (3 + you - me) % 3; // 0 draw, 1 you win, 2 I win
const char* winner = "draw";
if (result == 0) { lights(0, 1, 0); tone(BUZZER_PIN, 660, 300); }
if (result == 1) { lights(1, 0, 0); yourScore++; winner = "you"; tone(BUZZER_PIN, 880, 150); delay(170); tone(BUZZER_PIN, 1175, 300); }
if (result == 2) { lights(0, 0, 1); myScore++; winner = "ai"; tone(BUZZER_PIN, 300, 400); }
// one line back to the web page
Serial.printf("RESULT,%s,%s,%s,%d,%d\n", NAMES[you], NAMES[me], winner, yourScore, myScore);
}
void setup() {
Serial.begin(115200);
pinMode(GREEN_PIN, OUTPUT);
pinMode(YELLOW_PIN, OUTPUT);
pinMode(RED_PIN, OUTPUT);
pointer.attach(SERVO_PIN);
pointer.write(ANGLE[1]);
Serial.println("READY");
}
void loop() {
if (!Serial.available()) return;
String line = Serial.readStringUntil('\n');
line.trim();
if (line == "COUNT") { // the countdown: three beeps
lights(0, 0, 0);
tone(BUZZER_PIN, 1000, 80);
} else if (line.startsWith("MOVE,") && line.length() >= 6) {
int you = moveFrom(line[5]);
if (you >= 0) play(you);
} else if (line == "RESET") {
myScore = yourScore = 0;
lights(0, 0, 0);
}
}The game page
Copy the code below into a text editor (Notepad on Windows, TextEdit in plain-text mode on a Mac), save it as rock-paper-scissors.html on your computer and open it in Google Chrome or Microsoft Edge. Other browsers don't support Web Serial yet. Close the Arduino IDE's Serial Monitor first: only one program can use the port.
<!doctype html>
<html><head><meta charset="utf-8"><title>Rock, paper, scissors vs AI</title>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@1.3.1/dist/tf.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/@teachablemachine/image@0.8/dist/teachablemachine-image.min.js"></script>
<style>
body{font-family:sans-serif;max-width:720px;margin:20px auto;padding:0 16px;text-align:center}
input{width:100%;padding:8px;box-sizing:border-box}button{padding:12px 18px;margin:6px;font-size:16px;border:0;border-radius:10px;background:#1B3A7B;color:#fff}
#cam canvas{border-radius:14px}#big{font-size:56px;font-weight:bold;height:70px}#score{font-size:24px}
</style></head><body>
<h1>Rock, paper, scissors vs AI</h1>
<p>Your Teachable Machine model link:<br><input id="url" placeholder="https://teachablemachine.withgoogle.com/models/..."></p>
<button onclick="connect()">1. Connect ESP32</button><button onclick="start()">2. Start camera</button><button onclick="round()">3. Play!</button>
<div id="cam"></div><p id="see">-</p><div id="big"></div><p id="score">You 0 : 0 AI</p><p id="log"></p>
<script>
let model, webcam, writer, reader, buffer = '', best = null;
async function connect() {
const port = await navigator.serial.requestPort(); // pick the ESP32's port
await port.open({ baudRate: 115200 });
writer = port.writable.getWriter();
const decoder = new TextDecoderStream();
port.readable.pipeTo(decoder.writable);
reader = decoder.readable.getReader();
readLines();
}
async function readLines() {
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += value;
let i;
while ((i = buffer.indexOf('\n')) >= 0) {
const line = buffer.slice(0, i).trim();
buffer = buffer.slice(i + 1);
if (line.startsWith('RESULT,')) showResult(line.split(','));
}
}
}
function send(text) {
if (writer) writer.write(new TextEncoder().encode(text + '\n'));
}
async function start() {
let url = document.getElementById('url').value.trim();
if (!url.endsWith('/')) url += '/';
model = await tmImage.load(url + 'model.json', url + 'metadata.json');
webcam = new tmImage.Webcam(320, 320, true); // true: mirror, like a mirror
await webcam.setup();
await webcam.play();
document.getElementById('cam').appendChild(webcam.canvas);
requestAnimationFrame(watch);
}
async function watch() {
webcam.update();
const predictions = await model.predict(webcam.canvas);
best = predictions.reduce((a, b) => (b.probability > a.probability ? b : a));
document.getElementById('see').textContent = 'I see: ' + best.className + ' (' + Math.round(best.probability * 100) + ' %)';
requestAnimationFrame(watch);
}
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function round() {
const big = document.getElementById('big');
for (const word of ['Rock...', 'Paper...', 'Scissors...']) {
big.textContent = word;
send('COUNT');
await sleep(700);
}
big.textContent = 'Shoot!';
await sleep(300); // a moment to show your hand
const letter = best ? best.className.trim().charAt(0).toUpperCase() : 'N';
if ('RPS'.includes(letter) && best.probability > 0.6) send('MOVE,' + letter);
else big.textContent = 'I can\'t see your hand!';
}
function showResult(p) {
const text = { you: 'You win!', ai: 'AI wins!', draw: 'Draw!' }[p[3]];
document.getElementById('big').textContent = text;
document.getElementById('log').textContent = 'You: ' + p[1] + ', AI: ' + p[2];
document.getElementById('score').textContent = 'You ' + p[4] + ' : ' + p[5] + ' AI';
}
</script></body></html>Play
- Upload the sketch and close the Serial Monitor. Open the page and paste your model link.
- Click Connect ESP32 and pick the ESP32's port. Click Start camera and allow the camera.
- Check that "I see" follows your hand. Then click Play!, make your move when it says "Shoot!", and watch the pointer.
- Play ten rounds. Who wins more often?
How it works
Teachable Machine uses transfer learning: it starts from a big image model that Google already trained on millions of pictures, which has learned to spot edges, shapes and textures. You only teach the last layer to tell your four classes apart, which is why 100 pictures and a minute of training are enough.
For each video frame, the model gives a probability for every class. The page takes the highest one, but only trusts it above 60 %. The Nothing class matters: without it, the model would have to call an empty picture rock, paper or scissors, because those are the only answers it knows.
Web Serial lets a web page talk to a board on a USB cable, after you pick the port and allow it. The page sends one line of text per message, and the ESP32 answers with lines of its own. A page opened from a file on your own computer counts as safe, so the browser also lets it use the camera and the microphone, which it wouldn't for a page served by the ESP32 over plain http.
The ESP32 picks its move with esp_random(), a hardware random number generator, at the moment it receives yours, so it can't cheat. The sum (3 + you - me) % 3 works out the winner in one line: in rock-paper-scissors, each move beats the one before it in the list.
Try this next
- Make the AI cheat: let it see your move and always win. Is it still fun?
- Add "lizard" and "Spock" as two more classes.
- Keep a record of your moves and let the ESP32 guess what you'll play next.
Troubleshooting
- "I see" is always the same class: record more varied images, with your hand in different places and lighting.
- Connect does nothing: use Chrome or Edge on a computer, and close the Arduino Serial Monitor.
- The model won't load: check the model link and that you clicked "Upload my model".
- The board restarts when the servo moves: servos draw bursts of current. Use a good USB port or a powered hub, and check that the servo's red wire is on 5V.
- A printed part is too tight or too loose: printers differ by a few tenths of a millimetre. Sand or file a tight hole, and fix a loose part with a drop of hot glue or a strip of tape.
