Teach a servo to find the light with reinforcement learning

In the sunflower lesson the UNO scanned every angle and simply picked the brightest one. That works, but it isn't learning: we told it exactly what to do. This time we only tell the servo what is good, more light, and let it find out how to get there. This is called reinforcement learning, the same idea that taught computers to play chess, Go and video games better than people. The UNO tries a move, gets a reward or a penalty, and remembers it in a table. After a minute or two it has learned, for every position, whether to turn left, turn right or stay.

What you'll learn

• What an agent, a state, an action and a reward are
• How a Q-table stores what the agent has learned
• Why an agent sometimes has to try random moves (exploration)
• How the learning rule updates the table after every move

What you need

• UNO R3 board and USB cable (UNO R3 Starter Kit)
• Servo motor SG90 with horns and screws (UNO R3 Starter Kit)
• Photoresistor module (45-in-1 Sensor Kit)
• Red LED, green LED and 2 resistors, 220 Ω (UNO R3 Starter Kit)
• 1 push button (UNO R3 Starter Kit)
• A torch or the light of a phone
• The printed parts from the 3D printing box below (servo stand and seeker arm), or the cardboard version described there
• Breadboard and jumper wires (UNO R3 Starter Kit)

Wiring

Sensor modules from different makers can label and order their pins differently. Always follow the labels printed on your module.

Check your sensor

1. Upload the sketch and open the Serial Monitor at 9600 baud.
2. Cover the sensor with your hand and watch the light number in the first lines.
3. If the number goes up when it gets dark, leave BRIGHTER_IS_HIGHER at false. If it goes down, change it to true and upload again. Either way, after this a bigger number always means more light.

The sketch

#include <Servo.h>

const int LIGHT_PIN = A0;
const int SERVO_PIN = 9;
const int TRYING_LED = 6, SURE_LED = 5, BUTTON_PIN = 2;
const bool BRIGHTER_IS_HIGHER = false;   // see "Check your sensor"

const int POSITIONS = 13;                // 0, 15, 30 ... 180 degrees
const int LEFT = 0, STAY = 1, RIGHT = 2;
const float ALPHA = 0.3;                 // learning rate: how much one move changes the table
const float GAMMA = 0.8;                 // how much the agent cares about later rewards

float Q[POSITIONS][3];                   // the agent's "brain": a score for every move in every position
float epsilon = 0.9;                     // chance of trying a random move
int pos = 6;                             // start in the middle (90 degrees)
int steps = 0;
Servo arm;

int readLight() {                        // average of 8 readings, bigger = brighter
  long sum = 0;
  for (int i = 0; i < 8; i++) {
    sum += analogRead(LIGHT_PIN);
    delay(2);
  }
  int value = sum / 8;
  return BRIGHTER_IS_HIGHER ? value : 1023 - value;
}

int best(int s) {                        // the move with the highest score; ties go to STAY
  int b = STAY;
  if (Q[s][LEFT] > Q[s][b]) b = LEFT;
  if (Q[s][RIGHT] > Q[s][b]) b = RIGHT;
  return b;
}

float bestScore(int s) {
  return max(Q[s][LEFT], max(Q[s][STAY], Q[s][RIGHT]));
}

void forget() {
  for (int s = 0; s < POSITIONS; s++)
    for (int a = 0; a < 3; a++) Q[s][a] = 0;
  epsilon = 0.9;
  Serial.println("Forgot everything. Learning from scratch.");
}

void printPolicy() {                     // < turn left, o stay, > turn right; [ ] = where it is now
  Serial.print("Policy: ");
  for (int s = 0; s < POSITIONS; s++) {
    int a = best(s);
    Serial.print(s == pos ? '[' : ' ');
    Serial.print(a == LEFT ? '<' : a == RIGHT ? '>' : 'o');
    Serial.print(s == pos ? ']' : ' ');
  }
  Serial.print("  random moves: ");
  Serial.print((int)(epsilon * 100));
  Serial.println("%");
}

void setup() {
  Serial.begin(9600);
  pinMode(TRYING_LED, OUTPUT);
  pinMode(SURE_LED, OUTPUT);
  pinMode(BUTTON_PIN, INPUT_PULLUP);
  arm.attach(SERVO_PIN);
  arm.write(pos * 15);
  randomSeed(analogRead(A1));
  delay(800);
  forget();
}

void loop() {
  // Button: a short press makes the agent curious again, holding it for 2 s wipes its memory
  if (digitalRead(BUTTON_PIN) == LOW) {
    unsigned long pressed = millis();
    while (digitalRead(BUTTON_PIN) == LOW) delay(10);
    if (millis() - pressed > 2000) forget();
    else {
      epsilon = 0.6;
      Serial.println("Exploring again...");
    }
  }

  int before = readLight();

  // 1. Choose a move: random (explore) or the best one so far (exploit)
  bool exploring = random(1000) < epsilon * 1000;
  int action = exploring ? random(3) : best(pos);
  digitalWrite(TRYING_LED, exploring);
  digitalWrite(SURE_LED, !exploring);

  // 2. Do it
  int next = pos + (action == LEFT ? -1 : action == RIGHT ? 1 : 0);
  bool bumped = next < 0 || next >= POSITIONS;
  next = constrain(next, 0, POSITIONS - 1);
  arm.write(next * 15);
  delay(250);                            // let the servo arrive

  // 3. Reward: more light is good, bumping into the end is bad
  int after = readLight();
  float reward = (after - before) / 40.0;
  if (bumped) reward = -1;

  // 4. Learn: nudge the score of this move towards reward + the best score of the new position
  float target = reward + GAMMA * bestScore(next);
  Q[pos][action] += ALPHA * (target - Q[pos][action]);

  pos = next;
  epsilon = max(0.05, epsilon * 0.97);   // fewer random moves as it learns
  steps++;
  if (steps % 5 == 0) {
    Serial.print("light ");
    Serial.print(after);
    Serial.print("  ");
    printPolicy();
  }
}

Try it

1. Put the light seeker on a table in a dim room and shine the torch at it from one side, about 50 cm away. Keep the torch still.
2. Watch the Serial Monitor. At first almost every move is random and the red "trying" LED is on most of the time. The policy line is full of o.
3. After a minute, arrows appear: on one side of the torch they point >, on the other side <, and near the torch an o. The green "sure" LED is on more and more, and the arm settles near the light.
4. Move the torch to the other side and press the button once. The agent becomes curious again and re-learns. Hold the button for 2 seconds to wipe its memory completely.

How it works

Reinforcement learning has four ingredients. The agent is the learner, here the UNO with its servo. The state is what it knows about the world: its position, one of 13 angles. The actions are what it can do: turn left, stay or turn right. The reward is a number that tells it how good a move was: how much brighter it got.

Everything the agent learns lives in the Q-table, 13 rows by 3 columns. Each cell is a score for "in this position, this move". After every move the sketch nudges one cell with the learning rule:

Q = Q + ALPHA × (reward + GAMMA × best score in the new position − Q)

The part with GAMMA is the clever bit. A move that doesn't make it brighter yet, but leads to a position where good moves are possible, also gets credit. That is how the arrows grow outwards from the torch, step by step, until even the far ends know which way to go.

Why random moves? If the agent always did what looked best so far, it would repeat its first lucky guess for ever and never discover the brightest spot. epsilon is the chance of a random move: it starts at 90 % and shrinks a little after every step. Scientists call this the balance between exploring and exploiting, and every learning agent, from a robot dog to a chess program, has to get it right.

Try this next

• Change ALPHA to 0.05 and to 0.9. Which learns faster, and which is calmer?
• Set GAMMA to 0. Do arrows still appear far from the torch? Why not?
• Give a small penalty for every move (reward -= 0.05) and see whether the arm stops wobbling sooner.
• Use two torches of different brightness. Does the agent find the brighter one, or get stuck at the nearer one?

Troubleshooting

• The arrows point away from the light: change BRIGHTER_IS_HIGHER (see "Check your sensor").
• It never settles: the room is too bright or the torch too weak. Darken the room, move the torch closer, and check that the tube points straight ahead.
• The light number jumps when the servo moves: wait a little longer after each move, for example delay(400).
• The board restarts when the servo moves: servos draw bursts of current. Use a good USB port or a powered hub, and check that the servo's red wire is on 5V.
• A printed part is too tight or too loose: printers differ by a few tenths of a millimetre. Sand or file a tight hole, and fix a loose part with a drop of hot glue or a strip of tape.