Rock · Paper · Scissors classroom ML, rebuilt
network: untrained
Everything runs in this page. The hands are synthetic, and a real convolutional network (written from scratch in this page's JavaScript) trains on them live. ·The original used students' phone photos and a Keras convnet.

Crowdsource a training set

In the real activity a room of 10–20 people photographed their own hands with an Android app I built; every shot landed, labeled, on a server. Here the phone is simulated: pick a gesture, shoot, and build the pile. Or let a whole classroom do it for you.

mode
train mode: the label you chose is sent with the photo

Simulate a classroom

~5 photos per minute per phone was the real rate; we compress the 10 minutes.

Why this step matters

The network will only ever be as good as this pile. The real crowd sent mislabeled shots and photos with no hand at all. That's why the app had a delete button and the teacher had a moderation queue. Hover a thumbnail to inspect it; click to delete the bad ones.

The training pile

What the server's three folders held. A separate, fixed set of 60 photos is kept aside as the evaluation set; the network never trains on it.

Train it, then meet reality

This reproduces the activity's arc. Train once and watch the training accuracy soar. Then look at the evaluation line. The gap is overfitting, and the fix the activity taught (data augmentation) is one toggle away.

Training run

The network sees only the training pile; the blue line is measured on the 60 held-out photos.

Accuracy, epoch by epoch

Dark: on the photos it trains on. Rose: on photos it has never seen. Dashed: guessing (33%).

training accuracy evaluation accuracy baseline (chance)

How the real one worked

A didactic activity for people with zero ML background (secondary-school students to adults), published at EAAI-18. My job as research assistant: build the machinery that let a classroom be the dataset.

The system

Android app camera · label · upload Java · Retrofit REST service stores by class · auth Python · hug/Falcon Convnet 64×64 in → 3 probs out Keras POST /photo + label test shots [p(rock), p(paper), p(scissors)] → drawn as bars on the phone

Running a classroom is an ops problem

Details the paper doesn't dwell on: the parts I built so the activity wouldn't derail.

Who may uploadEvery phone announced its device ID; new ones landed in a pending queue until the teacher approved them. A blacklist handled the inevitable joker.
Which classroomA session code separated parallel classes; the wrong code got its own HTTP status and an in-app prompt.
PhasesTesting stayed locked (HTTP 412: "it's not time yet!") until the teacher opened it, so nobody could peek before the network was trained.
Mess controlStudents could delete their last shot; the teacher could prune anything. The pile still ended up with black frames and empty desks, a lesson in itself.
Live finaleThe trained net ran on the presenter's laptop against a webcam feed, drawing class probabilities in real time while the audience tried to fool it.

The activity's script, compressed

  1. Frame the problem. Three classes, 100 held-out evaluation photos, a 33% baseline: what would "good" even mean?
  2. The room shoots the dataset. 80–200 photos per class in ~10 minutes, straight into the server's three folders.
  3. Train, and celebrate too early. Training accuracy marches to 100%. Applause.
  4. Evaluate, and deflate. On unseen photos it scores under 50%. The network memorized, it didn't learn. That word: overfitting.
  5. Augment and retrain. Rotate, flip, zoom, recolor: the net never sees the same photo twice. Evaluation climbs to ~75%.
  6. Play against it live. Webcam, probability bars, and a room trying its best to break their own model.
What's smaller here: this page trains a ~3,600-parameter convnet (two conv layers + pooling) on 24×24 grayscale drawings so it converges in seconds; the classroom trained a Keras convnet on 64×64 color photos for a few minutes. Same curves, same lesson.

The activity and its paper, Introducing Machine Learning Concepts by Training a Neural Network to Recognize Hand Gestures (Giusti, Huber & Gambardella, EAAI-18), are the work of my IDSIA colleagues; I built the data-collection app, the web service and the model-serving around it, which this page recreates. Original repository ↗