How I trained sheet reading using the Web MIDI API

July 07, 2015 • hack

As an enthusiastic developer and piano player, I like to connect my interests. Since my sheet reading skills have always been rather moderate, I saw an opportunity to improve them with the new Web MIDI API. It allows the browser to communicate with a MIDI-enabled device (especially receiving MIDI notes from it).

Using this API, I built a side project called Piano-Trainer (Github link) which is a web app that renders random music sheets. The user has to play the notes on a real piano (provided that it supports MIDI) and gets feedback when played correctly. In theory it should also work with other MIDI-enabled instruments.

Piano-Trainer will measure the time it took you to play the displayed chords and display statistics to keep track of your progress. Think of it as Guitar Hero for the piano. Currently, the rhythm is not checked (so it does not matter if you need some seconds to play a chord). But being able to train the rhythm, is certainly a feature I want to work on in the future.

Screenshoto of Piano Trainer

After having played the chord correctly, it is highlighted as green and you can play the next chord. If you play a wrong note, your only feedback is that the chord doesn’t get green. Since reading real sheets does not give any additional feedback either, I wanted to restrict the feedback to a minimum.

Implementation

Before the Web MIDI API existed, accessing MIDI devices in the browser was only possible with workarounds (e.g. Java applets or special browser extensions). Today, the Web MIDI API only needs the user’s permission. Then, it is ready to go (provided that the browser supports it).

The following code shows a simple demo of the API:

function init() {
  if(!navigator.requestMIDIAccess) {
    // handle unsupported browser
  }
  this.currentInputState = {};
  this.midiPromise = navigator.requestMIDIAccess({sysexEnabled: true});
  this.midiPromise.then(onMidiAccess, errorCallback);
}

function onMidiAccess(midi){
  var inputs = midi.inputs();
  if(inputs.length == 0) {
    // handle no inputs
  }

  var input = inputs[0];
  input.onmidimessage = onMidiMessage;
}

function onMidiMessage(msg){
  // ignore "active sensing"-event
  if (msg.data[0] == 254) {
    return;
  }

  var status = msg.data[0];
  var note = msg.data[1];
  var velocity = msg.data[2];

  if (status <= 143) {
    // off event
    intensity = 0
  }

  if (status <= 159) {
    // on or off event
    setNote(note, velocity);
  }

}

function setNote(note, velocity){
  if (velocity == 0) {
    delete this.currentInputState[note];
  } else {
    this.currentInputState[note] = true;
  }
}

The onMidiMessage function receives a message event which contains a data array of size 3. The first element is the status byte, the second and third are data bytes which mainly give information about the note number and note velocity. If you want to find more information about the data and what it encodes, take a look at this table.

In order to convert the integer which encodes the played note to a meaningful representation, I wrote a small “converter”. So, if you are interested in finding out that the note number 22 encodes a#/0, take a look at this code. Using the library VexFlow, you can instantiate a Vex.Flow.StaveNote with such a string representation which can be used to plot sheets in a canvas.

new Vex.Flow.StaveNote({ keys: ["c/4"], duration: "q" })

For a more extensive example of how to use VexFlow, visit VexFlow’s tutorial.

With the Web MIDI API and VexFlow, it is only a matter of generating and rendering random notes, listening to the midi device and comparing the requested notes with the received ones.

At the moment, I only generate notes in the C major scale, in four-four time (with four quarter notes) while ensuring that the maximum interval per hand does not exceed 12 half steps (so that the hand can hit all notes at once). Of course, this should be customizable to adapt to the training objective of the user.

Learning success

I have played piano for almost eight years now, but I never felt that I made a huge progress in my sheet reading skills (I would rather call it deciphering than reading). For a long time, I thought that playing piano regularly would be enough to improve my sheet reading skills. Unfortunately, it actually caused me to memorize the notes of a song instead of learning to read the notes faster.

That is why Piano-Trainer generates random notes without any patterns. This avoids that the user memorizes the notes or patterns.

So, let’s crunch some numbers. The Piano-Trainer web app collects data about the user’s performance and saves it to LocalStorage. In total, I practiced for 2.5 hours. That is not really long, compared to my total piano practice time within the last 8 years.

Here is a scatter plot of the time needed to play a note. The data was obtained by dividing the time needed to play a chord by the number of notes within each chord.

Development of time over time

Does not look like a huge learning success, does it?

Let’s smooth these numbers by sliding a window over the data and taking the mean of each window. Here’s a little animation where the window size increases over time:

A window size of 300 results in the following plot:

Smoothed development

On average I improved from 1723 ms per note to 1076 ms per note, which I think of as a satisfactory improvement regarding that I only practiced for 2.5 hours. Total beginners would probably have an even better learning curve.

All numbers aside, I’m also feeling a vast improvement while playing piano. Instead of reading each note separately, I’m seeing patterns (within each chord) and distances which really astonished me. Of course, there is still room for improvement and I’m looking forward to the challenge.

Hopefully, this motivates some of you to train your sheet reading skills, too. Just visit Piano-Trainer and start practicing (I highly recommend to use Chrome). I would also advise you to turn the volume of your piano down. The random notes do not really sound nice ;)

by Philipp Otto


Related posts