Early in the first few generations of iPads, we were contracted to develop the AtPlayMusic applications, a series of educational applications to teach K-12 students how to play a variety of instruments.

Over the years we worked on it, it developed a collection of features of which we’re really proud.

We trained a neural network to evaluate guitar chords to determine if the user was playing the correct chord or not. Compared to identifying single notes via FFTs, chord detection is a much more difficult and interesting problem, and it was largely unsolved in 2013. Destructive interference between each individual note in a chord, plus all of the overtones, plus the body of the instrument result in an FFT profile that isn’t identifiably equivalent to the sum of the individual notes in the chord. We trained a neural network on the output of an FFT using libFANN, and evaluated the network on-device in realtime for the chord versions of the app.

• Fully customizable dressing room for your avatar
• 25+ apps in the suite, representing pretty much every instrument in a high school band
• Scripted and animated lesson content for the recorder and guitar
• Sophisticated build system in Unity to automate building of each app variant
• Rendered and played back music using a format similar to the ABC Music Notation format.
• Collaborated with the MN Symphonic orchestra for instrument samples and promotion.
• wrote a few gamified versions using our tech, such as the NoteRacer / ChordRacer apps.

The app got great reviews and hearing feedback from students and teachers was very endearing.

This is one of several projects we’ve done with customizable avatars.