Length: Honours and Master projects run over two semesters.
Short project: Also available as a six-week summer research scholarship or a one-semester COMP3740 project (see the short-project scope below).
Prerequisites: C++ and some embedded or audio programming (e.g., COMP2300 or COMP4350/COMP8350 (Sound and Music Computing)). Some machine learning background is helpful.
About this project
This is a research project in the Sound, Music and Creative Computing Lab (SMCClab) at the ANU School of Computing, supervised by Charles Martin. The lab studies intelligent musical instruments: instruments with a built-in AI model that generates musical actions of its own, responding to the musician or playing alongside them. Most of our projects use IMPSY, the lab’s open-source system that records a musician’s gestures from a MIDI controller, trains a small neural network on that data, and then plays back and responds in live performance on a Raspberry Pi, laptop, iPad, or in a web browser. You don’t need to know these systems before you start: the background reading at the end of this page is the place to begin.
Bela Gem is the new generation of the Bela embedded audio platform, built on the PocketBeagle 2 (quad-core ARM Cortex-A53, 512MB RAM). It offers sub-millisecond audio latency, 8 analog sensor inputs, 16 digital I/O, and USB MIDI. The Gem Multi version has 10 audio channels, including 8 DC-coupled outputs that can drive modular synthesisers directly. The lab has a Bela Gem Multi and an 8-channel audio system at the ANU School of Music for these projects.
This project
IMPSY currently runs on a Raspberry Pi connected to external MIDI instruments. Bela Gem could combine sensing, the AI model, and sound synthesis in one low-latency device. IMPSY’s models are small (two 64-unit LSTM layers and a mixture density output), so real-time inference on Bela Gem should be well within reach. In this project you’ll write an MDRNN (mixture density recurrent neural network) inference engine in C++ using weights exported from IMPSY’s models, and verify it step by step against IMPSY’s reference test vectors.
- Short project: a C++ MDRNN runtime that passes IMPSY’s reference tests, plus latency and timing benchmarks against the Raspberry Pi version.
- Full project: connect the model to Bela’s sensor inputs and audio synthesis to build a playable standalone intelligent instrument, and evaluate it in performance.
Background reading
- Martin, C. P. and Torresen, J. (2019). An Interactive Musical Prediction System with Mixture Density Recurrent Neural Networks. NIME 2019 (the original IMPSY paper).
- Martin, C. P., Jensenius, A. R. and Torresen, J. (2018). Composing an Ensemble Standstill Work for Myo and Bela. NIME 2018 (embedded Bela instruments in performance).
- Pelinski, T., Diaz, R., Benito Temprano, A. L. and McPherson, A. (2023). Pipeline for recording datasets and running neural networks on the Bela embedded hardware platform. NIME 2023.
More projects in the lab
This is one of more than a dozen projects currently available in the SMCClab, covering machine learning, embedded hardware, HCI, visualisation, theory, and artistic research. Only a selection is listed on the School of Computing website, so see the full list of SMCClab student projects on the lab website. You can also read this project on the lab website.
How to Apply
To apply for this project, contact Charles Martin.
Include:
- your CV
- your unofficial transcript (if you are an ANU student)
- a brief statement (200 words) in your email explaining how you would approach this project
Make sure to specify the skills and accomplishments you have that would help you to complete this project. Please read about joining the SMCClab and our project expectations before applying.