Length: Honours and Master projects run over two semesters.
Short project: Also available as a six-week summer research scholarship or a one-semester COMP3740 project (see the short-project scope below).
Prerequisites: Python and some machine learning (e.g., COMP3670 or COMP4670). Experience with Keras/TensorFlow is helpful.
About this project
This is a research project in the Sound, Music and Creative Computing Lab (SMCClab) at the ANU School of Computing, supervised by Charles Martin. The lab studies intelligent musical instruments: instruments with a built-in AI model that generates musical actions of its own, responding to the musician or playing alongside them. Most of our projects use IMPSY, the lab’s open-source system that records a musician’s gestures from a MIDI controller, trains a small neural network on that data, and then plays back and responds in live performance on a Raspberry Pi, laptop, iPad, or in a web browser. You don’t need to know these systems before you start: the background reading at the end of this page is the place to begin.
So far, we have mostly studied our intelligent instruments in short lab sessions or through our own performance practice. We don’t know much about what happens when musicians live with an intelligent instrument over weeks: collecting more data, retraining, and deciding whether the model is getting better or just different. This project takes on one part of that question. Two related projects on the lab website take on the others, and the three are designed to run alongside each other from December to June.
This project
IMPSY models are trained on tiny datasets: a few hours of one person’s playing. In our recent experiments, a model trained on about seven hours of controller data from several performances started to overfit after a few dozen epochs. Musicians collect more data every time they play, so what’s the best way to update a model? Retrain from scratch? Fine-tune the old model? How much new data does it take to change how the model behaves?
In this project you’ll treat existing performance logs as a sequence of practice sessions, then compare strategies for updating an MDRNN (mixture density recurrent neural network) after each one. A key part of the project is defining how to tell when a model has actually changed its musical behaviour, not just its validation loss (for example, by comparing the timing and values of gestures sampled from each model).
- Short project: a reproducible experiment pipeline using IMPSY’s dataset tools; retraining vs. fine-tuning on one instrument’s data; validation loss plus one behavioural measure.
- Full project: add regularisation strategies and a second dataset (e.g., logs from the related multi-week study project or other lab instruments), and recommend a procedure for updating models between sessions.
Background reading
- Martin, C. P. and Torresen, J. (2019). An Interactive Musical Prediction System with Mixture Density Recurrent Neural Networks. NIME 2019 (the original IMPSY paper).
- Martin, C. P. (2024). Generative AI for Musicians: Small-Data Prototyping to Design Intelligent Musical Instruments. GenAICHI workshop, CHI 2024.
- Fiebrink, R., Cook, P. R. and Trueman, D. (2011). Human Model Evaluation in Interactive Supervised Learning. CHI 2011 (evaluating models by how they behave, not just by accuracy).
More projects in the lab
This is one of more than a dozen projects currently available in the SMCClab, covering machine learning, embedded hardware, HCI, visualisation, theory, and artistic research. Only a selection is listed on the School of Computing website, so see the full list of SMCClab student projects on the lab website. You can also read this project on the lab website.
How to Apply
To apply for this project, contact Charles Martin.
Include:
- your CV
- your unofficial transcript (if you are an ANU student)
- a brief statement (200 words) in your email explaining how you would approach this project
Make sure to specify the skills and accomplishments you have that would help you to complete this project. Please read about joining the SMCClab and our project expectations before applying.