COMP4350/8350

Sound and Music Computing

Making Hardware Interfaces

Yichen Wang & Dr Charles Martin

Outline

  • High-level concepts and discussion about physical computer instruments
  • Two appoarches: instrumental and compositional (adapted from “Composing Interactions” - Marije Baalman)
  • The question of mapping
  • Micro:bit live demo making a simple movement-based MIDI interface

Recap: Incorporating Soft-/Hardware Interfaces in Pd

  • Buttons, sliders, selectors.
  • Keyboard: key, mouse, joystick, etc.
  • External library: Processing

More interaction options beyond your computer?

Hardware Interface Devices

Recap: A Simple Workflow

Where do I start ?

Instrumental Approach

  • Inspired by the acoustic instrument and well-established music tradition

  • Task-oriented interface design

  • Classifications (Miranda and Wanderlay, 2006):
    • Augmented musical instruments: extended by sensors and/or controllers
    • Instrument-like gestural controllers: model an acoustic instrument as closely as possible
    • Instrument-inspired gestural controllers: inspired by acoustic ones + new configuration
    • Alternate gestural controllers: not directly modeled or inspired (say - “imaginary”)

The extended clarinet (2016) / Carl Normark et al.

Digital sound layer

Preserve the interaction complexity and playing virtuosity

Extending the clarinet’s bell through the performer’s motion and gestures

Extra pitch bending and note playback options with visuals

Svampolin (2019) / Laurel S. Pardue et al.

A custom-designed electrodynamic pickup capturing the velocity of each string

Separating sound from source: sonic transformation of the violin through electrodynamic pickups and acoustic actuation, NIME2019

The Adapted Bass Guitar and The Strummi (2015 & 2018)/ Jacob Harrison

Accessible guitar instrument - touch screen guitar (?!)

When is a Guitar not a Guitar? Cultural Form, Input Modality and Expertise, NIME2018

Phaserings (2015) / Charles Martin

Percussion-inspired?

A granular synthesis component in Pd

Cubing Sound (2022) / Yichen Wang

Freehand gestures and embodiment

Imaginary?

… or I haven’t clearly figured out yet!

Other cool physical computer instruments

Compositional Approach

  • Focuses on the concept of a performance
  • You are the composer
  • Week 6: algorithmic composition and interactive music system
  • “From providing instructions for performers to create music, to creating contexts for performances in which music may be experienced” - Marije Baalman

Vital LMTD (2009) / Last Man to Die

Exploring three art forms: acting, percussion, and drawing through new interactive technologies and experimental performance

Video and more

Twilight (2013) / SLOrk

Inspired by the classic science-fiction short story “Twilight” by John W. Campbell

Video

The Question of Mapping

  • Connection between an action in the environment to sonic output

  • Steps in Mapping:

    • How do you want to control the sound & what is the interface?
      • E.g., Micro:bit, computer keyboard
    • What sonic / musical process is controlled by the interface?
    • What is the representation od sound and music that you are working with?
      • FM synth
      • String synth
      • Sequence-based melody (Gibber)

Steps in Mapping

  • Gesture/Input action
  • Sensor and electronic handware
  • Accessing data from hardware (sometimes over network and can be tricky!)
  • Computational model (E.g., algorithmic composition, synth mechanism, sampling)
  • Sound output

Live demo: Micro:bit + Pd making interactive musical system

Check out Captain Credible for more Micro:bit works/ideas!

Live demo: steps in mapping

  1. Gesture/Input action - hold micro:bit and move around
  2. Sensor - Micro:bit sensors
  3. Accessing MIDI data from Micro:bit hardware and to Pd over radio
  4. Computational model - Playing different pitches of an osc (envelope parameters, FM synthese, etc)
  5. Sound output - Pd

Micro:bit

Accessing data from Micro:bit and radio transmission

  • Get accelerometer data from Micro:bit
  • Sending data over radio communication
  • Receiving data at the laptop-end
  • Therefore, you need a sender & receiver

Receiver

// set communication channel
radio.setGroup(1)

// proxies all radio buffers to serial
radio.onReceivedBuffer(function (buffer) {
    serial.writeBuffer(buffer)
    // light on showing receiving in-progress
    led.toggle(4, 4)
})

Sender

radio.setGroup(1)
let accelx = 0

// set midi channel
let accelx_send = midi.channel(1)

basic.forever(function () {
  // access accelerometer data
    accelx = input.acceleration(Dimension.X)
  // some sound representation
  ...
  // note "bang"
  accelx_send.noteOn(accelx_note)

})

// send midi messages over radio transmission
midi.setTransport(function (data: Buffer) {
    radio.sendBuffer(data);

})

Radio Serial to Pd in your computer

MIDI and sound process in Pd

  • Read MIDI data
  • Transform it to other forms …

If you want to try Arduino

Thanks to Charles - he got you something to start with.