Week 6: sound and music computing

Ushini Attanayake

Semester 2, 2023

admin (week 6)

assignment 2 due this coming Monday 9pm!

now covered the coding topics.

have some more to learn about how to use p5

…and practice skills in art and design!

this week: sound and music computing!

admin (week 7)

assignment 2 marks to be returned later today.

next challenge: assignment 3!

Few admin arrangements:

week 9 Monday lab replacement or sth…

sound

why should you, the code artist, care about sound?

sound demands attention

sound communicates emotion

enhances realism & immersion

provides feedback during interaction

types of sound

musical

abstract

skeuomorphic/sfx/diagetic

sonic interaction design: key questions

  1. what sound to make?

  2. when to start & stop?

  3. how high (or low)?

  4. how loud?

sound lingo

pitch: high or low (measured in Hz) ranging from 40Hz (low bass rumble) to 20kHz (high-pitched buzz)

amplitude: AKA volume or loudness, usually ranging from 0.0 (silence) to 1.0 (full volume)

duration: long or short, measured in seconds (it’s just time!)

timbre: often called the “colour” or “quality” of a sound (e.g. piano & saxophone can play the same pitch, but have different timbre)

what/when/how high/how loud

timbre (i.e. what)

in computer music (and in p5.sound), there are two ways to make a sound:

  • synthesis (mathematically calculating the signal)

  • sampling (playback of recorded sound)

Synthesis

Image: Wikipedia

Sampling

Image: Tascam

control (i.e. when)

Lots of options:

direct control through an input device (e.g. keyboard)

time-based

scene-based (e.g. connected to events in the sketch)

random

pitch (i.e. how high/low)

different pitches give us music

pitch can also communicate other things e.g. size, mood

amplitude (i.e. how loud)

slow changes in loudness—build tension, think verse vs chorus

quicker changes in loudness—accents in music

difference between loudest & softest sound is called the dynamic range—make use of it!

interaction

connect sound input to action on the screen

connect keyboard/mouse to sounds

connect camera/accelerometer to sounds

can we think about this as a new musical instrument?

let's make some sounds!

synthesis in p5!

sampling in p5!

interactive music in p5!

sound is different to graphics

when we say frame rate, what are we talking about?

what’s the p5 draw() loop frame rate? how would we find out?

what’s the normal sound frame rate?

44100Hz–96000Hz

44100 vs 60: that’s a huge difference (735x!)

making sound in p5

p5 has a sound library: it’s called p5.sound

note: there’s no draw() loop for sound to directly program frames of sound (it would be too slow)

instead, p5.sound provides a bunch of sound “building blocks” (which are just objects!) which you need to create and modify.

You need to make sure the p5.sound library is loaded in your index.html

building blocks

sources produce sound: e.g., playing a sound file, “synthesising” a sine wave

processors process sounds: e.g., bass boost, reverb

sources: synthesis

p5.sound provides a p5.Oscillator which creates a basic sound signal—here’s a simple example sketch

let osc;

function setup() {
  osc = new p5.Oscillator();
  osc.setType("sine"); // or "triangle", "sawtooth", "square"
  osc.start();
}

function draw() {
  osc.freq(1000*mouseY/height);
  osc.amp(mouseX/width);
}

basic synthesis parameters

// assign p5.Oscillator object to variable
osc = new p5.Oscillator();

remember the key sonic interaction design questions:

  1. what sound to make? use the .setType() method
  2. when to start & stop? start with .start() method
  3. how high/low? use the .freq() method
  4. how loud? use the .amp() method

look at the reference for the full list of properties/methods

sources: sampling

loadSound() returns a p5.SoundFile object useful for playing pre-recorded sounds (e.g. mp3s, wav files, etc.)

let unilodgeSound;
function preload() {
  unilodgeSound = loadSound("assets/unilodge.mp3");
}

function setup() {
  unilodge.setVolume(0.5);
  unilodge.play();
}

basic sampling parameters

// assign p5.SoundFile object to variable
unilodgeSound = loadSound("assets/unilodge.mp3");
  1. what sound to make? depends on the sound file
  2. when to start & stop? start with the .play() method, stop when file ends (maybe)
  3. how high/low? use the .rate() method
  4. how loud? use the .setVolume() method

note the subtle differences: the p5.SoundFile object (sampling) has different methods to the p5.Oscillator object (synthesis)

the preload() function

we’ve talked about setup() and draw()

but in some of the examples (again, especially the sound/image ones) there’s a preload() function…

let’s check the reference

necessary because of the way the web browser works

talk

where should unilodgeSound.play() go?

function preload(){
  // here?
}

function setup(){
  // here?
}

function draw(){
  // here?
}

function mousePressed(){
  // here?
}

the rest of the p5.sound library

the reference has the full list, but some highlights:

so now I know all about sound?

nope, sorry! we could have a whole course on computer music, but you know how to make some basic sounds in p5

let’s dig a bit deeper and look at making some interactive music.

Let’s look at a bit of synth theory…

Amplitude Envelope

An enveloped sound

  • Amplitude is the “volume” of our note.
  • Envelope is the chunk of time for our note to exist in.
  • We can change the amplitude over the envelope to give a note a sonic “shape”.
  • In synth lingo, an EG (envelope generator) makes envelopes.

ADSR Envelope

The ADSR Envelope

  • The adsr shape is often used for pitched sounds.
  • ADSR: attack, decay, sustain, release

Additive Synthesis

Adding two sine waves together

  • Take multiple oscillators and add them together!
  • Need lots of oscillators to make complex sound.

Subtractive Synthesis

Applying a low pass filter to a square wave

FM synthesis

An frequency modulated sound

  • “frequency modulation”
  • Use one oscillator to control the frequency of another.
  • Cool sounds with few oscillators (see Yamaha DX7)

Let's make some interactive music

We need to connect some events in time to an interactive input…

art theory: sound art

can sound be used as a material in art?

wait… isn’t that music?

how can we do this in p5?

synthesisers

Éliane Radigue (b. 1932)

Éliane Radigue in her studio, Paris

c. 1970s

📷 Yves Arman

listen: Trilogie de la Mort

tape

Musique Concrete Research Group (GRM)

François Bayle, Pierre Schaeffer and Bernard Parmegiani at GRM

1972

listen: Études de bruits (1948)

sounds of silence

John Cage (1912-1992)

4’33”

1952

Piano score, three movements

John Cage (1912-1992)

4’33’

1973

watch: Performance in Harvard Square, Cambridge, Massachusetts

sounds around us

John Cage (1912-1992)

Water Walk

1959

Various objects, microphone

watch: performed on US Television show I’ve got a secret (1960)

silent spaces

John Cage (1912-1992)

John Cage in the anechoic chamber

1951

anechoic chamber

sound of art

Michael Asher (b. 1953)

A view of Michael Asher’s installation at Pomona College, looking out toward the street

1970

Pomona College of Art

La Mont Young & Marian Zazeela: _Dream House_ (1969-)

Sound is spatial…

Working with Sound Files

Let’s make some musique concrète.

Take sound files as a raw material

and create an interactive music work.

Let’s look at p5.SoundFile

// assign p5.SoundFile object to variable
unilodgeSound = loadSound("assets/unilodge.mp3");

Playback control:

  • .pause() stop playing (and resume later)
  • .resume() un-pause
  • .stop() stop playing.

Composing with SoundFiles

Use the parameters for play to create “notes”:

.play([startTime], [rate], [amp], [cueStart], [duration])

Change parameters:

  • .rate() speed, pitch
  • .jump() change playback position
  • .setLoop() start/stop looping

userStartAudio()

Browsers don’t like tabs to “autoplay” music. If you’re only using sound files, and .play()ing them in setup, some browsers won’t make any sound at all.

p5.sound has a built-in trick to help, a function that starts audio that you can use after an interaction (e.g., clicking in the window):

function mousePressed() {
  userStartAudio();
}

SoundFiles and Envelopes

We can make “notes” out of soundfiles as well…

// in setup...
env = new p5.Envelope();
synthSample.amp(0);
synthSample.setLoop(true);
synthSample.play();
env.setInput(synthSample);
// somewhere else...
env.play();

Recording audio with p5.AudioIn

You can use p5.AudioIn and p5.SoundRecorder to access a microphone.

\\ setup
mic = new p5.AudioIn();
mic.start();
recorder = new p5.SoundRecorder();
recorder.setInput(mic);
soundFile = new p5.SoundFile();
\\ somewhere else...
recorder.record(soundFile);
recorder.stop();

Be very careful with this technique: feedback and unpredictable input are very annoying!

manage the sonic experience for your user

record, generate, project

Janet Cardiff's Audio Walks

Runar Magnusson and Thor Magnusson

SameSameButDifferent v0.2

Custom software

(link)

Brian Eno

Ambient 1: Music for Airports

1978

link

Strike on Stage

Chi-Hsia Lai and Charles Martin

2010

Computer video, graphics, vision, audio

(link), (paper), (performance)

Hélène Vogelsinger

Hélène Vogelsinger

2021

Modular synthesisers

patch notes video

Remember

  • Sound creates atmosphere
  • Sound is spatial
  • Sound drives interaction
  • Sound enhances experience

p5.js can do (almost) everything you could want with sound

To do more look at tone.js or gibber

Dos and Don’ts

  • Do use sound!
  • Don’t just play back mp3 files from the internet.
  • Do think about enhancing the experience.
  • Don’t annoy the user.

further reading/watching

SYNTHESIZE ME: Sasha Frere-Jones on Éliane Radigue

Janet Cardiff and George Bures Miller, Night Walk for Edinburgh (2019)

Pierre Schaeffer: “etude aux chemins de fer”

Physicist of Sound | Carsten Nicolai

Freesound.org: CC-licensed sound files and field recordings

questions?