time for the finishing touches...

Major Project is due at 28 October 2024.

Still looking forward to seeing your discussions on the forum!

creative machine learning with ml5.js

What is machine learning?

How can I use ML in p5?

How can we make art with this?

What is Machine Learning Anyway?

Creating computer programs without explicitly programming them.

Algorithms that learn by example.

Algorithms that learn through experience.

Kind of a big deal ($$$)

Kind of problematic (!!!)

Let’s Solve a Problem

Suppose the boss wants a program where the screen colour changes to red when the mouse moves to certain locations.

(N.B.: the screen is 400 pixels wide)

How would you do it?

Let’s write a configurable algorithm!

if (mouseX > ??) {
    background(255,0,0);
} else {
    background(0,0,0);
}

one decision (red or black background)

one input (mouseX)

What if we had more inputs?

Maybe we could make more complicated decisions?

Likely to get more complicated to configure the algorithm.

Pictures as Inputs

Pictures are 2D arrays of colours! (represented as numbers)

So we had enough ifs and elses then maybe we could make a doggo classifier!

Simplifying inputs

One trick we often use is to design a configurable algorithm which can:

• take lots of numbers as inputs
• boil this all down to just one number as output.

The “configuration” would be “choosing how much of each input to listen to”

One example is a “perceptron” (1958)

Fast forward 50 years.

Trick 1: feed the outputs of perceptrons into more perceptrons in a kind of network

Trick 2: tricky algorithms to choose the configuration

Trick 3: big fast computers with lots of data to learn from

By the way, another name for a perceptron is an artificial neuron. So the above is a… neural network…

Some terminology

Model: an instance of a trainable algorithm

Pre-Trained Model: a trainable algorithm which has… already been trained.

Training/Optimising: using training data to make a trained model.

Prediction/Inference: using a trained model to generate an output using unseen data

Classification: an ML task for choosing a “class” (or description) for a piece of data

Wait, where's all the maths?

Want to make the learning process work faster? Great idea to have a maths/CS major.

Want to make interesting art with the most relevant new technology of today? Ok, you’re ready to go!

ml5.js: Friendly Machine Learning

ml5.js is a JavaScript library that provides access to machine learning models in a web browser.

You can load up pre-trained models and start doing prediction right away!

Related to and inspired by p5.js.

Get started

Just need to load it in our index.html:

<script src="https://unpkg.com/ml5@0.5.0/dist/ml5.min.js" type="text/javascript"></script>

Open up a p5 web editor sketch with ml5 already loaded.

Classifying Images

Let’s classify some doggos. We’ll use a pretrained model called MobileNet

// load the classifier
classifier = ml5.imageClassifier('MobileNet');
// classify an image
classifier.classify(img, gotResult)

Where does the result go? Need to define a callback function gotResult(error, results).

Classifying Video

We can access a webcam in our sketch:

video = createCapture(VIDEO);
video.size(320, 240);
video.hide();

And we can just ask the classifier to only make predictions from this video stream:

classifier = ml5.imageClassifier('MobileNet', video);
classifier.classify(gotResult); // classify one frame

What is problematic about doggos?

What is MobileNet?

Photo by Alina Grubnyak on Unsplash

ImageNet

What was the training data?

Are there hidden costs?

Artistinal Bespoke Machine Learning

Let’s make our own custom image classifier with Teachable Machine

SketchRNN

Sketch RNN is a model that generates a drawing by moving an imaginary pen around the screen.

Each “prediction” is a pen movement in x and y coordinates.

(This is a really fun model to try out).

Using SketchRNN

model = ml5.sketchRNN('cat'); // load the model with a "class" to draw.

The prediction is an object with x, y, and pen states (the pen can move up and down to make spaces in the drawing.

Here’s an example

More ml5.js models

There are lots of models available! (check the reference)

Many (e.g., posenet) are related to image classification and processing (images are so hot right now).

There are also sound recognition models.

And text generation models.

Machine Learning Art

further reading/watching

Artificial Intelligence Art

AI Art Tools

AI Art Paintings of Australia

Art and Interaction Bibliography

We made it!

Thanks for coming on this journey with me!

For more creative coding, see Sound and Music Computing

If you’re a CS student, think about a project with Charles Martin