Outline

In this lab you will:

  1. load a pre-trained ml model into p5 and use it to modify images
  2. experiment with re-applying style transfer models to images

Introduction

You might have seen some of the cool “AI Art” that’s being done these days:

While there are lots of different AI and artistic techniques which might get the “AI Art” that label, one of the more famous techniques is called “style transfer” and uses an AI technique called a Generative Adversarial Network (or GAN).

Today you’re going to be using these techniques to apply transformations to images and see what happens.

Part 0: Crit Session

For the first 20 mins of this workshop we are going to discuss your blog posts in what we call a “crit”. Crits are a common practice in arts schools; they are an opportunity to share your work with your peers and discuss new ideas and perspectives. This activity (and this classroom) is a safe space for you to share ideas and learn from others. The purpose of this activity is to help you become comfortable talking about your work and receiving feedback.

We will be doing this activity in small groups of 3-4 people. For this week, you can work with someone next to you, but we will be shuffling you around for this activity in the coming weeks so you can see more of your classmates. Form a group of 3-4 people for your crit as directed.

Once you have formed your small groups we will start a crit session. In the crit session, you will each take turns talking about your own work. Here are some points you can draw on for the discussion:

  • What was your interpretation of “Extra Dimensions”?
  • Which alternative offered did you choose and why?
  • What did you do to generate variation in your sketch?
  • What did you find challenging about the process of developing your sketch (including its variations)?
  • What’s your favourite thing about your sketch(es) and why?

After each person in your group has discussed their own work, each of the other members in that group will give one piece of positive feedback. When giving feedback, we encourage you to stay away from generic statements like “I really like this aspect..” or “This was interesting…”. Instead, try to think about how the piece of work makes you feel or what it makes you think of. Some examples are “I found this aspect of your work visually powerful because…”, or “the motion you generated here was whimsical. It reminded me of…”.

Your instructor will also give you some feedback on your work, along with some things you might like to try to push your work further.

It’s not always easy talking about your work – we hear you – but everyone brings their own unique perspective to the table in this class. One of those perspectives is bound to inspire you!

During this activity, it can be helpful to keep some distance between your work and yourself. The feedback we give you is not a reflection of you are what you are capable of in this class, it is a reflection of one single piece of work you made. You should use it as an opportunity to get ideas and feedback.

In your groups of 3-4, take turns sharing your work and the process you took to develop it. Give one piece of positive feedback to each member in your group.

Part 1: drawing an image on the canvas

First, a quick refresher on using images with p5. We (humans) are visual creatures. We made rock art for a long time, and then we invented the internet and now we have instagram1. Thankfully, p5 has got your back. All the p5 functionality to do with images is bundled together inside the p5.Image class object (see the reference for the details).

Fork and clone the template repo for this week’s lab. Your task in part 1 is to load an image into p5 (you can store it in the contentImg variable) and draw it somewhere on the canvas.

In this week’s lab template you’ll see there’s a small amount of stuff in there related to images, but you’ll need to fill in the blanks. Remember to look through the p5 reference and examples if you get stuck. You can either pick one of the provided images (have a look in the img/ folder), or you can provide your own.

Once you’ve done that we can move on to step 2.

Part 2: style transfer

Ok, here’s the AI art part of the lab. For the next two weeks, we’re going to be looking at ml5js’s style transfer functionality.

The basic idea is to take the style from one image (often called the “style image”) and transfer the style to the content of another image (the “content image”).

This is a two part process. First, you need to train your AI model on a particular style and then we can apply this style to other images. We will be using pre-trained models (so the first part is already done) that have been trained on The Great Wave off Kanagawa (1829) by Katsushika Hokusai and Udnie (Young American Girl), (1913) by Francis Picabia.

That leaves the second part—which is your job in Part 2—to apply the style to a new image. In fact, we’re going to apply it to the whole canvas.

Let’s start by loading up the style transfer model (we’ll use the Great Wave one for now). The We follow the same process as we did last week to load a model into p5. We first specify that we want to use the styleTransfer() function which takes two arguments (just like the imageClassifier() function we used last week). The first argument specifies the path to the model (which is actually just a bunch of files on disk, as we’ll explore more next week) and the second specifies the callback function which will be called when the model has loaded.

This is already in your template in the preload() function.

style = ml5.styleTransfer("models/wave", modelLoaded);

To use the model (i.e. to apply the style to a new content image) you can call the .transfer() function like so:

function transferComplete(err, result) {
  resultImg = p5ImageFromImgElement(result);
}

function keyPressed() {
  if (key === " ") {
    // note that it's the whole canvas that's the "content image"
    style.transfer(canvas, transferComplete);
    background("red");
  }
}

The transferComplete() and keyPressed() callback functions provided are there to start you off, and when the “style transfer” has occurred you’ll have a new p5 image assigned to the resultImg variable (can you see where that happens above?). Your final step, which you can do by yourself (we believe in you!) is to draw the new, processed resultImg on the canvas somewhere.

One thing to be careful of: at the start of your sketch resultImg will be undefined, so if you just put image(resultImg, 0, 0) in your draw loop you’ll get an error. What you need to do is have some sort of if test in your draw loop to check that resultImg is actually an image (i.e. has transferComplete() done its job successfully), and only then actually draw the resultImg on the canvas.

Once that’s done, you should see your “newly styled” version of the starting image. Try experimenting with different source images, (and even the other model which is at "models/udnie") to see what results you get.

If you get stuck, remember that there are a few different moving parts here:

  1. you need to draw your content image to the canvas
  2. you need to apply the style transfer model with style.transfer()
  3. you need to do something in the transferComplete callback so that you actually keep track of the processed image
  4. you need to draw the processed image somewhere so you can see the results of your AI art manipulations

Part 3: re-applying transfer models

This kind of model is called a Generative Adversarial Network (GAN)—a type of deep neural network. GANs are generative which means that they learn from an image data set (like the classifier from last week) and then can generate images. They are adversarial which means they are built around two neural networks—a generator G and a discriminator D. These two networks play a ‘game’ between each other. G learns to generate images and D learns to tell fake images apart from real images.

Training a model takes a considerable amount of time and computing resources. On your laptops that process may take up to several months! But we can create a range of interesting effects by applying and re-applying different models to our input image.

In Part 3, we’ll take the (couple of) pre-trained models we have and use them in different ways to give a range of interesting results.

This is the open ended part of the lab. Some things to think about:

  • what happens if you apply the style transfer to a bunch of p5 shapes you drew on the canvas, rather than an image?
  • what happens if you apply multiple conflicting styles on top of each other?
  • what happens if you re-apply the same style multiple times?
  • how many times do you need to re-apply a style before the starting image is no longer recognisable?
  • does this change based on your starting image?
  • if you apply one style to an image, how many times must you apply a different style to that image before the first style disappears?

Creating a plan

One of the skills you’ll develop in this course—especially the Year 12 part—is taking a creative computing technique, coming up with ideas for how you could use that technique in an interesting way, and then writing the code to bring your idea to life. That’s what these open ended parts of the lab are all about.

As a final step in today’s lab, open up the plans.md file in your editor and write out your plans for what sort of creative code things you’d like to make with the style transfer stuff you’ve used today. It doesn’t have to be super long, one paragraph is fine. And while you

Remember to commit your code (including your plans.md file) and push it up to Gitlab, and log out of the lab computer.

Summary

Congratulations! In this lab you:

  • used a pre-trained style transfer model to modify input images

Next week we might even get to the point of training our own style models following the linked ml5js tutorial. It has several different instructions of how to train our own models, but access to a good GPU is necessary for the computation time to be several hours rather than several months. We (EXTN1019B) have access to a machine with nice GPUs, so if you’ve got any favourite images you think might work well as “style” images then let us know in class.

Think about artists which might be worth looking into (hopefully not all dead white men).

  1. Hey, I think a few things happened in the middle as well, but hey—this isn’t an art history course. You’ll have to go somewhere else to learn that stuff :) ↩

bars search caret-down plus minus arrow-right times arrow-up creative-commons creative-commons-by creative-commons-nc creative-commons-sa