Lab 9: Restyled Video

Outline

In this lab you will:

create an animation in p5.js
take a screen recording of your animation
apply style transfer to the screen recording of your animation
look at training a style transfer model
start your blog post

Part 1: Style Transfer for Video

In this section we will be adding some new inputs. Instead of working with static images, we are using the dynamic medium of video! The structure is very similar to the style transfer lab from last week but we are using webcam or video as an input. The starter-code for this week already applies style transfer to your webcam feed. All you need to do is uncomment the style.transfer() method inside the gotResult() function. To see how this works, go ahead and fork and clone the repo for this week. Your instructor will talk you through how this code works in class.

Fork and clone the template repo for this week’s lab.

Apply style transfer to any video

Define a video variable
Use the createVideo() function to load a video into p5.js. You can refer to the p5.js references to help you. You’ll need to call video.loop() after loading the video in order to play the video on your canvas. There is a video called fish.mov which you can use. It’s already in your images/ subfolder. You can also use any video you’d like as long as it’s in .mov format.
Display the original video and the style-transferred version. You will need to uncomment the style.transfer() method inside the gotResult() function.

Cool! Hopefully you have your webcam and/or video working as input with style transfer. Can you think of any other inputs you could use? Using webcam makes the style transfer a more interactive experience, what other interactive elements could you incorporate with style transfer? How does using a video as input with style transfer speak to the concept of style transfer as ‘remixing’?

Combinations and permutations

We have already investigated and explored the capabilities of shaders.

Do you think it is possible to apply a shader and a style transfer?

Does the order of application matter?

Can you apply multiple style transfers?

How many layers of transformation is it possible to apply without compromising the speed of the rendered video?

Part 2: Training for Style Transfer

Pre-Trained Style Transfer Models

It is perfectly fine to use the pre-trained models which are provided with this lab - you DO NOT HAVE TO build your own models. You may use other models you have found for ml5.js style transfer.

Training your own model

BEWARE - this is very, very complicated in terms of both machine requirements, and code dependencies. You may run into problems which are not easily resolvable. DO NOT SPEND TOO MUCH TIME ON THIS.

This Google Colab Notebook has been built to train a new style transfer model. The last step, converting the model for use in ml5.js, will need to be completed on your local machine due to unsupported dependencies. Follow the demonstration in the lab for more details.

Part 3: Blog Provocation

This week, your task will be to implement a sketch which addresses the following provocation:

DO: Use the pre-trained style model(s) from last week’s lab to create an augmented version of a p5.js animation. These models are already available in your template repo for this week.

Part 3.1: Create an animation

To address this provocation, your first task will be to create an animated sketch in p5.js. You’ll be using much of the same skills we learned during the visuals part of this course from last year. Your sketch will have to evolve over time in some way – this is the animation part.

You can either stick to the standard p5.js shapes like curves(), rect(), quad() or you can manipulate the pixels in an image.

Once you are happy with your animation, commit and push your code to git.

Part 3.2: Take a screen recording

The next step is to take a screen recording of your animation.

If you’re working on Windows or Mac, you’ll already have a built-in screen recorder you can use. For Mac: use the keys CMD+Shift+5 to open the screen capture menu and record only the portion of your browser window where your animation appears. For Windows 10: Use Windows Key + ALT + R to open the XBox Game Bar for Screen Recording. For Windows 11, this feature has been added to the Snipping Tool: Windows Key + Shift + S. If you need help with this step, ask you one of your instructors.

With Mac machines, the screen recording you take will automatically be in the .mov format, which is great because sometimes the style transfer model doesn’t play nice with .mp4.

If you’re working on Linux machine, or want more control of your screen recording, it’s helpful to install one during this class since we’ll be using screen recording software for the final project of this course too.

Open Broadcaster Software (OBS) is a great option for most Windows and Linux, and even Mac, machines. You can follow the install instructions through this link: install OBS. If you need help with this, ask one of your instructors.

Once you’ve installed OBS on your machine, you can use these instructions to take a screen recording of your animation. Make sure you export your screen recording in the .mov format.

Part 3.3: Apply style transfer

In the same way that we applied style transfer to the fish.mov video in Part 1, we’re going to load the screen recording of our animation into this week’s template repo and apply style transfer to the screen recording.

Drag and drop the screen recording into the images/ subfolder of your template repo and follow the instructions from Part 1 of this lab to apply style transfer to your animation.

Which aspects of your animation are effectively interpreted by the style transfer model? Are there different scales, shapes, colours which have stronger impact on the resultant video after applying style transfer?

Remember that the sketch(es) you make during this activity will be used for your blog post, so you will need to create 2-3 version of this sketch to discuss in your blog post. Take a screen recording of your animation after applying style transfer to submit in your blog post. Since we’d like you to submit 2-3 variations, you’ll need to make 2-3 modifications to you animation and apply style transfer(s) to each version.

Remember to commit your code and push it up to Gitlab, and log out of the lab computer.