Learning how to create a music visualizer using p5.js
Introduction:
I aimed to create a music visualizer using P5.js in this project, which is made up of two phases, Phase 1 which consisted of developing a visualizer from scratch and Phase 2 which comprised of developing the idea into something more tangible and connected to real life.
Overview of Phase 1:
This phase involved creating simple shapes that could be connected to a microphone input and based on the frequency levels, the shape’s forms could be altered.
Phase 2
Vision:
To develop the idea further into a more interactive and tangible experience. Since music visualizers have always been so abstract and digital, I wanted to find a way to connect it to reality.
Idea:
To map shapes to our faces and have them react based on excitement levels, in order to create an interactive experience between the user and the visual.
Development Process:
I started out by using posenet to track my face and attached a rectangular shape to it as you can see below, just testing the waters, exploring posenet.
Once I got a bit comfortable with the code, I was able to connect the microphone input to alter 3D shapes based on frequency levels and I increased the amount of shapes present to cover the entirety of my face.
I then started to look for libraries and API’s that would allow me to map the features of my face and track the movement in a more fluid manner, which is when I stumbled upon a code that allowed for the use of ML5 to analyze and identify your facial features while also working perfectly in P5.js.
I started playing around with it and found this algorithm to work flawlessly on my website while creating this live digital version of me. To make it a bit more expressive I started tweaking it and below are my iterations.
You can try this out yourself on my website here: https://arnavbavishi.github.io/Arnav-Repository/9phase2.html
Final Visualizer:
Embed on Website — Tutorial:
<!DOCTYPE html><html>
<head>
<meta charset="UTF-8" /><title>Phase 2 testing</title><link href="https://fonts.googleapis.com/css2?family=Kumbh+Sans:wght@400&display=swap" rel="stylesheet">
<link rel="stylesheet" href="mystyle.css"><script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script><script src="https://unpkg.com/ml5@latest/dist/ml5.min.js" type="text/javascript"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.1.9/addons/p5.sound.js" integrity="sha512-KxzVm+IqxNNq0+SzT/zzd5PHxY4LPrN+v5gZJ6+JKqjeU3Cr4y/djAg5eNlKDWurn1SeKZpql/yeOMWblMSzOg==" crossorigin="anonymous"></script>
</head><script src="sketch.js"></script><body><h1> Phase 2 Testing </h1>
</body>
</html>
Now, lets break it down.
In order to make this load on your website following are the libraries that you need to import into your html code.
Libraries Used:
- P5 Dom.js
2. P5 Min.js
3. ML5.js
4. P5 Sound.js
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/p5.min.js"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.9.0/addons/p5.dom.min.js"></script><script src="https://unpkg.com/ml5@latest/dist/ml5.min.js" type="text/javascript"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/1.1.9/addons/p5.sound.js" integrity="sha512-KxzVm+IqxNNq0+SzT/zzd5PHxY4LPrN+v5gZJ6+JKqjeU3Cr4y/djAg5eNlKDWurn1SeKZpql/yeOMWblMSzOg==" crossorigin="anonymous"></script>
You will need to create a ‘.js’ file where the code to the facemesh will be written. That ‘.js’ file is then linked to an html file on your website. Through the follwing code:
<script src="sketch.js"></script>Now to the facemesh code:
Enter the facemesh code into a new file and save it as a ‘.js’ file. Make sure the name of the file matches in your html file.
let facemesh;
let video;
let predictions = [];function setup() {
createCanvas(640, 480);
video = createCapture(VIDEO);
video.size(width, height);facemesh = ml5.facemesh(video, modelReady);// This sets up an event that fills the global variable "predictions"
// with an array every time new predictions are made
facemesh.on("predict", results => {
predictions = results;
});
background (0);
// Hide the video element, and just show the canvas
video.hide();
}function modelReady() {
console.log("Model ready!");
}function draw() {
background ("#00000030");
// image(video, 0, 0, width, height);
// background("#00000030")
// We can call both functions to draw all keypoints
drawKeypoints();
}// A function to draw ellipses over the detected keypoints
function drawKeypoints() {
for (let i = 0; i < predictions.length; i += 1) {
const keypoints = predictions[i].scaledMesh;// Draw facial keypoints.
for (let j = 0; j < keypoints.length; j += 1) {
const [x, y] = keypoints[j];fill(255, 0, 0);
noStroke();
ellipse(x, y, 5, 5);
}
}
}
This code uses a machine learning algorithm that analyzes images and distinguishes certain facial features as coordinates. Shapes are then assigned to these coordinates.
Future Direction:
For my future direction of this project, something interesting I would like to do, is to develop a virtual disco party where people can use this algorithm as a personalized Face-mesh that would represent them in these parties, that way they would not be revealing their complete identity but could still enjoy the company of others and enjoy music all-together. This would be a great way to bring people together from across the world and could help people find others who have the same music taste as them.
Conclusion
This entire process was very eye opening for me while also being quite fun and educational at the same time since I was able to combine something I absolutely love — Music, with something that can get very boring to me — coding. I think this combination has helped me enjoy code and has definitely proven to me that learning coding can open up an insane amount of creative opportunities.
