The Ideas

After a short discussion we quickly came to the conclusion that we wanted to do a project the revolved around the idea of music and visuals. I came up with the idea that we should do some visuals that react to music by using some sort of beat recognition. Darren then came up with a few ideas of how we could link this with either projection mapping or some sort of augmented reality set up. Originally I came up with the idea of doing a small Peppers Ghost type projection. Darren also mentioned a program to use called VDMX.

[15/02/2016 09:03:22] Darren: I think you need to figure out what you’re going to do if we go down the projection music visuals route
[15/02/2016 09:03:31] Darren: cos VDMX has built in beat detection
[15/02/2016 09:04:09] Darren: and also a FFT based volume analyser thing that gives you number values for different frequencies
[15/02/2016 09:04:51] Darren: which lets you really easily animate things to react to bass or treble, or if you can pin point the eq you can pick out certain things like snares or whatever
[15/02/2016 09:05:32] Kit: ok cool, I haven’t looked at VDMX yet but I will do today, looks like a good route to go down
[15/02/2016 09:07:53] Darren: https://vimeo.com/116407490
[15/02/2016 09:08:47] Darren: that’s not a great video actually
[15/02/2016 09:09:21] Darren: https://vimeo.com/69400478
[15/02/2016 09:09:28] Kit: program looks really good though
[15/02/2016 09:09:41] Darren: so one video is audio reaction, second is beat detection
[15/02/2016 09:17:11] Darren: Do some googling of projection mapping projects and see what interesting stuff comes up
[15/02/2016 09:17:47] Kit: yeah i did have an idea which may be doable, seen a few people do it, making holograms with glass and projectors
[15/02/2016 09:18:12] Darren: Peppers ghost…

After a while of Darren being away we arranged to have some time to talk about the collaboration – It was during this latter discussion that we came up with the idea of what we are going to move forward with. Originally I wanted to use MAX to create the interface and some beat detection, but after trying a few different things it proved to be more difficult than originally thought, and have decided to look at using VDMX.

[21/03/2016 09:45:17] Kit: yo!
[21/03/2016 09:50:49] Darren: heya
[21/03/2016 09:51:02] Kit: hows it going?
[21/03/2016 09:59:33] Darren: alright, back in the frey today
[21/03/2016 10:01:04] Kit: still doing the remote work?
[21/03/2016 10:01:39] Darren: not so much
[21/03/2016 10:01:43] Darren: but back to all the other stuff
[21/03/2016 10:02:59] Kit: thats cool, will you be free to chat about the colab thing later?
[21/03/2016 10:03:20] Darren: Yes, let me get sorted and then lets chat this afternoon
[21/03/2016 10:03:43] Kit: yeah thats cool i need to go to a tutorial this morning and going to the gym so thats good for me

[23/03/2016 10:18:44] Darren: so have you looked at vdmx?
[23/03/2016 10:19:00] Kit: had a little look but not properly
[23/03/2016 10:19:12] Darren: it’s got brilliant beat detection
[23/03/2016 10:19:31] Kit: yeah, i looked at how they’re made and had a little go in max and its pretty solid
[23/03/2016 10:19:51] Darren: ok cool
[23/03/2016 10:20:02] Darren: can you show me anything about the beat detection in max?
[23/03/2016 10:20:05] Darren: any online vids?
[23/03/2016 10:20:55] Kit: couldn’t find any vids only some chat on the forums, there are a few external objects for it but couldn’t get them to work
[23/03/2016 10:22:30] Darren: yeh
[23/03/2016 10:23:12] Darren: vdmx actually incorporates this standalone beat tracker: http://wavesum.net/products.html
[23/03/2016 10:23:34] Kit: haha just as i said that i got the external object to work in max… sort of…
[23/03/2016 10:23:43] Darren: well, this will be better 😉

I had raised concerns that the project so far wouldn’t be that great for collaboration, as it could all be done inside VDMX. So Darren also mentioned another program he had heard of called waveclock, which is beat detection software. We then came up with the idea that we are now going to be moving forward with…

We are now going to be using Unity to create a 3D model of a dancer who will react to any music that a user will plug in and play. This will be on a screen with access to a 3.5mm jack for a user to plug in an audio device – mp3 player, phone, laptop. Darren is going to be working on creating the 3D model of a dancer and the environment while are start looking at how to use waveclock effectively as well as incorporating it with the 3.5mm jack and user interface.

[24/03/2016 14:31:09] Darren: I can help with the connection between waveclock and unity
[24/03/2016 14:31:15] Darren: if it’s unity you want to use
[24/03/2016 14:32:03] Kit: we could use unity to create some 3d model that can then be controlled by music?
[24/03/2016 14:32:33] Darren: yeh depends on the output, projection?
[24/03/2016 14:32:34] Darren: onto what?
[24/03/2016 14:33:22] Kit: we could try mocapping some dancers and then make some 3d characters dance in time to the music
[24/03/2016 14:33:37] Kit: that would be cool…
[24/03/2016 14:34:12] Kit: well thats kind of what I had in mind in the first place anyway, but thought it would be pretty hard
[24/03/2016 14:34:25] Darren: I can work on getting the dancing models
[24/03/2016 14:34:29] Darren: and making an environment for them
[24/03/2016 14:34:36] Darren: maybe just like a disco or something
[24/03/2016 14:34:55] Kit: ha yeah thats pretty cool
[24/03/2016 14:35:27] Darren: should think about output though, like is that just a screen that it’s showed on?
[24/03/2016 14:36:11] Kit: the only projector really that i can use here is the big one that i used the other day
[24/03/2016 14:36:22] Darren: ah in the big space
[24/03/2016 14:36:27] Kit: yeah
[24/03/2016 14:36:41] Darren: if you used the performance space we could play with lighting as well?
[24/03/2016 14:37:01] Kit: yeah they’ve got a pretty big lighting rig there i think
[24/03/2016 14:37:20] Darren: so we could make like a disco with blue green and red lights
[24/03/2016 14:37:27] Darren: and make the lights pulse in time with the music
[24/03/2016 14:37:36] Darren: and project some dancers onto the back wall
[24/03/2016 14:38:03] Kit: yeah thats cool
[24/03/2016 14:38:08] Darren: it’s all pretty full on stuff though thtat!
[24/03/2016 14:38:09] Darren: lol
[24/03/2016 14:38:52] Kit: yeah, probably best to scale down given the time frame
[24/03/2016 14:39:49] Kit: I could imagine it as a screen set up though with maybe a jack input so you could play music into it and have a character dancing on screen
[24/03/2016 14:40:14] Darren: yeah that could work
[24/03/2016 14:41:05] Darren: https://www.mixamo.com/store/#/search?page=1&type=Motion%252CMotionPack&query=dancing
[24/03/2016 14:41:52] Kit: HA thats cool
[24/03/2016 14:42:15] Darren: if you select ana nimation and then change ‘overdrive’ you can change the speed
[24/03/2016 14:42:48] Kit: so we can put these into unity?
[24/03/2016 14:42:53] Darren: yep
[24/03/2016 14:43:10] Kit: ok awesome!
[24/03/2016 14:43:49] Darren: so my thought is the character would just be idling on screen looking bored and then when an jack is plugged in it would transition into dancing in time with the music
[24/03/2016 14:44:11] Kit: yeah definitely!
[24/03/2016 14:44:30] Darren: if we have time we could look into mocapping some dancers but tbh we can just use some of these animations
[24/03/2016 14:44:57] Kit: yeah, thinking finding a dancer and the time to get some good mocap from it could take a while
[24/03/2016 14:45:06] Darren: yeh
[24/03/2016 14:45:11] Kit: looking thorough these some of them are really good
[24/03/2016 14:45:22] Darren: yeh
[24/03/2016 14:45:34] Darren: and I’ll make some sort of environment for them
[24/03/2016 14:45:38] Darren: probably with some nice lighting
[24/03/2016 14:46:03] Darren: I can make lighting in unity pulse with the beats or even audio reactive probably
[24/03/2016 14:46:13] Kit: ok so I will start looking at waveclock and the beat detection
[24/03/2016 14:46:23] Darren: yeh
[24/03/2016 14:48:32] Kit: would be cool to have this projection mapped somehwere
[24/03/2016 14:48:42] Kit: but thats a lot of work
[24/03/2016 14:48:52] Darren: I dunno I quite like the idea of a single screen, like a kiosk
[24/03/2016 14:49:10] Darren: some sort of kiosk with a 3.5mm jack popping out
[24/03/2016 14:49:37] Kit: yeah so you can just plug in and play
[24/03/2016 14:49:41] Kit: i like that
[24/03/2016 14:49:50] Darren: yeh
[24/03/2016 14:52:25] Kit: kinda of works as well i guess sort of linking to my other project with using dance
[24/03/2016 14:52:29] Darren: yeh
[24/03/2016 14:53:39] Kit: i like it, i like it a lot
[24/03/2016 14:53:43] Darren: cool
[24/03/2016 14:54:30] Darren: you start using waveclock or whatever its called and look at how it functions specifically
[24/03/2016 14:54:48] Darren: like, what does it do when there’s no music, and then how to it start up when music starts playing
[24/03/2016 14:54:52] Kit: ha was literally just writing that
[24/03/2016 14:55:02] Darren: cos I’ll need to work with whatever it spits out to control the idle/dancing state of the character
[24/03/2016 14:55:49] Darren: I can also potentially use general audio levels to trigger the dancing but will need the clock coming through as well
[24/03/2016 14:56:03] Kit: yeah ok cool, heading back to mums tonight (taking desktop pc and everything) so will either start looking tonight or tomorrow morning
[24/03/2016 14:56:10] Darren: cool
[24/03/2016 14:56:39] Kit: yeah thats cool, guessing you can control some other perimeters of the dancer using more than just the clock?
[24/03/2016 14:56:55] Darren: dunno
[24/03/2016 14:56:58] Darren: I wouldn’t like to commit to it
[24/03/2016 14:57:02] Kit: or maybe the lighting
[24/03/2016 14:57:23] Darren: I can make audio reactive lighting mixed with beat mapped lighting
[24/03/2016 14:57:26] Darren: that should look pretty good
[24/03/2016 14:57:40] Darren: and I can play with the camera too
[24/03/2016 14:57:49] Kit: yeah that sounds cool
[24/03/2016 14:58:06] Kit: like give it shakes or something on audio levels or control it’s pace if it’s panning around based on the beat map
[24/03/2016 14:58:55] Kit: i will start looking at ideas for a kiosk or something too
[24/03/2016 14:59:00] Darren: ok cool

Introduction

This blog is serving the purpose to document all information, research and communication during my Collaboration project.

For my collaboration project I have chosen to work with my brother, Darren Perry, of The Media Workshop. We have not started with a goal set, we have decided to pull our interests together and decided on a project that will utilize both of our talents with technology. Mine based around music and Darren’s based around visuals. Here is a timeline that I am proposing for the project –

February 1st – March 1st – Finding Partnership

March 1st – March 7th – Creating Ideas

March 7th – March 21st – Darren Away/Finalise Ideas

March 21st – March 28th – Decide on Visuals

March 28th – April 11 – Work on Programming

April 11th – April 18th – Meet-up/Catch-Up

April 18th – May 2nd – Work on Programming

May 2nd – May 9th – Meet-Up/Catch-Up

May 9th – May 19th – Prepare for Hand In

I will be posting any important decisions that we make for the project as well as any research for the project as well as research for working in a collaboration. Although I have done a vast amount of collaboration work before I have not done any collaboration in this area of technology so it will be a good learning experience for me.

Part of my reasoning for choosing to work with Darren, and an initial idea for the project was based on a project that was done by Darren and The Media Workshop and shown at the Durham Light Festival.

 

In the video you can see that there is large projection mapping on to the side of a building with visuals that are representing the audio being played. I really liked the idea of how the visuals reacted to the music and also the scale of the project itself, being on a large scale building. Another Idea I have had aside from projection mapping was holograms, initially with the type of holograms that you can make at home as shown below, working with 3D visuals is also something that greatly interests me.