The Ideas

After a short discussion we quickly came to the conclusion that we wanted to do a project the revolved around the idea of music and visuals. I came up with the idea that we should do some visuals that react to music by using some sort of beat recognition. Darren then came up with a few ideas of how we could link this with either projection mapping or some sort of augmented reality set up. Originally I came up with the idea of doing a small Peppers Ghost type projection. Darren also mentioned a program to use called VDMX.

[15/02/2016 09:03:22] Darren: I think you need to figure out what you’re going to do if we go down the projection music visuals route
[15/02/2016 09:03:31] Darren: cos VDMX has built in beat detection
[15/02/2016 09:04:09] Darren: and also a FFT based volume analyser thing that gives you number values for different frequencies
[15/02/2016 09:04:51] Darren: which lets you really easily animate things to react to bass or treble, or if you can pin point the eq you can pick out certain things like snares or whatever
[15/02/2016 09:05:32] Kit: ok cool, I haven’t looked at VDMX yet but I will do today, looks like a good route to go down
[15/02/2016 09:07:53] Darren: https://vimeo.com/116407490
[15/02/2016 09:08:47] Darren: that’s not a great video actually
[15/02/2016 09:09:21] Darren: https://vimeo.com/69400478
[15/02/2016 09:09:28] Kit: program looks really good though
[15/02/2016 09:09:41] Darren: so one video is audio reaction, second is beat detection
[15/02/2016 09:17:11] Darren: Do some googling of projection mapping projects and see what interesting stuff comes up
[15/02/2016 09:17:47] Kit: yeah i did have an idea which may be doable, seen a few people do it, making holograms with glass and projectors
[15/02/2016 09:18:12] Darren: Peppers ghost…

After a while of Darren being away we arranged to have some time to talk about the collaboration – It was during this latter discussion that we came up with the idea of what we are going to move forward with. Originally I wanted to use MAX to create the interface and some beat detection, but after trying a few different things it proved to be more difficult than originally thought, and have decided to look at using VDMX.

[21/03/2016 09:45:17] Kit: yo!
[21/03/2016 09:50:49] Darren: heya
[21/03/2016 09:51:02] Kit: hows it going?
[21/03/2016 09:59:33] Darren: alright, back in the frey today
[21/03/2016 10:01:04] Kit: still doing the remote work?
[21/03/2016 10:01:39] Darren: not so much
[21/03/2016 10:01:43] Darren: but back to all the other stuff
[21/03/2016 10:02:59] Kit: thats cool, will you be free to chat about the colab thing later?
[21/03/2016 10:03:20] Darren: Yes, let me get sorted and then lets chat this afternoon
[21/03/2016 10:03:43] Kit: yeah thats cool i need to go to a tutorial this morning and going to the gym so thats good for me

[23/03/2016 10:18:44] Darren: so have you looked at vdmx?
[23/03/2016 10:19:00] Kit: had a little look but not properly
[23/03/2016 10:19:12] Darren: it’s got brilliant beat detection
[23/03/2016 10:19:31] Kit: yeah, i looked at how they’re made and had a little go in max and its pretty solid
[23/03/2016 10:19:51] Darren: ok cool
[23/03/2016 10:20:02] Darren: can you show me anything about the beat detection in max?
[23/03/2016 10:20:05] Darren: any online vids?
[23/03/2016 10:20:55] Kit: couldn’t find any vids only some chat on the forums, there are a few external objects for it but couldn’t get them to work
[23/03/2016 10:22:30] Darren: yeh
[23/03/2016 10:23:12] Darren: vdmx actually incorporates this standalone beat tracker: http://wavesum.net/products.html
[23/03/2016 10:23:34] Kit: haha just as i said that i got the external object to work in max… sort of…
[23/03/2016 10:23:43] Darren: well, this will be better 😉

I had raised concerns that the project so far wouldn’t be that great for collaboration, as it could all be done inside VDMX. So Darren also mentioned another program he had heard of called waveclock, which is beat detection software. We then came up with the idea that we are now going to be moving forward with…

We are now going to be using Unity to create a 3D model of a dancer who will react to any music that a user will plug in and play. This will be on a screen with access to a 3.5mm jack for a user to plug in an audio device – mp3 player, phone, laptop. Darren is going to be working on creating the 3D model of a dancer and the environment while are start looking at how to use waveclock effectively as well as incorporating it with the 3.5mm jack and user interface.

[24/03/2016 14:31:09] Darren: I can help with the connection between waveclock and unity
[24/03/2016 14:31:15] Darren: if it’s unity you want to use
[24/03/2016 14:32:03] Kit: we could use unity to create some 3d model that can then be controlled by music?
[24/03/2016 14:32:33] Darren: yeh depends on the output, projection?
[24/03/2016 14:32:34] Darren: onto what?
[24/03/2016 14:33:22] Kit: we could try mocapping some dancers and then make some 3d characters dance in time to the music
[24/03/2016 14:33:37] Kit: that would be cool…
[24/03/2016 14:34:12] Kit: well thats kind of what I had in mind in the first place anyway, but thought it would be pretty hard
[24/03/2016 14:34:25] Darren: I can work on getting the dancing models
[24/03/2016 14:34:29] Darren: and making an environment for them
[24/03/2016 14:34:36] Darren: maybe just like a disco or something
[24/03/2016 14:34:55] Kit: ha yeah thats pretty cool
[24/03/2016 14:35:27] Darren: should think about output though, like is that just a screen that it’s showed on?
[24/03/2016 14:36:11] Kit: the only projector really that i can use here is the big one that i used the other day
[24/03/2016 14:36:22] Darren: ah in the big space
[24/03/2016 14:36:27] Kit: yeah
[24/03/2016 14:36:41] Darren: if you used the performance space we could play with lighting as well?
[24/03/2016 14:37:01] Kit: yeah they’ve got a pretty big lighting rig there i think
[24/03/2016 14:37:20] Darren: so we could make like a disco with blue green and red lights
[24/03/2016 14:37:27] Darren: and make the lights pulse in time with the music
[24/03/2016 14:37:36] Darren: and project some dancers onto the back wall
[24/03/2016 14:38:03] Kit: yeah thats cool
[24/03/2016 14:38:08] Darren: it’s all pretty full on stuff though thtat!
[24/03/2016 14:38:09] Darren: lol
[24/03/2016 14:38:52] Kit: yeah, probably best to scale down given the time frame
[24/03/2016 14:39:49] Kit: I could imagine it as a screen set up though with maybe a jack input so you could play music into it and have a character dancing on screen
[24/03/2016 14:40:14] Darren: yeah that could work
[24/03/2016 14:41:05] Darren: https://www.mixamo.com/store/#/search?page=1&type=Motion%252CMotionPack&query=dancing
[24/03/2016 14:41:52] Kit: HA thats cool
[24/03/2016 14:42:15] Darren: if you select ana nimation and then change ‘overdrive’ you can change the speed
[24/03/2016 14:42:48] Kit: so we can put these into unity?
[24/03/2016 14:42:53] Darren: yep
[24/03/2016 14:43:10] Kit: ok awesome!
[24/03/2016 14:43:49] Darren: so my thought is the character would just be idling on screen looking bored and then when an jack is plugged in it would transition into dancing in time with the music
[24/03/2016 14:44:11] Kit: yeah definitely!
[24/03/2016 14:44:30] Darren: if we have time we could look into mocapping some dancers but tbh we can just use some of these animations
[24/03/2016 14:44:57] Kit: yeah, thinking finding a dancer and the time to get some good mocap from it could take a while
[24/03/2016 14:45:06] Darren: yeh
[24/03/2016 14:45:11] Kit: looking thorough these some of them are really good
[24/03/2016 14:45:22] Darren: yeh
[24/03/2016 14:45:34] Darren: and I’ll make some sort of environment for them
[24/03/2016 14:45:38] Darren: probably with some nice lighting
[24/03/2016 14:46:03] Darren: I can make lighting in unity pulse with the beats or even audio reactive probably
[24/03/2016 14:46:13] Kit: ok so I will start looking at waveclock and the beat detection
[24/03/2016 14:46:23] Darren: yeh
[24/03/2016 14:48:32] Kit: would be cool to have this projection mapped somehwere
[24/03/2016 14:48:42] Kit: but thats a lot of work
[24/03/2016 14:48:52] Darren: I dunno I quite like the idea of a single screen, like a kiosk
[24/03/2016 14:49:10] Darren: some sort of kiosk with a 3.5mm jack popping out
[24/03/2016 14:49:37] Kit: yeah so you can just plug in and play
[24/03/2016 14:49:41] Kit: i like that
[24/03/2016 14:49:50] Darren: yeh
[24/03/2016 14:52:25] Kit: kinda of works as well i guess sort of linking to my other project with using dance
[24/03/2016 14:52:29] Darren: yeh
[24/03/2016 14:53:39] Kit: i like it, i like it a lot
[24/03/2016 14:53:43] Darren: cool
[24/03/2016 14:54:30] Darren: you start using waveclock or whatever its called and look at how it functions specifically
[24/03/2016 14:54:48] Darren: like, what does it do when there’s no music, and then how to it start up when music starts playing
[24/03/2016 14:54:52] Kit: ha was literally just writing that
[24/03/2016 14:55:02] Darren: cos I’ll need to work with whatever it spits out to control the idle/dancing state of the character
[24/03/2016 14:55:49] Darren: I can also potentially use general audio levels to trigger the dancing but will need the clock coming through as well
[24/03/2016 14:56:03] Kit: yeah ok cool, heading back to mums tonight (taking desktop pc and everything) so will either start looking tonight or tomorrow morning
[24/03/2016 14:56:10] Darren: cool
[24/03/2016 14:56:39] Kit: yeah thats cool, guessing you can control some other perimeters of the dancer using more than just the clock?
[24/03/2016 14:56:55] Darren: dunno
[24/03/2016 14:56:58] Darren: I wouldn’t like to commit to it
[24/03/2016 14:57:02] Kit: or maybe the lighting
[24/03/2016 14:57:23] Darren: I can make audio reactive lighting mixed with beat mapped lighting
[24/03/2016 14:57:26] Darren: that should look pretty good
[24/03/2016 14:57:40] Darren: and I can play with the camera too
[24/03/2016 14:57:49] Kit: yeah that sounds cool
[24/03/2016 14:58:06] Kit: like give it shakes or something on audio levels or control it’s pace if it’s panning around based on the beat map
[24/03/2016 14:58:55] Kit: i will start looking at ideas for a kiosk or something too
[24/03/2016 14:59:00] Darren: ok cool

Advertisements