Processing Applications

Here are some of the applications that we have been using with our students.  The downloads are hosted on our wiki – feel free to download them and any feedback would be greatly appreciated.

Drawing with Sounds
A cause and effect application which draws bezier curves and ellipses depending where the mouse is on the screen. It also produces sounds when the shapes are drawn adding an audio stimulus to the visual one. Instructions are included in the download. This sketch requires the SoundCipher Library for Processing by Andrew R. Brown.

Click here to be taken to the download link.

  1. Hello James, Your colleague, Keith gave me your site info. I am very thrilled to find this app and have some questions for you. For my study, I need to create a sound painting app with both gesture and mouse tracking.

    1. How do I translate the mouse tracking to gesture tracking? Either using skelton or simple hand movement. Any code suggestions are appreciated!

    2. Can I fade away the images like your other app, Shape_Sounds_Soundcipher?

    Looking forward to hearing from you.


    • Hi Chiaki,

      Thanks for the comments, in terms of the drawing with sound sketch that Keith produced we had looked in to developing this into a kinect sketch but due the sketch working using mousebutton we felt the kinect was not suitable for it as it would be continuously playing the sounds as soon as it detected the users.

      This is what led me to create the Shape_sounds_soundcipher sketch. This uses closest point tracking as mentioned in the Borenstien book that Keith mentioned. Uses the code from that book to detect the closest point using depth array data and the pictures are drawn when the closest point enters specific areas on the screen. Also used basic shapes as these use up less processing power than beziers.

      I have played around with skeleton tracking and have a version that works by tracking the left hand. Have not got round to adding the whole skeleton tracking to this yet.

      To make the shapes fade away, I get the sketch to redraw a white background in the draw function. This also has a transparency setting to give the illusion of the shapes fading away.

      I understand Keith sent you code that runs in processing, should work also in openframeworks.

      Hope this answers some of your questions, if you need any other guidance I recommend the Borenstien book as this is explains the skeleton tracking functions of the kinect, but am happy to help if you have more questions.

  2. Thanks, James!
    It makes sense. I find skeleton tracking is a bit tricky for small children(my participants). The closest point tracking might be a good solution, perhaps. I will try my best with coding plus Borenstien’s book(got it two weeks ago…) for my project but I may consult you if it gets out of my hands.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: