I’ve spent a lot of time working [...]]]> Rendering video in Processing is easy. The MovieMaker class makes it incredibly easy to render Quicktime video files from a Processing sketch. Unfortunately, Processing doesn’t supply tools for rendering audio using the MovieMaker class. Hence, rendering the output from a multimedia program can really be a headache.

I’ve spent a lot of time working on this problem in the last few months. I tried screen-capture software, but even the professional screen capture apps aren’t suited to the task. They cause glitches in the audio, drop frames and slow down the sketch itself. I also tried rendering using external hardware. Unfortunately, the only affordable device for capturing VGA output averages a mediocre 10 frames per second, and the frame rate is unacceptably inconsistent.

So the solution had to come from code, and in the end, the solution is pretty simple. Admittedly, this solution still slows down your sketch, but if you lower the resolution, you can get acceptable, synchronized audio and video which can be combined in any video editor.

Synchronizing MovieMaker Based on the Audio Stream

The solution is to render video frames based on the position in the audio output buffer. Simply monitor the position in the audio stream, and render a video frame every so many samples.

There are three basic code changes that are necessary to get this working. First, calculate the number of audio samples that will occur per frame of video. For this to work, the frame rate must be relatively low. 12 works well for me.


int MovieFrameRate = 12;
float AudioSamplesPerFrame = 44100.0f / (float)MovieFrameRate;


Then set up your audio recording objects as detailed in my free ebook: Sonifying Processing: The Beads Tutorial.


AudioFormat af = new AudioFormat(44100.0f, 16, 1, true, true);
outputSample = new Sample(af, 44100);
rts = new RecordToSample(ac, outputSample, RecordToSample.Mode.INFINITE);


Finally, call this subroutine in your draw function, and make sure to finalize the audio and the video when the program ends.


// this routines adds video frames based on how much audio has been processed
void SyncVideoAndAudio()
{
  // if we have enough audio to do so, then add a frame to the video
  if( rts.getNumFramesRecorded() > MovieFrameCount * AudioSamplesPerFrame )
  {
    // we may have to add multiple frames
    float AudioSamples = rts.getNumFramesRecorded() - (MovieFrameCount * AudioSamplesPerFrame);
    while( AudioSamples > AudioSamplesPerFrame )
    {
      mm.addFrame();
      MovieFrameCount++;
      AudioSamples -= AudioSamplesPerFrame;
    }
  }
}


After your program completes, you just need to stitch the audio and video together using any old video editor at your disposal.

Here’s an example sketch rendered using this method.

And here is the source code for that sketch: Video_Audio_Sync_Test_03

I hope this saves you some time and money!

[...]]]>

This is a preview of a video that is apparently being released in 2012. It is probably the most compelling “preview” that I have ever seen. The contrast of diagetic/nondiagetic sound with narrative/non-narrative film is really interesting. I can’t wait to see what the full production looks like.

[1]

Thinking about the relation of music and moving images gave me the idea of bending the codes of composition for visuals and music [...]]]>

BeatMe is a solo performance with video projection, sound and double bass.
The video/sound/Text is handled with Max/MSP, triggered by a midi-foot switch, giving the control of what is happening while playing the music.

Thinking about the relation of music and moving images gave me the idea of bending the codes of composition for visuals and music a little, changing the rules of narratives to repetition and variation, bringing them closer, making the images dance and give the music a different sense. It is all about perception, thinking and emotions. We have ideas and a feeling about ourselves in the world and all that relations inbetween. All that processes that determine who we think we are and how we feel about that. We live in an optical and sound situation.

Seeing and hearing through the things.
What you see is not what is there. Don’t misunderstand. What you see is what your neuro-circuits are allowing you to see. Waves and particles of molecularities hitting on your sensory system recalling/actualizing knowledge bases built up in the past and you are using them to look into the future, forgetting about the moment. Can this be correct?
The things you see are always representations of yourself. And you are ab/using them to reflect yourself, to tell to yourself what you dis/like about yourself. The world as you see it is nothing more than a mirror, reflecting back your desires and your fears.