Quantizing in audio is nothing new. This is where an audio recording of a drum track for instance is moved around as necessary to fit into a precise, predetermined matrix.

So if you have a sloppy drummer, you quantize the recording so it moves late beats backwards and early beats forward. It's the rhythmic equivalent of autotune for singers.

The proposed idea is to have the audio track for a video quantized, but retaining its alignment to the video track. Easiest example would be to show a drummer playing sloppily, the audio/video quantize function would be activated and you would hear AND SEE a drummer playing perfectly on beat.

This effect is currently done by hand where people manually cut up videos and drop them into audio tracks to make novelty, usually humorous music videos.

So this could be done automatically or with some manual editing ti fine tweak it. You'd take a video of your aunt complaining about her bunion, click "analyze audio track" and it would move the main "beats" of her conversation to the nearest quarter note for instance giving it the appearance that she's rapping. Alternatively, you could manually look at the peaks and valleys of the audio signal over-layed on a beat matrix and you could move it by hand. Once it sounded right, you'd hit "play" and watch the video of the person wrapping.

Doesn't have to be just people talking either. Any video with sound could be made a rhythmic performance piece.

You could also add auto tune to the audio track and make any video where people were talking or even making noise a music video.
-- doctorremulac3, Jan 30 2016

This is an example of the effect where everything was done manually. https://www.youtube...watch?v=WsqdmqRgrIc
The idea would be to do this automatically. [doctorremulac3, Jan 30 2016]

Another great example. https://www.youtube...watch?v=bFEoMO0pc7k
Brilliant. Ripoff of this idea coming in 3...2...1... [doctorremulac3, Jan 30 2016]

Only vaguely related... _22Mom_20and_20Dad_...is_20is_20Chasey_22
[normzone, Jan 30 2016]

Video Rewrite http://mrl.nyu.edu/~bregler/videorewrite/
Pretty much this but only for videos of people speaking [notexactly, Feb 10 2016]

Video Rewrite paper http://cims.nyu.edu...er/VideoRewrite.pdf
SIGGRAPH 1997 [notexactly, Feb 10 2016]

Disney Research VideoSnapping http://phys.org/new...ultiple-videos.html
Pretty much this but for temporally aligning one video to another. SIGGRAPH 2014 [notexactly, Feb 10 2016]

Inverse-Foley Animation https://www.youtube...watch?v=EGkQkdCKztM
Pretty much this for generating CGI video temporally aligned to existing sound. SIGGRAPH 2014 [notexactly, Feb 10 2016]

My idea would be to have a audio/video linked stretchable timeline where audio peaks would be automatically moved to the next grid point.

Fine tuning could be done with a graphic interface, moving left or right to taste.

You could have a music audio track as the guide for your video audio track. Instant rap video complete with choreography moving to the music.

You could jump around waving your arms like reciting the alphabet and it would come out a rap/dance video moving exactly to the beat.
-- doctorremulac3, Feb 10 2016


See my [links] for prior art. Inverse-Foley Animation is the one I remembered, but I didn't remember the title, so I did a Google search for [siggraph video audio sync] and found all of those.
-- notexactly, Feb 10 2016



random, halfbakery