Fractals and Music Visualization with FFmpeg
FFmpeg is a powerful video compression tool that has been previously talked about on this blog here. But did you know that it is also a serious video generator in it’s own right?
To get started, it is remarkably easy to install FFmpeg on Unix. Since we’re using OS X here, we will install using homebrew:
[code language=”bash”]brew install ffmpeg[/code]
However, if you actually want to do anything with FFmpeg, you need to install some libraries. This should take care of most use cases:
[code language=”bash”]brew install ffmpeg –with-vpx –with-vorbis –with-libvorbis –with-vpx –with-vorbis –with-theora –with-libogg –with-libvorbis –with-gpl –with-version3 –with-nonfree –with-postproc –with-libaacplus –with-libass –with-libcelt –with-libfaac –with-libfdk-aac –with-freetype –with-libmp3lame –with-libopencore-amrnb –with-libopencore-amrwb –with-libopenjpeg –with-openssl –with-libopus –with-libschroedinger –with-libspeex –with-libtheora –with-libvo-aacenc –with-libvorbis –with-libvpx –with-libx264 –with-libxvid –with-fontconfig –with-ffplay –with-tools –with-zimg[/code]
Once you have all that installed, you can make a mandelbrot zoom video with the following command:
[code language=”shell”]ffmpeg -f lavfi -i mandelbrot mandelbrot.mkv[/code]
This is going to take a long time. I eventually got tired of it running on my laptop (not the best choice anyways, anything with a serious graphics card would see serious performance gains) and ended the process prematurely. The video I got was still much longer than I care to sit through but had the downside of not skipping forward well in VLC. Speaking of, .mkv is a superior format if you don’t mind downloading VLC to watch videos as it is not supported natively on OS X nor Windows to my knowledge. While .avi should be supported natively on Windows (it is a Microsoft-developed format), it is not open source and has far fewer modern video container features.
The video is here:
Now, part of what makes FFmpeg great is the documentation. It’s not always perfect but it’s usually pretty good. See the text filters below for an example of some sweet examples.
Now, it is relatively simple to add music:
[code language=”bash”]FFmpeg -i Imagine\ The\ Future.mp3 -i mandelbrot.mkv -b 2048k -to 03:57 mandelbrot-imagine-short.avi[/code]
-i stands for input, the first is our audio channel while the second is our previously created mandelbrot video. You can see that we’re outputting to .avi this time which is natively supported on OS X but that we have to set a higher bitstream with the -b option. The -to 03:57 says not to generate any video after the 3:57 mark because that’s when our previous video ends (otherwise the music will play out to the end of the video). This is one of the few times I’ve found FFmpeg’s defaults to be disappointing. For that reason, I will try not to post commands overloaded with unnecessary options.
All the music reproduced in audio form is expressly permitted by the licenses of said music. The song above is attributed in the video itself as under this Creative Commons 3.0 License
Moving on, it is also fairly easy to create spectrums in image and video form from audio sources.
For images:
dig up code and insert result
For video:
[code language=”bash”]ffmpeg -i high-adventure.mp3 -filter_complex "[0:a]showspectrum=s=1280×720,format=yuv420p[v]" -map "[v]" -map 0:a -c:v libx264 -c:a copy spectrum.mkv[/code]
However, if we want to make the video HD:
[code language=”bash”]ffmpeg -i high-adventure.mp3 -filter_complex "[0:a]showspectrum=s=hd1080[v]" -map "[v]" -map 0:a -c:v libx264 -c:a copy spectrum.mkv[/code]
And we get the beautiful video shown here:
As you view the spectrum video, it becomes apparent that time is the x-axis. The y-axis represents frequency and is determined by the square root function. This represents the lower frequencies much better than linear. While on a linear function something twice as high visually is double the Hz, on a square root function it is more difficult to tell.
One thing to note about these spectrums is that the stereo width is color coded. The right channel is represented purple while the left channel is represented green. When both are present, the frequency is shown as white. The image shown in the preview for this blog post is actually the short edit of the track (Days of High Adventure by Open Source)
Simple text filters are also possible:
[code language=”bash”]ffmpeg -i spectrum.mkv -vf drawtext="fontsize=50:fontfile=/Library/Fonts/Arial.ttf:text=’Wolf-e-Wolf – Purple Planet’:fontcolor=black:x=(w-text_w)/2:y=h-(2*text_h)" -b 2048k text-hd.avi[/code]
FFmpeg has considerably more features than it is commonly known for. I highly recommend you check out the docs here and try something unusual. Conway’s Game of Life in particular was something I didn’t have time to explore.
