video encoding using 'handbrake'
courtesy of Yan D
You can download the software and presets at the bottom of the page!
THE secrets of YIFY and high quality and small file sizes are not so secret after all…
Converting/Transcoding/Encoding High quality Low bitrate Videos in Handbrake for any device
If you’re a regular pirate of films and TV shows (unlike me, since I love obeying the law), you will notice that one of the very popular communities and trends in HD film piracy is YIFY whose release group offers films at a “high” Full HD quality at about 10% the file size (bit rate) of your common blue-ray disk.
This method of the distribution of films is very popular due to the low file sizes which can be very useful for those with limited HDD spaces and/or limited bandwidth. Many people, despite not being illegal film distributors themselves, are still fascinated by the group’s seemingly uncanny ability to make films tiny, and which to apply the techniques for legal purposes, such as ripping their legally purchased blue-ray disks in a fashion so that they could store 30 Good Quality 1080p films on their 64GB iPad.
Unfortunately, for most of those aspiring imbeciles, achieving such a feat could be a little difficult, considering how your average “movie converter” offers little options more than that of “set bit rate” and “set file type”. This is highly problematic because film converter software were designed for one purpose and one purpose ONLY: to make a file compatible with a device while consuming as little time in the process as possible. That is why if you try to encode a film in 720p @ the same bitrate of YIFY @ say… 800Kbps in your average media converter (such as freemake video converter, format factory, Ultra Video Converter, Arcsoft video converter or even Apple Quicktime), you’ll end up with something so ugly and so full of video artifacts that you would want to kill yourself for being so stupid as to think you could match the quality of YIFY in the first place. The reason for that is because these video converters utilize professionally determined presets that are not only good enough for the average user, but also quick enough for the impatient bunch. Which means that hardware acceleration such as nVidia CUDA or AMD’s equivalent would be used, and while those are incredibly useful for capturing games or on-screen activity on limited hard disk space, they can (and almost always will) result in file sizes that are a little too big for long term storage. Plus, graphic cards lack the ability to encode some of the complex algorithm abilities (yes, even your GTX680 will SUCK at encoding YIFY, despite being incredibly fast in the process), so therefore most professional encodes will try to avoid hardware acceleration from graphics cards.
So you see, there are many factors in influencing a good encode, of which the basics will be covered by this article.
Lesson #1, THE CONTAINER DOES NOT INFLUENCE THE FILE SIZE.
There is a common misconception that MP4 is better than AVI, or MKV is bigger than MOV, and to be fair, that is true to some extent. Provided the same content, an MKV file can be as much as 300KBs bigger than its equivalent MP4 file. This is because the container used in the MKV might store more data, or store less data in a slightly more inefficient way. However, as you might know, a 300KB difference is pale in comparison to the range of file sizes offered of the same films in allegedly the same qualities. So what could be causing the difference?
The “MKV” or “MOV” is just a representation of the CONTAINER to the file. The container is what stores the video and audio content known as “streams”, which are the actual content encoded in different codecs.
THIS is what really matters
The only difference between containers is that sometimes certain containers can only support certain formats, or in avi’s case there is a 2GB size limit on the total length of content.
The best codec as of 2012 is still h264, a standard since 2004. While some argue that Youtube’s VP8 is comparable, x264’s developers have claimed that VP8 losses valuable compression with a limit on reference frames and the complete absence of B-frames and weighted prediction in an effort to be royalty free. And if you’re reading this article searching for YIFY, you probably don’t give a shit about royalties, so VP8 is out of the question.
Some Hong Kong advertisements for RMVB-capable products say that HongKongers love RMVB. If you love RMVB, then I got news for you: RV40, RMVB’s current codec, is based off h264 like everyone else. However, I have doubts that any RMVB encoder out there will give you as many options as x264, so essentially to you, the average user of the computer world, RMVB is about as useless to this cause as the H263 based DivX. (Apparently DivX has its own h264 encoder, but 1. It’s free version is very limited and 2. it either costs money/ or watermarks your shit, plus it comes with bloatware, and its quality is not on par, so fuck it)
So currently by raw comparison, the most appropriate codec for the cause is x264, and mkv, flv or M4V has almost absolutely nothing to do with better or worse quality of videos.
Lesson #2, THE CODEC DOES NOT AUTOMATICALLY IMPLY QUALITY
“Whaaa~? But you just said-”
Yes- while this is true in your average simple “converter”, (again,) x264’s developers have taught us that a well encoded MPEG2 file will be of much better quality than a badly encoded H264 file. And this is true, because codecs do have different levels of compression.
For example, H264 offers several profiles from 1 to 5.1, with each increase in profile implying higher playback requirements.
Handbrake makes it easy to make your file compatible. Select a preset or just tick off “iPod 5G support” and “Web optimized”, or manually disable B frames and set reference frames to 1, and disable CABAC, 8×8 etc.
For example, your expensive second hand iPhone 4 supports a puny baseline 3.1, which apparently means it will not playback files with resolutions of higher than 1280×720, and is also limited to a bit rate of 14000 in the main profile.
Not so premium anymore, huh.
Even worse, an older iPod might support H264, but will certainly not support B-frames and CABAC, which both contribute to compression levels significantly. For now, you should stick with H264.
Here’s something you’re not psyched about: H265(HEVC), the successor to H264(AVC), is supposed to be released sometime in the (near future), offering as much as 4x better compression in its (beta? Alpha?) current state of development. Hopefully an HEVC encoder will be available for the general public soon.
Lesson #3 THE BITRATE DOES NOT AUTOMATICALLY IMPLY QUALITY
That’s the point of this article. While a higher bitrate generally means higher quality, this is only valid within the limits of the same compression settings. Simple comparison is a CUDA encoding VS a CPU based encoding. The “advanced settings” and options in real encoders do exist for a reason.
Lesson #4 FASTER ENCODING GENERALLY = LESS QUALITY, and avoid (cannot stress this enough) video converters (and some video editors).
Again, why professional encoders never use CUDA for reasons I have already discussed.
“Video Converters” Are good looking software that offer very limited options, its unalterable presets designed only to make your file playable on your iPod in as short a time as possible, and nothing more.
My dad always tells me that if I really want to work with multimedia, I should go work for a big multimedia company with fancy machines because it will never be possible for me to set up a studio in my apartment. In a similar aspect, if you are trying to be a film-maker, the file size of your video should be the last of your concerns. When “exporting” from Premier etc, always export in a format and settings that are near-lossless, then re-encode them later for specific channels of distribution.
Lesson #5 USE SINGLE CORE IF POSSIBLE*, USE SINGLE CORE IF BATCH PROCESSING
“No way”, you say, “I may be an idiot attempting to encode similarly to YIFY when I clearly should shut up and just leave the encoding to the professionals, but even I know that multicore is good for multimedia processing”.
Too bad, moron. While multicore does have numerous (I cannot stress this enough) speed advantages, today’s topic is compression. And the thing about multicore processing is that (to my understanding) data is split into chunks for the processors to process, meaning thread 1 will receive chunk A, thread 2 will receive chunk B and thread 3 will receive chunk C. However, while they are processing, thread 1 cannot actually alter chunk B. Now compression is based on finding similar items and patterns, among other stuff, but when thread 1 is occupying chunk A, thread 2 cannot occupy chunk A to see if there are any similarities between them, thus resulting in a loss of compression. A single threaded operation however, will process the video bits one by one in an orderly fashion, thereby eliminating the chances of missing anything and achieving greatest compression in comparison to multicore operations.
Your average high end nVidia GPU has a shit load of CUDA cores. If it even attempted to do a high compression encoding, I’ll bet 95% of the cores would be inactive half the time, just waiting for the other core to finish processing that chunk of data.
Update: Lemme clarify this some more, as folks at at this page have expressed doubts that a single thread encoding “improves quality”.
What I mean to say is that say on a quad core cpu, running four separate encoding threads with each thread using only one core, in parallel (ie encoding video1, video2, video3 and video4 at the same time) is much more efficient than having four multithreaded encodings run sequentially (ie encoding video1, and then video2, and then video3, and then video4), as restricting the encoding to one core would result in on average a 99% to 100% core usage during encoding, while if you encode in a multithreaded operation you might only have a average cpu load of 90%, because of the cores waiting for the other cores as described above.
So no you won’t lose quality in a multithreaded high compression encode because x264 is smart and understands when you value compression over speed, but you will lose speed and efficiency. Unless you force your multithread encode to operate your CPU at full load, then, the resulting product would definitely be of worst quality than a single core encode.
(assuming “ultra” compression settings) Just as if you run 4 single threaded 7zip compressions in parallel on a quad thread/quad core computer, it would be much faster than running 4 multithreaded 7zip compressions sequentially. the 4 single threaded 7zip threads would put your CPU at 100% load constantly when a single multithreaded 7zip thread would use between 60% to 90% of your CPU (and therefore 60%-90% the speed) at any given moment.
But then again CPUs are so damned fast nowadays, so does my advice on speed and efficiency really matter?
edit again: dammit my incompetence sucks ass. yes you can restrict handbrake/x264 to use only one thread by adding threads=1 to the command line.
Lesson #6 BE AS LOSSLESS AS POSSIBLE WITH YOUR ORIGINAL CONTENT
If, say you are not trying to rip a newest blockbuster off your new Blu-ray disk, but rather you are trying to compress your family videos into a smaller size or you are an aspiring Youtuber (or a porno producer), please don’t do a YIFY-styled encoding, because the truth is, despite being far from rubbish, they still suck- the artifacts will still be unpleasantly visible. As a family man/woman/guy, you want to retain your memories with junior in as good a quality as your camera captured it. This means no RF30 for you; go straight to RF15 or less. (This RF stuff will be explained later)
As a producer/ video editor, you want to have the highest possible quality videos available for streaming. Let Youtube or Porn Site screw it up with their own crappy compression, it’s not your job. Your job is to make sure that when a new codec comes out, they will find your old file, re-encode it, and hopefully make it look better.
This is especially true for desktop captures. If you are making a screencap tutorial on using software, for GOD’S SAKE make it lossless! The last thing anyone wants to do is to find out that the lines for writing a GUI for smudge is smudge smudge smudge smudge smudge.
Lesson #7 DO NOT DO FILE SIZE/ BITRATE BASED ENCODES
A lot of P2P groups encode videos to file sizes of 400mb, 500mb, 600mb, 700mb, 1gb, 1.5gb etcetera. This was because it was easy for people to estimate file sizes, and also filling up a 700mb cd with a 699mb file felt good. However, that’s not a very smart thing to do. You see, people are terrible at estimating bitrates, which is why some dvdrips come out with a better quality than others. It was an impractical thing from a time where “699mb” was better than “701mb”- but that time is passed. Now, more and more groups are doing RateFactor based encodes, the “RF” I talked about earlier. The Xvid encoder has a rate factor slider, so does x264. To ensure that every file you ever transcode, whether action movie or cheap animation- comes out with the same quality, you have to do quality based encodes. The default slider in Handbrake is set at 20, however it is suggested that only a RF of 15 or lower can a file seem almost lossless. From my personal estimates, both 720p and 1080p encodes from YIFY are roughly equivalent to a ratefactor of 30, quality wise, and ratefactors of around 30 are recommended.
Lesson #8 DO NOT UPSIZE FILMS
x264’s default deblocking- from 352×144 to 1366×768
I’ve seen dumb people ask for instructions on “making a DVD into HD quality”, aka interpolating it. Hey- newsflash: basic h264 decoder offers built in deblocking which is similar to “smoothing”, which is also interpolating, which is upsizing, and the deblocking filter is only deactivated optionally on lesser devices, so unless your choice of interpolation software is REALLY FRIGGIN FLY, I suggest you just leave your film at the resolution it is, and let the guest do the dirty work.
Now there IS an exception to this. If you plan on uploading footage from 3D games or you kidnapped Michael bay to make your youtube videos, then it is perfectly acceptable to upscale a 480p or 720p film to 1080p, just so that youtube would give it a higher bitrate and therefore better quality, because for youtube and streaming sites in general, quality will always be crappy for high motion videos, and upscaling is one way of countering that.
Lesson #9 DO NOT RE-ENCODE AT A HIGHER BITRATE FOR NO REASON
Especially when at the same encoding settings.
Re-encoding in general is a bad idea. If you’ve ever dealt with early versions of ms paint, you’ll know that if your file is saved as a jpeg image, each time you press ctrl-s the quality degradation becomes more and more obvious. That is because the jpg file is saved at a quality of around 75% the original in order to achieve what is deemed the appropriate level of compression.
Re-encoding at a higher bitrate will not “interpolate quality” by itself, which leads to -
Lesson #10 Use filters and “effects”
If you’re a hardcore file sharer who wants to keep the film in as much of its original form and detail, then too bad- “600mb 720p mkv” style ain’t for you. On the YIFY website he clearly states that he uses a combination of software and settings that work best, which will most likely include filters such as “contrast”, “cartoon” or “denoise”.
A light amount of Cartoon special effects will simplify colors, reduce file size requirements while making the film still seem high quality.
Contrast will help simplified colors look better.
Not only does denoise cancel out “noisy” scenes, it also helps compression significantly, because it also smooths out grains, which are bad for video compression. I had a personal experience with this, where I filmed s short walk down a mountain, then used a movie editor to give it noise. The resulting 5min 352×240 file @RF15 in similar AVC encoding settings occupied around 1GB of HDD. Noise is not good for compression, so use denoise filters.
Lesson #11 Single pass bitrate encoding is not necessarily worse than two-pass bitrate encoding
Multi-pass encoding wasn’t just a way to improve quality, it’s purpose was to generate a stats file in the first pass which will be used to determine the appropriate RF used so that the resulting file is same size as that which the user specified. Normally, single pass bitrate encodings convert your bitrate into a RF value with a maximum bitrate limit and takes it from there. The result may be larger or smaller than estimated- multipass merely makes sure it’s the exact bitrate you want it to be.
And Finally- Lesson #12 VIDEO ENCODING TAKES TIME
Encoding in H264 takes a lot of time. I’ll assume this guy was using one of those gay presets (I automatically assume that almost all presets are gay) in AutoGK, and even that predicted 14hrs for *just 4 hours of film.
It takes guts for the average human being to try and encode like YIFY. For example, I’m on an Acer Aspire-481G ultraportable, and that means a 17W i5-3317 ULV Processor. I have no secondary computer, so this means that whenever I encode, I encode on the go, everywhere, anywhere, and anytime, and naturally there are many problems I’ve experienced. For one, I have to put the computer to hibernation every time I stow it in my bag or it’ll overheat. Secondly, the battery drains superfast every time I bring my computer out of hibernation and it continues its task. I have to constantly keep my laptop plugged in, and that is a “bit” damaging to the battery.
So far, it’s just me bitchin about my slim ultrabook wannabe. But here are some things that might apply to you too:
1. If you don’t have a fast processor, it’s gonna take a while. In my trial and error session with a laptop I borrowed, it took almost 4 days for an i7-2630QM to encode Total Recall (2012) to a YIFY-like 1080p output. If you have an old computer, you’re electricity bills are gonna get fagged up, because old processors are totally less efficient.
2. When you have the encoder running in the background, you cannot play games. You’re limited to small tasks such as waiting for the mouse to move from 1230, 230 to 934, 317 (well, not unless you’ve got lots of other shit open), and maybe checking your email. You can’t edit videos, and you can’t play old games even though they’ve got lesser system requirements since when all the threads of your hyper threaded quad core 3.6ghz CPU are used simultaneously, the old single threaded game will only be able to use one thread at a speed and efficiency of around 200-300 Mhz (3.6/4/2=400)
3. You cannot reboot
4. Everything is slow
5. Even if you have the newest fastest consumer processor, you will still be unable to fully utilize your computer for at least a day if you wanna do a single threaded 1080p encode of a 2 hour film, and if you make a mistake, you start over.
Bottom line is, why do it yourself if someone with the time and equipment has done it for you? Therefore I discourage anyone who actually needs their computer every day from doing video encoding. It’s messy and not for the impatient or the morons.
Continue reading if you still think you’re up to the task.
Now that you’ve learnt the basics…
1. Install, then open up Handbrake (this tutorial covers version 0.98)
2. Drag your film onto the program or select Source-> Video file
3. Downloadthis preset pack for handbrake 0.98 x64 , and select options->import from the bottom right corner of the program. Locate the downloaded file from the window and open it.
4. Locate “Placebo” in the preset list and double click
5. If you wanna downsize the video, change the width in the appropriate box (green circle) The “display size” might show weird numbers, but it is most beneficial to keep it that way, as leaving the resolution non-anamorphic will result in worse compression. Don’t worry, your video will still be displayed in the resolution it should be.
Check the red circle’s area. If you see that there are numbers other than zero in the listboxes but your video doesn’t need to be cropped, select “custom” and edit all values to zero. This function automatically removes “black bars”, but is buggy in some cases.
If your video is in an incorrect aspect ratio and you want to correct it using a custom resolution, select Anamorphic and choose none. Now you can set custom values to make it 4:3, 16:9, etc.
If you want to keep your original resolution, select this too.
6. Now you can flip to the filters tab and apply some simple filters. Note that if denoise is not set to “Medium”, set it to “Medium”. Grayscale encoding might help compression by 0.5%, but the end result is, you guessed it- grayscale.
Note: If you’re producing videos, turn off denoise.
7. Flip to the video tab. Make sure the codec is set to H.264. Make sure the FPS is same as source, and you can decide whether you want variable or constant framerates. VFR helps compression a bit, and its effects aren’t that obvious as the resulting framerate wouldn’t be hugely affected- you probably wouldn’t even notice it were there.However, constant frame rate is preferable for high motion videos- better safe than sorry, right?
8. The default RF value is set as 30, good for 1080p (1920*xxxx) and 720p (1280*xxx) all the way to DVD quality (720*xxx). For 480p, use a RF of 25. For 240p, use a RF of 23.
If you are producing, use RF15 (or23) for full HD, RF14 (or20) for 720p, RF12 (or 18) for DVD and 480p, and RF 10(or17) for 240p.
*OR values are for if you really value your HDD space that much, then fine.
If you insist on a specific file size, calculate the bit rate (including the audio of 96kbps by default), select Avg Bitrate, enter value, select 2-Pass Encoding and leave turbo first pass alone. This is only useful if you’re encoding for cd or DVD. I personally use the xvid encoder that comes with klite codec pack to calculate bit rate, but I’m sure much better solutions are availabe.
9. Switch to the audio tab. If the box is empty and your video is supposed to have audio, click “add track”. If your video contains multiple audio tracks and you want to add a specific one, click on the listbox beneath “add track”, and select from the list. You can add multiple audio streams, however if you do, please change the container from MP4 to MKV, as the MP4 container does not support multiple audio streams. If a track is labeled as o 0 (as shown in picture below), you have to remove it and add it again. To remove audio tracks first click on a track, and then press “remove”.
Make sure the audio codec is faac, mixdown stereo, samplerate auto and bitrate @96. If you really hate faac that much, you could always mux in your nero/itunes encoded aac stream later.
If you’re a producer for (especially music videos), select “360” for the audio bitrate.
10. Switch to subtitles tab. If your video contains subs, now is the time to add them. I recommend that you add all of them since subs are so damned small anyways, however if you add subs you have to set the container as mkv (matroska)- mp4 does not support built in text subs. You add subs the same way you add audio streams, and here you get to change several properties of the subs as well.
update: If you know your video contains special subtitles (such as the Jabba the Hutt translation in Star Wars) and you want to preserve it in your encode but it isn’t listed in the list of subtitles, add “Foreign Audio Search” to the list of subtitle tracks.
11. Switch to the advanced tab.
“Holy crap, what the hell is that?”
You are now at the very last stage. This is where you set the compatibility of your video if you want it to work with a specific device. Check the spec sheets of your target device for things on this list you might want to disable to meet various profiles. If the preset don’t import properly, copy and paste this into the box at the bottom of the window:
Note: disable (ie. delete) threads=1 if you’re only encoding 1 video
These settings basically emulate the “placebo” preset in x264:
This setting pushes h264 to its very limit. It’s a general purpose set of options that ensure maximum compression happens. For more info on the settings (and this is why I like Handbrake so much), hover your mouse on each of the options and a description will appear explaining it.
*edit: add “threads=1″ to the options. this is a life saver in both batch processing videos and in cases where you might want to be able to utilize the rest of your cores/threads for other uses if on multicore computers. However if you only plan on encoding one film in the near future, might wanna disable threads=1 as it is seriously slow on multicore and/or hyperthreaded systems and is only useful for batch processing.
If you are encoding for production, you might want to completely disable Psychovisual Rate Distortion and Psychovisual Trellis, as well as Deblocking. These can cause artifacts. Or blurring.
12. Now you’re all set.
Press Start, dumbass
Last but not least…
Video quality is nearly always more important than video resolution. The way YIFY has been able to evade its lawsuits… the ugly truth is YIFY encodes are still sub-par copies that no sane film producer would sell (aside from youtubers, amiright). I personally archive my films in a relatively high quality- low resolution format which I call “Recap Format”, basically VCD resolution @ RF23. And I’m happy to say that watching recap format can be a much better experience than watching a YIFY low bitrate 720p encode, even if the source of the recapped video is YIFY.
In the future when bandwidth, lossless video compression and storage mediums evolve, we will be able to stream lossless videos without artifacts off Youtube and download a few TBs of near-lossless 4320p films off iTunes- and by that time, YIFY styled encodes might just be headed towards obsolescence.
New: if you want you can download a sample of a recap format encoding by visiting recapstore.wordpress.com, where I am setting up a website dedicated to archiving films in the format.