Checksums and Verification
Part 3: Speed vs Security
This a multi part blog. As more blogs are posted, links to those posts will be included in this blog. Before reading further, we encourage you to begin with Checksums Part 1: The 5 W's and Checksums Part 2: Define and Decide.
Speed: Let's face it, we are all constantly trying to do everything faster. When my computer takes two minutes to boot up, it feels like 2 years. When a page doesn't load IMMEDIATELY it seems like the world is ending, slowly and painfully
So when it comes to offloading securely and quickly there can be a lot of factors that can seriously affect it. Here are some friendly reminders and things to consider when offloading, especially when you are considering foregoing security for speed.
Bus speed and drive speed:
Ok what's a bus and what's the speed have to do with it: The bus is a data connection between two or more devices connected to the computer. A bus enables different parts of the computer to communicate with one another.
The connection between the hard drive and laptop is connected via bus and is therefore affected by that bus and it's throughput. And as Jason Johnson so keenly put it, "This is why a RedMag on USB 3 will only offload at 120MBps. The card's bus for data transfer is the limiting factor. The same goes for Lacie Rugged drives, etc." (If you don't know Jason you should, he's a wealth of information!) Although USB 3 has a throughput of 640 MBps, the transfer is limited to the throughput of the RedMag.
Different connections, have different speed throughputs. This is important to remember! Whatever part of your workflow has the slowest speed or throughput is your bottleneck.
Ryan Nguyen also blessed me with this wonderful metaphor: "USB 3 and Thunderbolt 2 are like speed limits on the highway versus the autobahn. Thunderbolt 2 certainly allows you to travel faster, but your real world speed depends on multiple factors including weather, traffic conditions, car tire performance, etc." I love this example because it reminds us that there are many factors to consider when offloading or copying files or folders. It also takes me to my next point and again I will use Jason's words, well, because they are spot on!
"Copying one large file is faster than copying multiple files. ProRes versus DNG or ARRI comes down to a series of handshakes. Imagine making a deal and shaking hands. That is ProRes. Now imagine making hundreds of smaller deals and shaking hands each time. That is ARRIRAW. Each side needs to stop and start the action every time a file is transferred." Yet another factor, and sometimes you can't afford transferring hundreds of files - that's ok! It's important to realize that you have to adjust your speed expectations.
Security: We've established in previous posts what checksums are, how to determine which one to use and if your content is actually being verified - if you're behind read Parts 1 and 2 now!
Checksums are another factor to consider when thinking about speed. Copy and Paste is a bad practice (as well all we know) because there is nothing ensuring the copies are 100% accurate. There's nothing saying "heck yeah man, we got it all" which, by the way, is what every checksum in the world has ever said.
So what do you do when you need something offloaded quickly and securely OR (and this is dangerous but we've had the request so it must be addressed) you're willing to forego some security for more speed?
If you need speed AND security find the right checksum: that's currently xxHash.
xxHash is an extremely fast, non-cryptographic hash algorithm, working as speeds close to RAM limits. It is proposed in two flavors, 32 and 64 bits. (SMHasher on github.io)
For ShotPut Pro, ShotSum and PreRoll Post we use xxHash 64 bit. We recommend using xxHash as the checksum type unless you have a requirement for some other type. xxHash can out perform MD5 for example because it can go at the speed of your RAM whereas MD5 is a CPU dependent process.
If you need less security and more speed you can opt for a file comparison verification over the checksum verification - remember you're not going to get the "heck yeah man, we got it all" (see Wayne above) with this option. But it's a possibility and one that has it's time and place. In ShotPut Pro and myLTO we call it File Size Comparison and we even go a step further and give the option of just calculating the checksum for the source for later reference. Again, not best practice but there is a time and a place for that kind of verification and something to think about when speed is your number one consideration.
Offload confidently and be on the lookout for Checksums and Verification Part 4. If you have something you'd like us to discuss, let us know!
And now, this...
You didn't really think I would mention Back to the Future and not include an awesome shot of Doc and Marty did you??? 1.21 jiggawatts!!!