Tarun made a case for me to write about technology. At first I resisted because I deal with it all day so why would I want to deal with it when seeking a creative outlet?! With his typical you-have-got-to-change-the-world persuasion, he gave me hundred reasons why this is a good idea. So, I am writing this post partly to make him stop and partly because I want to tell you how each one of us is contributing to enormous changes in our digital landscape with our pings, posts, tweets, emails, blogs, and searches. To my tech friends, apologies for distilling these complex topics into a single post.
Big data is indeed yuge.
Yup, check. But let me take you through the journey to "big".
A bit is what stores a 0 or 1, the basis of computing. From there you move up to a byte (8 bits) and then
kilobyte (1024 bytes). Keep multiplying each term by 1024 and you get to megabyte, gigabyte, terabyte, petabyte, exabyte, zettabyte, and yottabyte.
When you get to a petabyte (13.3 years of HD-TV video) you have entered the big data zone. Now consider this: Facebook processes 105 terabytes of data each half hour. Google processes 20 petabytes of data per day. These volumes require whole new processing paradigms - both for hardware and software.
In addition to volume, big data has two other interesting characteristics - variation and velocity. Big data is unstructured data, meaning you can't bucket the information into neat rows and columns. Think social media pictures and comments. You can't really parse the comments into cleanly designated columns, can you? You could keep each word in a column but to what point? How do you structure a picture? Instead you end up with a massive volume of comments, tweets, posts, videos, pictures, and whatnot which can only be tagged by a person, place, and time.
The variety of this data is enormous. Think phones and appliances, think grids and GPS, think wearable devices. Think Alexa, Siri and Whatsapp. Think of all of the data that these digital critters are spewing - because of us - and your mind will blow into bits (oh, pun!) at the enormity of the digital ocean we are drowning in.
By 2019, global internet traffic will be 2 zettabytes per year. When we arrive at a yottabyte we will require data centers as big as the states of Delaware and Rhode Island that cost a $100 trillion dollars. Ouch!
Hadoop is the world's cutest yellow elephant.
Handling big data requires special techniques. Enter Hadoop. Every time I see this little guy, I just want to hug him!
It is a file system that allows you to store and retrieve big data with ginormous speed. With low overhead and distributed data processing, you get efficiency fit for our new world.
Alongside Hadoop come lovable names like like Flume, Sqoop, Hive, Pig, Mahout, and the popular MapReduce and Spark. These are the technologies that help us traverse the world of big data - analyze in light speed and transact business at a massive scale.
Think billions of users on Facebook, Insta, Twitter. Without technologies such as Hadoop it would be entirely impossible for social media to exist.
Virtual Machines (VMs) aren't smoke and mirrors.
Now that we have the elephant to handle our data, where are we going to process it all?
Not that long ago, we knew what "servers" were. Those boxy things that whirred while we pounded on keyboards the size of a small mammal. One day, the world literally ran out of shelf space and power to light up another server. Data centers filled up. Every bit of space under desks was gone.
Brilliant minds came up with a radical scheme. It was like a time-share. Instead of owning a server, you'd lease computing horsepower - you'd say I need so much speed, so much memory, so much disk space - and rent it when you need it from providers who sit "somewhere out there", meaning the "cloud".
An unlikely candidate of sorts made virtual machines leap into our consciousness - the bookseller once known as Amazon.
We all know that Amazon got so big, so quickly by diversifying from books to everything under the sun. Underlying their astounding growth was their computing magic. Although not the first, they mastered the art of virtualizing computing horsepower through the use of VMs. Instead of one machine being dedicated for one purpose, they had a bank of physical servers that served many different applications and millions of consumers simultaneously, while loaning and distributing computing power.
As they scaled, they realized that their computing business itself was self-sustaining and world-changing, and voila, Amazon Web Services and cloud computing went mainstream. Now there is Microsoft Azure, Oracle Cloud, IBM cloud, and others who are competing for our dollars. You’ll be hard pressed to find software that isn’t on the cloud and does not use VMs.
Our best friend Gmail sits on VMs in the cloud and every single day between all of the apps you use, you are keeping this alternate universe very, very busy.
Blockchain and cryptocurrency aren't sci-fi elements.
The changes in the hardware and software paradigms are inevitably giving rise to new applications. Here's an example.
How often do you use cash these days? Enter digital currency, stage right. Exit cash, stage left.
Digital currency, aka cryptocurrency, is a unique beast. It is maintained using a digital ledger. The ledger keeps track of how much digital currency an individual owns. As you buy and sell in digital marketplaces, the ledger keeps track of your transactions.
The ledger uses a currency called Bitcoin and technology called blockchain and it is replicated across many nodes (on virtual machines, of course) across the world. If a hack occurs on a particular node, the other nodes recognize the breach and auto-correct the compromised node. In the end your transactions and your currency are safe because the ledger retains integrity.
Now extend the blockchain concept to real estate, health care, and the electoral process and you'll see the transformative impacts. The ubiquity of blockchains is years away but keep an eye on it, or else it could engulf you.
So why is all this important?
One of two things is going to happen in our future. Masses of people will lose their digital identity to theft and we will revert back to paper and postage stamps or we will rapidly keep moving towards more and more digital transactions.
I daresay we aren't going to move backwards because...um...there is no cell signal in caves. You really want to go to the bank to deposit a check? Heck, retail branches of banks are shutting faster than I can keep track. Besides, Russians are in the caves.
So yes, we are going to move to increased digital processing at light speed. My suggestion to you is this: since Bitcoin is a bit too esoteric, try Apple Pay and Venmo to keep pace. The next generation is already surviving and thriving on these technologies.
Since we all want to be hip, let's get with the program.
(Stats in this post are from Gizmodo, Forbes, and other sources.)