The
Evolution of Big
Data and the Future
of the Data Platform
–
How organizations use data platforms
to get more value from data
Introduction
–
The field of big data has developed from the discipline of statistical
analysis all the way to today’s advanced data platform technologies.
In this ebook, we’ll describe how we got here,
the challenges big data
presented along the way, and how organizations are using data platforms
to get more value from data than ever before. You’ll learn how big data
technology is evolving to better connect us,
improve our decisions, grow
our economies, and more.
Introduction
Big data beginnings
New big data approaches
Big data challenges
Data lakes
Data platforms
AI and ML
Business Use Cases
Conclusion
02
The Evolution of Big Data and the Future of the Data Platform
Introduction
Big data beginnings
New big data approaches
Big data challenges
Data lakes
Data platforms
AI and ML
Business Use Cases
Conclusion
Big data beginnings
–
Put simply,
big data
is a concept describing data sets that exceed the
size that can be managed by
traditional tools. It’s defined by three Vs: variety, volume, and velocity. The growing variety of data
sources that arrives in increasing volumes and with more velocity (the high rate at which data is
received and acted on).
The roots of big data come from “business intelligence,” a term
IBM (PDF)
coined in 1958,
defining it as
“the ability to apprehend the interrelationships of presented facts in
such a way as to guide action towards a desired goal.”
IBM, 1958
The 1960s and ‘70s saw significant advancements in data technology with the development of
mainframes and databases. The 1980s saw the emergence of personal computers and client-
server computing and, along with that, relational databases and SQL (Structured
Query
Language). With each of these breakthroughs, the utility and volume of data grew.
Data volumes exploded in the ‘90s with the rise of the internet, ecommerce, and
search technologies. For transactional databases, this
meant new architectures
to support more performance, scalability, and redundancy. At the same
time, the need for business intelligence across these data volumes
drove companies to create new types of databases—data warehouses,
specialized relational databases optimized for analytics—to store curated
data from a wide variety of sources. The
data warehouses
became core
infrastructure that companies
used to track their operations, complete
reporting, perform analysis, and support decision-making.
