Introduction
Big data beginnings
New big data approaches
Big data challenges
Data lakes
Data platforms
AI and ML
Business Use Cases
Conclusion
Big data challenges
–
Many companies that adopted these big data technologies have found that
managing the various open source Hadoop tools was complex and expensive.
Companies, such as Cloudera and Hortonworks, sold
packaged versions of the
open source projects to address that market.
Even with these packaged solutions, a major challenge for companies adopting
Hadoop is managing the required data center infrastructure.
Clusters must be
able to scale to hundreds or even thousands of nodes for certain jobs. They
must be available when needed but may be idle for periods as well. Those
requirements make it difficult to find the right balance between economics and
the availability of the cluster infrastructure.
The cost of storage also becomes an issue as organizations
struggle to keep
pace with the amount of data they need to store. Tiering of storage to support
production, backups, archives, and so forth is crucial to contain costs.
The emergence of public clouds in 2010 promised to address these challenges.
With flexible compute
and reliable low-cost storage, they became an attractive
option for building one’s own data clusters.
05
The Evolution of Big Data and the Future of the Data Platform
