Apr 25 2024
/
Post Detail
Data lake and Datawarehouse technologies have been prevalent in market for more than two decades+ now. It all started with Hadoop (HDFS) and Apache spark project somewhere in year 2000.
What did it really offer….
- It helped in consolidating (dumping ground) all data at one place.
- Transform or load the data into various warehouses or Godowns (which no one likes to visit) based on business needs
- Use it to generate intelligence, for businesses to act and take informed decisions (hoping the data is accurate and complete)
- It also helped in training data for AI by running MLops to build “feature store” to predict, find anomalies or build recommendation engines.
It was all good until ….
The need of business speed was slow. Where decisions could be taken in weeks or even months.
Where business could afford to wait for a day or two to generate report (I was taking to a bank CIO and he said and I quote “it takes 14 hours for them to generate an compliance report that needs to be sent to regulator”)
Where products MVP were delivered in months or in some cases years (remember those SAP implementations…)
Current Architecture:
But things changed….
- Mobile apps, 5G, low cost of data specially in India with 1.4 billion population, DigiFi-ing economy — customer expectations have skyrocketed
- Everyone wants things in splits of seconds or minutes whether you want to book a cab, order a dosa, fund transfer or some of the extreme cases like book an apartment completely on a digital platform
- On top of all these — Generative AI has created a huge wave of expectations from IT to deliver information from systems not by coding but by inputting simple English human readable language
To deliver on above businesses needs the architecture of future which will address two things:
- Help businesses to take decisions in real-time within seconds if not milliseconds
- Provide instant access to data to generate reports, build AI models whether it is for GenAI apps or traditional AI
Modern Architecture:
