Srinivas Reddy’s Weblog

My Tech Rants

Next generation real time system design

leave a comment »

Having recently read a research paper and attended a related session on how “In-Memory column Database” can revolutionize Enterprise application development (the paper is a must read for anyone in Application development domain), I would like to share my views on it.  

By challenging the need for having different data models for transactional and analytical applications (i.e. OLTP & OLAP) and offering a new data modeling paradigm which would better leverage the emerging hardware trends (ie multi-core processors and huge main memory availability) provides insights into how next generation application infrastructure can look like. The paper proposes to organize data in columns and use column based operations then the traditional row based storage and operations for data manipulation, as this would better use the hardware capabilities and meet the application requirements. While ‘Storage’ is considered as the bottleneck in the current infrastructure it will be interesting to see how it can be better leveraged as is done with processors and main memory. Products such as ‘hp oracle exadata storage server’ may be showing the way to go here.

If one looks under the covers for most leading SaaS applications (be it social networking sites as Facebook, CRM applications like Salesforce or auctions from E-bay, MapReduce app from Google), their “secret sauce” is the data modelling that would allow them to scale at huge work loads (Internet scale) with low TCO. While lot of research work is happening in design of Very large databases, its interesting to note that domain specific custom solutions are being developed by the industry (the proposal in this paper is one of them!) to address specific use cases.

While its clear a new data model abstraction will be needed to design next generation real time systems, the jury is still out there of how “parallel computing”  can sufficiently address non functional requirements such a Development productivity, Reliability, Availability and Extensibility. What is clear is that the whole software infrastructure stack (Operating system, databases and application servers) needs to be redesigned. Apple has shown with iPhone what is possible when the whole stack is redesigned, we can expect similar domain specific innovations on the server side as well. The consolidation we are seeing in the IT industry are a part of this evolving next generation IT stack landscape. While its difficult to predict which vendors will actually survive the transition, for the end-users we can only see benefits from next generation real time systems.

Written by srinivasreddy

July 24, 2009 at 7:19 am

Posted in 1

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: