Srinivas Reddy’s Weblog

My Tech Rants

Emerging Information Architectures

with one comment

Having just read the book on “Beautiful Data” I would like to share some of my insights/learning’s from it. The book provides examples of elegant data solutions from different domains (including space, music, medical research, web apps, government data,..). A common pattern across domains is the need for smart handling of large data sets in the different stages of information processing (be it for data collection, storage, processing, visualization or collaboration around data). While ‘beauty’ (be it in data, code, art, person..) lies in the eye of the beholder, there can be some common metrics for it.

When we deal with web scale data ‘discoverability’ of information is key. While ‘web search’ provides a lot of value today what we really need is to enable ‘data find data’.  I like the differentiation in the book between ‘federated search’ and ‘federated fetch’. The latter needs adaptive systems that can discover new data correlations based on user context and new data collected. In the current mobile age of ‘iPads’ and ‘smart phones’ we have the ‘client hardware’ to collect/process huge amounts of data. The challenge is to redesign the server side architecture for real time processing of massive web data that can keep the mobile user engaged in real time. While ‘NoSQL’ (not only SQL!) design is established in large consumer web applications (be it facebook, twitter, google…) its usage in enterprise applications is still at early stage. With the growing popularity of ‘cloud solutions’ we can expect this to change for the enterprises as well. In this context developments in ‘In-Memory Databases’ and how they leverage modern hardware (with cheap main memory, disk will be used only as a secondary storage /backup device) are significant. While federated search relies on ‘Context-less directories’ the move to ‘semantically reconciled and relationship-aware directories’ that can make use of ‘persistent context’ is the way forward.

A transparent way to work with structured and unstructured/semi-structured data is a problem being worked on for some time now. With massive new data being added to the web daily (be it due to social networks, location based apps, sensor networks,…) the need for a information platform to smartly handle this and the emerging role of ‘Data Scientist’ become obvious. The paper on “From Databases to Dataspaces: A new abstraction for Information Management” is worth a read.

Trends in enabling data portability in real time need notice.  While most web apps increasingly expose a ‘REST’ interface, emerging real time notification systems (‘Comet’, ‘Web Hooks’, ‘PubSubHubbub’..) try to address real time data needs. Basically all try to add some state to the stateless HTTP request/response model to make the client interactions richer/more efficient. The trend is to move away from a ‘polling’ based model to ‘event’ based model where scaling bottlenecks exist. The support in HTML 5 for a event model and ‘web sockets’ is a significant new development.

‘Data quality’ is the major hurdle to beautiful data. While many smart algorithms maybe used to work around it, ‘crowd sourcing’ data and enabling collaboration around it is here to stay. While collaboration can be used to share and create new insights around data, information platform needs to support this in a scalable & secure way. We can expect a lot of action in the future around exposing the beauty hidden in the data and creating some real value in the process.


Written by srinivasreddy

August 21, 2010 at 3:28 am

Posted in Uncategorized

One Response

Subscribe to comments with RSS.

  1. […] to share some of my thoughts here. I had blogged in the past about some of these trends here and here, they provide a good context to think about potential applications for the emerging technology […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: