Cloud BI

Cloud , cloud every where .. Don't you think so ? In case not , you will see it soon . Atleast I am in clouds now a days :). 

Day before yesterday I was listening to Marc Benioff ( CEO of Sales force ) key note address during Dream Force 2011 and panel discussion with Eric Schmidt( Executive Chairman of Google). I was amazed to feel their energy ,passion and future vision. No doubt , the Cloud is the turning point of IT industry , the last one we had during Mainframes to client server transition. As things are evolving so there would be mixed responses during this phase. Some will say , this is a bubble and it will burst soon. Who cares , though I am in clouds , but i can see very far :). Post that I listened to some old keynotes of Steve Jobs, Mark Zuckerberg , Larry Ellison and concluded that all are pointing in one direction.

So..I am very excited to be part of this transition and see how quick things will be adopted. They say , Every disaster has an opportunity hidden. To relate this I would say the current economic situations are acting as catalyst to strongly push cloud market. Also I am equally excited to see what changes this transition will bring to BI industry, how BI will fit in, how the future BI architecture look like?? 

There are numerous advantages Cloud offers, the major are Time to market and economic ( initial cost as well as Total cost of ownership). The other ones are better performance ( subject to architecture) , scalability , almost no upgrades overheads and so on..

Now its lot about the trend and benefits, Its time to have some basic understanding of Cloud computing? Wikipedia definition goes like this:

Cloud computing is the delivery of computing as a service rather than a product, whereby shared resources, software, and information are provided to computers and other devices as a utility (like the electricity grid) over a network (typically the Internet).

In simple words its just moving computational power from on premise to cloud in Internet and accessing this computing power by web browser/web services etc.  Vendors are offering three kind of flavors

1. SaaS : Software as a service . e.g. Sales Force
2. PaaS : Platform as a service. e.g. Microstrategy cloud
3. IaaS : Infrastructure as a service. e.g. Amazon EC2

Talking about BI in cloud , I feel most of the would fall in PaaS(Platform as a service) where vendors provides world class hardware and Software strictly in line with BI requirements, be it DWH appliance, Data integration tools, Reporting layer etc. Also Companies can immediately start developing /deploying on these world class environment offering wealth of tools. E.g. Microstrategy Cloud offers Microstrategy suite on reporting layer, Netezza as DWH appliance and Informatica for Data Integration.

IaaS ( generic cloud ) is also be used by some of organizations , however this might throw additional tasks like platform setting up, tuning etc to inhouse team. Amazon EC2 , database.com , Azure are leading examples in this category.

We can also see some companies opting for third party BI applications on SaaS. e.g. Sales force appexchange offers thousands of applications and hundreds of consulting partners.

Effectively we can see BI foot print would be there on all three flavours.  
 
Some Points to take care

1. Proof of concept is highly recommended prior to overall change. This will help to set expectations, benchmark etc.Lot many vendors offer free trial, FYI.
2. Please check the data policy of your organization as some of the organization would not like to have the data stored outside the company firewall.
3. Performance can take a hit , If cloud architecture involves accessing data from on premise. 
4. Ensure adequate Security measures for data access and storage in multi tenancy environment as well as cloud accessing on premise data. Do thorough due diligence with while selecting a vendor and during POC phase.

So Cloud is a way to go .. Guys gear up.. Happy clouding:)..

regards, Ajay

Data Integration Series part 1 : Data Replication/Synchronization

Whenever we come across terms " Data Replication " & " Data Synchronization" , our brains start painting pictures of different databases, their inter-connections, data volume / frequency etc. This is a very interesting field as one can feel the data flowing in organization. Its more or less setting up and maintaining nerves/veins system of any organization. Personally , I have been fascinated by Data integration techniques since start of this century when I was responsible for managing multiple DBMS in different locations. 

Indeed these terms are around for quite some time, however with time and growing business and IT demands, there are lot many changes happening. In this blog (which is the first one in the data Integration series) lets try to understand what is the meaning of these terms, approach, usage/use cases , trend and important consideration etc.

Definition/s:

Data Replication : This is the process of sharing information so as to ensure consistency between redundant resources, to improve reliability, fault-tolerance, or accessibility.( Source : Wiki). The above definition sounds very generic and almost complete as this covers most of important aspects of data Integration. If we add the direction part to it then I believe this becomes complete. When I say direction , it is usually data source to destination/target i.e. one way only. We will talk more about direction in Data Synchronization . One caution , Like others technology area, this too is evolving fast so you might come across numerous definition versions.

Data Synchronization: This is the process of establishing consistency among data from a source to a target data storage and vice versa and the continuous harmonization of the data over time( Wiki). This is mere extension of Data Replication or can also be called  Advanced Data Replication. To understand better, just visualize, data replication happening between resource A to B and B to A so that both the data sources A & B are synchronized.

Implementation Approach/s: There are are various ways to implement Data replication. I feel these fall under following two categories

1. Direct approach : Using Data base Tables directly by using SQL or tools
2. Indirect approach :Using logs created by RDBMS in real time ( Change Data capture).

Both of above have pros and cons depending on the requirements. However the recommended approach is CDC(change data capture) since its in-obtrusive as this doesn't chock the Database server performance.Tools implementing CDC reads the logs generated by DBMS system/s and not on tables so no additional I/Os operations are carried out by DBMS system.

Usecase : Usecases of Data replication are for DW/BI and non-DW/BI purposes. As the name suggests the first one is for integrating DWH and for BI ( operational reporting, dashboards etc). The non DW/BI falls under application integration category e.g. 360 degree view of customer data. Also for another entities like product, reference data this is widely used.   

Trends:
External drivers like Increasing competition, shrinking economy has put organizations to strategize in terms of enabling Faster decision making, cost reduction and higher availability. So we can very well see initiatives are coming from IT as well as Business to meet company strategic vision. 

As per TDWI survey 17% are using real time DW functionality and more than 90% are committed to use in coming 2-3 years.So one of the biggest trend in realtime Datawarehousing and analytics. There are many many cases in BI/DW area which gets impacted by this , some of these are Operational BI, on Demand management Dashboard, Alerts and Notifications etc.

For non BI/DW, the biggest following trend is single view of Customer data integration, Product data , reference data and some other data entities.  And in order to have in-obstrusive and scalable implementations , organizations are opting for CDC ( Change Data Capture ) methodologies. Here comes Data Synchronization into picture because multiple applications / architecture are to be accommodated. 

Points to take care:

1. Understand 3Vs** of data ( Volume, Variety and Velocity ) requirements. Volume is about size of data. Variety is about the data type ( structured, un structured etc) and velocity is about frequency ( Batch, realtime etc).
2. Choose the replication tool with advanced Capabilities in order to cope with scalability by handling heterogeneous sources, conflicts aroused during Data Synchronization , alerts and Notifications etc
3. Document the interface specification( source and Target) clearly and keep it updated. This is mandatory as any small change in any source can create havoc.
4. Real time integration brings technical challenges and complexities, however also provides tons of opportunities for Business Analyst to see different dimension of data which was not experienced before.
5. Use the services of Data integration specialist is highly recommended. 

Resources :

1. TDWI webinar ' Data replication for DWH and BI ' by Phillip Russom
2. Wikipedia for Definitions

regards , Ajay

Legends
**  3Vs were coined by Philip Russom of TDWI in his Big data analytics research, however I think these beautiful terms can also be applied to any integration techniques