Cloud BI

Cloud , cloud every where .. Don't you think so ? In case not , you will see it soon . Atleast I am in clouds now a days :). 

Day before yesterday I was listening to Marc Benioff ( CEO of Sales force ) key note address during Dream Force 2011 and panel discussion with Eric Schmidt( Executive Chairman of Google). I was amazed to feel their energy ,passion and future vision. No doubt , the Cloud is the turning point of IT industry , the last one we had during Mainframes to client server transition. As things are evolving so there would be mixed responses during this phase. Some will say , this is a bubble and it will burst soon. Who cares , though I am in clouds , but i can see very far :). Post that I listened to some old keynotes of Steve Jobs, Mark Zuckerberg , Larry Ellison and concluded that all are pointing in one direction.

So..I am very excited to be part of this transition and see how quick things will be adopted. They say , Every disaster has an opportunity hidden. To relate this I would say the current economic situations are acting as catalyst to strongly push cloud market. Also I am equally excited to see what changes this transition will bring to BI industry, how BI will fit in, how the future BI architecture look like?? 

There are numerous advantages Cloud offers, the major are Time to market and economic ( initial cost as well as Total cost of ownership). The other ones are better performance ( subject to architecture) , scalability , almost no upgrades overheads and so on..

Now its lot about the trend and benefits, Its time to have some basic understanding of Cloud computing? Wikipedia definition goes like this:

Cloud computing is the delivery of computing as a service rather than a product, whereby shared resources, software, and information are provided to computers and other devices as a utility (like the electricity grid) over a network (typically the Internet).

In simple words its just moving computational power from on premise to cloud in Internet and accessing this computing power by web browser/web services etc.  Vendors are offering three kind of flavors

1. SaaS : Software as a service . e.g. Sales Force
2. PaaS : Platform as a service. e.g. Microstrategy cloud
3. IaaS : Infrastructure as a service. e.g. Amazon EC2

Talking about BI in cloud , I feel most of the would fall in PaaS(Platform as a service) where vendors provides world class hardware and Software strictly in line with BI requirements, be it DWH appliance, Data integration tools, Reporting layer etc. Also Companies can immediately start developing /deploying on these world class environment offering wealth of tools. E.g. Microstrategy Cloud offers Microstrategy suite on reporting layer, Netezza as DWH appliance and Informatica for Data Integration.

IaaS ( generic cloud ) is also be used by some of organizations , however this might throw additional tasks like platform setting up, tuning etc to inhouse team. Amazon EC2 , database.com , Azure are leading examples in this category.

We can also see some companies opting for third party BI applications on SaaS. e.g. Sales force appexchange offers thousands of applications and hundreds of consulting partners.

Effectively we can see BI foot print would be there on all three flavours.  
 
Some Points to take care

1. Proof of concept is highly recommended prior to overall change. This will help to set expectations, benchmark etc.Lot many vendors offer free trial, FYI.
2. Please check the data policy of your organization as some of the organization would not like to have the data stored outside the company firewall.
3. Performance can take a hit , If cloud architecture involves accessing data from on premise. 
4. Ensure adequate Security measures for data access and storage in multi tenancy environment as well as cloud accessing on premise data. Do thorough due diligence with while selecting a vendor and during POC phase.

So Cloud is a way to go .. Guys gear up.. Happy clouding:)..

regards, Ajay

Data Integration Series part 1 : Data Replication/Synchronization

Whenever we come across terms " Data Replication " & " Data Synchronization" , our brains start painting pictures of different databases, their inter-connections, data volume / frequency etc. This is a very interesting field as one can feel the data flowing in organization. Its more or less setting up and maintaining nerves/veins system of any organization. Personally , I have been fascinated by Data integration techniques since start of this century when I was responsible for managing multiple DBMS in different locations. 

Indeed these terms are around for quite some time, however with time and growing business and IT demands, there are lot many changes happening. In this blog (which is the first one in the data Integration series) lets try to understand what is the meaning of these terms, approach, usage/use cases , trend and important consideration etc.

Definition/s:

Data Replication : This is the process of sharing information so as to ensure consistency between redundant resources, to improve reliability, fault-tolerance, or accessibility.( Source : Wiki). The above definition sounds very generic and almost complete as this covers most of important aspects of data Integration. If we add the direction part to it then I believe this becomes complete. When I say direction , it is usually data source to destination/target i.e. one way only. We will talk more about direction in Data Synchronization . One caution , Like others technology area, this too is evolving fast so you might come across numerous definition versions.

Data Synchronization: This is the process of establishing consistency among data from a source to a target data storage and vice versa and the continuous harmonization of the data over time( Wiki). This is mere extension of Data Replication or can also be called  Advanced Data Replication. To understand better, just visualize, data replication happening between resource A to B and B to A so that both the data sources A & B are synchronized.

Implementation Approach/s: There are are various ways to implement Data replication. I feel these fall under following two categories

1. Direct approach : Using Data base Tables directly by using SQL or tools
2. Indirect approach :Using logs created by RDBMS in real time ( Change Data capture).

Both of above have pros and cons depending on the requirements. However the recommended approach is CDC(change data capture) since its in-obtrusive as this doesn't chock the Database server performance.Tools implementing CDC reads the logs generated by DBMS system/s and not on tables so no additional I/Os operations are carried out by DBMS system.

Usecase : Usecases of Data replication are for DW/BI and non-DW/BI purposes. As the name suggests the first one is for integrating DWH and for BI ( operational reporting, dashboards etc). The non DW/BI falls under application integration category e.g. 360 degree view of customer data. Also for another entities like product, reference data this is widely used.   

Trends:
External drivers like Increasing competition, shrinking economy has put organizations to strategize in terms of enabling Faster decision making, cost reduction and higher availability. So we can very well see initiatives are coming from IT as well as Business to meet company strategic vision. 

As per TDWI survey 17% are using real time DW functionality and more than 90% are committed to use in coming 2-3 years.So one of the biggest trend in realtime Datawarehousing and analytics. There are many many cases in BI/DW area which gets impacted by this , some of these are Operational BI, on Demand management Dashboard, Alerts and Notifications etc.

For non BI/DW, the biggest following trend is single view of Customer data integration, Product data , reference data and some other data entities.  And in order to have in-obstrusive and scalable implementations , organizations are opting for CDC ( Change Data Capture ) methodologies. Here comes Data Synchronization into picture because multiple applications / architecture are to be accommodated. 

Points to take care:

1. Understand 3Vs** of data ( Volume, Variety and Velocity ) requirements. Volume is about size of data. Variety is about the data type ( structured, un structured etc) and velocity is about frequency ( Batch, realtime etc).
2. Choose the replication tool with advanced Capabilities in order to cope with scalability by handling heterogeneous sources, conflicts aroused during Data Synchronization , alerts and Notifications etc
3. Document the interface specification( source and Target) clearly and keep it updated. This is mandatory as any small change in any source can create havoc.
4. Real time integration brings technical challenges and complexities, however also provides tons of opportunities for Business Analyst to see different dimension of data which was not experienced before.
5. Use the services of Data integration specialist is highly recommended. 

Resources :

1. TDWI webinar ' Data replication for DWH and BI ' by Phillip Russom
2. Wikipedia for Definitions

regards , Ajay

Legends
**  3Vs were coined by Philip Russom of TDWI in his Big data analytics research, however I think these beautiful terms can also be applied to any integration techniques

Project/Program Management Extra Steps series, Part 1

The objective of this series of blogs to talk about the extra step / out of box thinking /common sense/extra cheese topping used by Project, Program, Portfolio and PMO to make the dramatic change in situation. I would be taking up real life scenarios either experienced by myself or as I used as consultation to other programs.

Today being the first part of this series and I am taking up very interesting scenario which falls under Communication Management. We all know this is one of the most critical area as when we talk of IT projects, its mostly interactions with human beings.

Background : A large SI is executing projects for a APAC customer. The account is in category of 3P ( triple platinum ) indicating direct monitoring by Top management. Multiple projects(support, development, enhancement) are being executed and team size goes in many hundreds too.Teams comprising of diverse cultures and are operating from different geographies. There are numerous PMs/Prgm M governed/mentored by a team of PMO which is central to engagement. The relationship with client and vendor swings up and down. Also most of the team members are used to work under fire mode.

Scenario : Every week end, PMO collates a status report for top management. Delivery timings , content and quality of report is quite critical to this engagement. There are frequent audits carried internally by Client and any deviation with actual to reported status is treated a severe compliance issue and can lead to legal action against vendor. For collation, the practice is that PMO sends out mails to find out status from respective PMs. Last week one of the PM didn't not receive the mail because the PMO mistakenly used the old id ( email id for this resource was changed due to some issue).There was couple of follow up emails( mentioning very clearly that no response means , no progress during last month) on that, but these mails too were directed to incorrect ID. Please note PM is not at all aware of this as he is not receiving any communication.

Analysis : PMO is frustrated that there has been no response even after escalation and finally concluded that there was no progress during last month. On the other side PM is too busy with project and has taken for guaranteed that no communication from PMO means no status report this week. This resulted in a gap in reporting as project couldnot communicate a critical issue to Top management as well as it missed out to provide details of deliverables made in reporting week. Next week unfortunately the project was selected for client Audit and this gap surfaced out and loud. This made vendor situation very tough as it there were series of apologies. 

I believe these kind of situations are very common in large engagements(which requires lots of collaboration) or during the times things are on fine(the priorities are different). In above situation , seems like the communication plan was not updated, also not planned well during planning phase.

My 2 cents : Any mature Project management professional can propose numerous solutions  like to have updated communication plan, process for making sure plans are updated and communicated, strict SLA behind every official communication, central project repository , automation of status report creation etc etc. Above recomendations are  must from process point of view, however one can again caught up in situation like this. 

Now lets talk about the extra step , I believe, just a single call from PMO to PM could have averted this situation. These kind of extra steps requires very less physical effort however requires a big change of attitude. So guys, whenever responses are not as per expectations, dont just stick to what age old good process says, simply pick up the phone/ visit the respective person. This will surely make life easier and act as stepping stone for project success. 


regards, Ajay

Understanding Social BI

Social BI: Topic sounds interesting, isn't ? Lets discover this a bit. The idea of this blog is to understand what Social BI is and how corporates can feast on its power.

The term social networking would certainly make you think the way you are connected and contributing to this rising market.Why it is rising , lets check out the numbers involved in this. Till last month Facebook had over 800 Million users( indeed a whopping figure) and Google+ crossed 50 Million mark in short time. These are leaders, however there are many more which are more regional. The best part is these are growing exponentially. Not to forget numerous countess twitters, blogs etc. As per Pew research on usage: about 83 % of youth for age group of 18 to 29 yrs which almost doubled in last 2-3 years and to me more interesting is 50-64 yrs bracket with a growing rate of over 1000 times. You can very well make out that the numbers involved are huge and power driven by these numbers would be enormous too. From another dimension , we can look this as mammoth CRM system, where data is maintained by users themselves.

Now, lets understand the usage of Social networking media from different perspective. Most of us use it for entertainment, collaboration, connection , sharing, learning and for many other things. Basically what is been happening is we say or follow something , somewhere and somehow. Please note, this in totality generates  huge data.  This means opportunities to know in terms of what is going on in society or what is the sentiment right now. Just imagine, if we are assisted with some powerful tool, then we can easily analyse this sentiment and have business/marketing strategy tuned, if we are part of corporate. Have personal strategy defined being an individual and have political strategy in case of Govt. The potential is huge , very huge. To provide other side of it , in recent times we have experienced several incidents/causes all around globe which were driven by social media and at many places where Social Networking acted as Saviour, specially during Japan earth quake crisis and Irene Hurricane. 

Now we know the source of this data, however tapping it has its own challenges. Firstly there is no direct access , secondly the data quality. There are various tools and solutions available which we will talk about in later section. Firstly lets see how organizations / individuals can feast on this.

Out of above ,lets see one usage from Business perspective with market leader ( Facebook) . I believe , any Global organization, can easily comment Its the Facebook where the customers are :). To unleash this power, corporate are using three following ways;

1. By creating Fan Pages
2. Doing Target Messaging 
3. Creating applications.

Fan pages build ongoing dialogs, target messages helps with limited target advertisement however both are very much limited. However the third option delivers rich interactivity in terms of bringing targeted , personalized and relevant content directly to consumers. For example,  just think of receiving discount coupons, loyalty perks, special offers and services from your favourite coffee brand.( I would certainly like to get some of these from Starbucks ;)). The steps involved are , users installs the facebook application and commits to share data(comprising of Demographics,behaviours, likes and friends etc) to application owner for various purpose as mentioned in agreement. Consumer intelligence is carried out on application users data .Organization do data mining , come up with marketing strategies/ campaigns and its go on and on.

Above looks great and simple. For sure, however everything comes with a cost tag. Firstly one needs to have proper tools to have access of data by using permission tokens and provision to enriching this from demographics perspective as  many of the users don't fill out complete information e.g. town name is mentioned without state and there can be many towns with same name  , the siblings are incorrect etc. There are many tools available in market , however please mind that these are still evolving. To name a few Microstrategy ( wisdom and Gateway ) , Antivia, Lyza , Panarama Necto etc. Secondly the organization is able to understand the market trend by doing proper data mining and align this with marketing startegy. Thirdly get the feedback and keep on improving ..

My recommendations:

1.  Social Media/Networking is evolving and it indeed offers tremendous potential for Corporates , Individuals, Govt. sector to carry out consumer analysis, sentiment analysis etc. BI professional needs to understand the influence on BI.
2. Data is huge and growing , however the Data quality is of concern.
3. Tools are evolving as the market is . Few big and small vendors are already making their presence felt . Many more Heavy weights vendors are coming in or planning to come in join this growing market. Have a close watch on upcoming tools.
4. Individuals need to understand the data share agreement with application provider.
   Hope you all enjoyed this.

I plan to cover another important aspect of Social Bi(Sentiment analysis) and text analytics some time later.. 

Can BI projects be offshored and delivered successfully?

Recently I was hearing to thoughts of Krish Krishnan , who is renowned industry in strategy architecture and implementation of high performance DWH solutions. It was quite an interesting session as we all know that many of the BI projects stalls due to numerous reasons.I would not repeat the whole talk as transcript as you can refer to same in TDWI webinar section( Managing risks of offshore Data warehousing). I would provide some gems and end it with my experience managing such like engagements.

Krish made a very important point that DWH program is special and is unlike other software projects, how,

1. Its an evolving program because of ever increasing demand of business requirements. None of enterprise who have implemented DWH /BI solution can confirm that the in place DWH system/s doesn't need any changes. All are experiencing some changes happening at some point of time, be it on Reporting layer or Integration layer.


2. Its a very complex Program as it involves complex technology , multiple stakeholders, multiple SMEs, multiple applications, and ongoing development. Bringing all under one roof is indeed challenging.

In order to have successful off-shoring there are various critical factors(people, process and organization) which need to be taken care by project owner. First two are obvious, however I was interested to find organization , and that surely made sense to me in order to transfer risk ( for data , multiple locations etc) and exploiting cost( blended cost) and talent and proven process advantages one needs to put this as important factor too. All the above points are based on work balance between offshore and onsite as any imbalance can lead to risk of ownership issue, communication issues availability of resources etc which can be devastating. I feel , Krish was very particular about management of communication & expectations effectively.

To summarize: I feel most of things are common Project management chores except for complexities of collaboration and evolution of DWH with in and outside enterprise/s. For sure there is no silver bullet which fits all . There is a need to be patient to achieve the right maturity levels. based on my experience ,I have following points to offer.

1. At onsite, the program should have Right set of people to understand and manage stakeholders expectations( technical, political, ego , cultural etc). And offshore owners must be imparted this very knowledge seamlessly else the program would not be in one world, rather two , simultaneously coexisting with each other which never meet.


2. Its best to have single Program Manager not like a usual onsite and offshore owners. I feel puts additional overhead and delays decision as both onsite and offshore reports to one who is managing multiple such like programs , and its obvious that there would be some conflicts.


3. Focus on mix of team at onsite , this is very important when it comes to manage global, co located project have diverse cultures.


4. If working on newer technologies, please plan accordingly as this could be a tipping point.


5. Try to retain core team members if not possible, insist to have effective Knowledge transfer. I feel usually 80% of the tacit knowledge is gone waste whenever there is turn around.


6. If possible , arrange for onsite visit for all key offshore members and vice versa. This needs to planned smartly.

Cheers!