Data Integration Series part 1 : Data Replication/Synchronization

Whenever we come across terms " Data Replication " & " Data Synchronization" , our brains start painting pictures of different databases, their inter-connections, data volume / frequency etc. This is a very interesting field as one can feel the data flowing in organization. Its more or less setting up and maintaining nerves/veins system of any organization. Personally , I have been fascinated by Data integration techniques since start of this century when I was responsible for managing multiple DBMS in different locations. 

Indeed these terms are around for quite some time, however with time and growing business and IT demands, there are lot many changes happening. In this blog (which is the first one in the data Integration series) lets try to understand what is the meaning of these terms, approach, usage/use cases , trend and important consideration etc.

Definition/s:

Data Replication : This is the process of sharing information so as to ensure consistency between redundant resources, to improve reliability, fault-tolerance, or accessibility.( Source : Wiki). The above definition sounds very generic and almost complete as this covers most of important aspects of data Integration. If we add the direction part to it then I believe this becomes complete. When I say direction , it is usually data source to destination/target i.e. one way only. We will talk more about direction in Data Synchronization . One caution , Like others technology area, this too is evolving fast so you might come across numerous definition versions.

Data Synchronization: This is the process of establishing consistency among data from a source to a target data storage and vice versa and the continuous harmonization of the data over time( Wiki). This is mere extension of Data Replication or can also be called  Advanced Data Replication. To understand better, just visualize, data replication happening between resource A to B and B to A so that both the data sources A & B are synchronized.

Implementation Approach/s: There are are various ways to implement Data replication. I feel these fall under following two categories

1. Direct approach : Using Data base Tables directly by using SQL or tools
2. Indirect approach :Using logs created by RDBMS in real time ( Change Data capture).

Both of above have pros and cons depending on the requirements. However the recommended approach is CDC(change data capture) since its in-obtrusive as this doesn't chock the Database server performance.Tools implementing CDC reads the logs generated by DBMS system/s and not on tables so no additional I/Os operations are carried out by DBMS system.

Usecase : Usecases of Data replication are for DW/BI and non-DW/BI purposes. As the name suggests the first one is for integrating DWH and for BI ( operational reporting, dashboards etc). The non DW/BI falls under application integration category e.g. 360 degree view of customer data. Also for another entities like product, reference data this is widely used.   

Trends:
External drivers like Increasing competition, shrinking economy has put organizations to strategize in terms of enabling Faster decision making, cost reduction and higher availability. So we can very well see initiatives are coming from IT as well as Business to meet company strategic vision. 

As per TDWI survey 17% are using real time DW functionality and more than 90% are committed to use in coming 2-3 years.So one of the biggest trend in realtime Datawarehousing and analytics. There are many many cases in BI/DW area which gets impacted by this , some of these are Operational BI, on Demand management Dashboard, Alerts and Notifications etc.

For non BI/DW, the biggest following trend is single view of Customer data integration, Product data , reference data and some other data entities.  And in order to have in-obstrusive and scalable implementations , organizations are opting for CDC ( Change Data Capture ) methodologies. Here comes Data Synchronization into picture because multiple applications / architecture are to be accommodated. 

Points to take care:

1. Understand 3Vs** of data ( Volume, Variety and Velocity ) requirements. Volume is about size of data. Variety is about the data type ( structured, un structured etc) and velocity is about frequency ( Batch, realtime etc).
2. Choose the replication tool with advanced Capabilities in order to cope with scalability by handling heterogeneous sources, conflicts aroused during Data Synchronization , alerts and Notifications etc
3. Document the interface specification( source and Target) clearly and keep it updated. This is mandatory as any small change in any source can create havoc.
4. Real time integration brings technical challenges and complexities, however also provides tons of opportunities for Business Analyst to see different dimension of data which was not experienced before.
5. Use the services of Data integration specialist is highly recommended. 

Resources :

1. TDWI webinar ' Data replication for DWH and BI ' by Phillip Russom
2. Wikipedia for Definitions

regards , Ajay

Legends
**  3Vs were coined by Philip Russom of TDWI in his Big data analytics research, however I think these beautiful terms can also be applied to any integration techniques

Project/Program Management Extra Steps series, Part 1

The objective of this series of blogs to talk about the extra step / out of box thinking /common sense/extra cheese topping used by Project, Program, Portfolio and PMO to make the dramatic change in situation. I would be taking up real life scenarios either experienced by myself or as I used as consultation to other programs.

Today being the first part of this series and I am taking up very interesting scenario which falls under Communication Management. We all know this is one of the most critical area as when we talk of IT projects, its mostly interactions with human beings.

Background : A large SI is executing projects for a APAC customer. The account is in category of 3P ( triple platinum ) indicating direct monitoring by Top management. Multiple projects(support, development, enhancement) are being executed and team size goes in many hundreds too.Teams comprising of diverse cultures and are operating from different geographies. There are numerous PMs/Prgm M governed/mentored by a team of PMO which is central to engagement. The relationship with client and vendor swings up and down. Also most of the team members are used to work under fire mode.

Scenario : Every week end, PMO collates a status report for top management. Delivery timings , content and quality of report is quite critical to this engagement. There are frequent audits carried internally by Client and any deviation with actual to reported status is treated a severe compliance issue and can lead to legal action against vendor. For collation, the practice is that PMO sends out mails to find out status from respective PMs. Last week one of the PM didn't not receive the mail because the PMO mistakenly used the old id ( email id for this resource was changed due to some issue).There was couple of follow up emails( mentioning very clearly that no response means , no progress during last month) on that, but these mails too were directed to incorrect ID. Please note PM is not at all aware of this as he is not receiving any communication.

Analysis : PMO is frustrated that there has been no response even after escalation and finally concluded that there was no progress during last month. On the other side PM is too busy with project and has taken for guaranteed that no communication from PMO means no status report this week. This resulted in a gap in reporting as project couldnot communicate a critical issue to Top management as well as it missed out to provide details of deliverables made in reporting week. Next week unfortunately the project was selected for client Audit and this gap surfaced out and loud. This made vendor situation very tough as it there were series of apologies. 

I believe these kind of situations are very common in large engagements(which requires lots of collaboration) or during the times things are on fine(the priorities are different). In above situation , seems like the communication plan was not updated, also not planned well during planning phase.

My 2 cents : Any mature Project management professional can propose numerous solutions  like to have updated communication plan, process for making sure plans are updated and communicated, strict SLA behind every official communication, central project repository , automation of status report creation etc etc. Above recomendations are  must from process point of view, however one can again caught up in situation like this. 

Now lets talk about the extra step , I believe, just a single call from PMO to PM could have averted this situation. These kind of extra steps requires very less physical effort however requires a big change of attitude. So guys, whenever responses are not as per expectations, dont just stick to what age old good process says, simply pick up the phone/ visit the respective person. This will surely make life easier and act as stepping stone for project success. 


regards, Ajay