Among the most significant challenges of Business Intelligence is to keep track on what, when, how and who of Data Integration. Unlike Data Warehouse and OLAP Server , data-integration is not 100% achieved through configuring data-integration tools. The reaons that you cannot rely only upon the DI tools to do the documentation are:
-
Source Systems Structure documentation- You DI system may be able to link-up to the structure of Data-Base of the source system, but the underlying programs which impact the data may not be included. You need to document the areas of the source system (Data-base, Busniess Rules impacting the data, Data- processing windows..) which impact your data integration.
-
Complex Routines- For complex integrations, you have to go beyond the sheer configuration ‘click and choose’ capabilities, and write programs to do the extraction and transformation. You need to maintain the functional specs of those programs, as finally business also has to sign-off on those transformation rules.
-
The risks and gaps left in DI- You may not be able to achieve 100% perfection in DI (say Extraction and Transformation as an example) and would have lost some data due to it being dirty or incomplete. You also might have done the cleansing and enrichment of historical data. One needs to keep track to enable the future investigation, for Audit reviews and also to be able to explain the mis-match between the reports from production systems and from BI platform.
More reasons and further details on this post can be seen at Documenting Data Integration in my portal www.bipminstitute.com .
