Business Intelligence

"... continuous integration (CI) implements continuous processes of applying quality control — small pieces of effort, applied frequently" - http://en.wikipedia.org/wiki/Continuous_integration

In my blog on Test-driven development I discuss the benefits of writing automated unit tests for business intelligence systems. If you are unfamiliar with the concept of unit tests, please read that blog first or one of the many articles on-line or on Wikipedia on the subject.

Whilst I may have been one of the first proponents of Agile Business Intelligence, I am the first to admit that I have been rather late jumping on the social media bandwagon. However I am firmly convinced that social media such as twitter has a huge role to play as a source of data for business intelligence and data warehousing systems.

Without further ado, I would like to discuss 4 different types of information that can be extracted from twitter to add value to your BI system.

It has been my experience that testing is always overlooked or poorly done within business intelligence environments. The consequence of this is inefficient designs, significant data integrity and validation issues, as well as defects within ETL and reporting scripts. This, in turn, often leads to expensive consultants who want to start the whole process from the start again. What I want to propose is the use of an agile quality assurance method, called Test-driven development, in the business intelligence context.

Proper analysis and integration of foreign data sources into a Business Intelligence system can be a time consuming and complicated process. For each new data source I always recommend that a brief business case, made up of the following 11 questions, be put together. This helps prioritise the ETLing and identify problem areas for each source prior to beginning the process.

One of the more common questions I get asked is “What is your ideal business intelligence team?”. Though my answer is dependent on specific organisational requirements, there are some commonalities and generalisations I can make.

I would start by saying that a good BI team consists of both technical and business people. By utilising business and technical experts working together, outcomes can generally be met sooner and more accurately than if the teams were separate.

To often I have seen organisations build a data warehouse and populate it with data from 2-3 corporate databases. Is this valuable, probably, but could we improve the value by combining it with data from outside the organisation, definitely. Consider what a data warehouse actually is – I would describe it as “an accessible repository of valuable data from disparate sources”. The keyword here is disparate. So before reading on, have a think about where you would usefully extract data from.