In all businesses, a new business requirement needs a new IT system to support it, and it is added to the existing IT infrastructure.  This is especially true of the move over the last couple of years to e-commerce and remote access.  As a result, one thing that is common across all IT installations is that over time they create a mixture of legacy and current systems.

What Causes Silos?

It is a common cause that business systems are in a continual process of evolution but need to remain operational to support the business.  The continual modifications to the IT Infrastructure have two principal effects, the creation of islands of technology, otherwise known as technology silos, and new applications systems that need new dedicated databases, creating multiple databases containing essentially the same information, otherwise known as data silos.

The Hidden Costs of Silos

Here are five common hidden costs and effects of silos on organizations:

1. Ineffective and slow decision-making processes, since data, sometimes different, is pulled from different databases.

2. Difficulties in establishing collaborative processes in an organisation, because each doesn’t trust the information provided by the other.

3. Cost escalation due to an incompatible legacy IT and applications infrastructure.

4. Poor data quality, which in addition to the effects set out in 1 above, makes it difficult to trust and use data analytics.

5. Reputational damage caused by customers experiencing a poor experience when interacting with the organisation.

To focus on one driver of hidden costs – data silos.

Data Silos

Data silos happen when the same data is held in different databases and the databases are not linked to ensure that the data is consistent across all the databases.   As a result, the information in each database can be different or incomplete, or just plain wrong.
Nowadays, more and more companies are data-driven.  They use Big Data databases, machine learning and artificial intelligence to make business decisions at all levels of the organisation.  This brings the question of data ownership into sharp focus and highlights the problems data silos bring.

Working with data silos is frustrating and leads to decreased productivity and even bad business decisions.  Data Silos bring increased costs to a business, not necessarily direct financial ones:

1. Reduced Productivity

   If staff need to spend time finding data or checking its correctness, the longer they will take to make a business decision from supportable data.  The bigger the team, the greater the cost.

2. Uncompleted projects

   If staff cannot find data, or must make guesses,  they cannot complete accurate data models. This again can lead to poor business decisions based on incomplete or incorrect data

3. Incorrect Data Models

   Discrepancies and differences between what could be the same data item held in different databases can lead staff to make incorrect decisions about data based on a wrong assumption about what the data means.  This again can mean poor or incorrect decisions based on those assumptions.

Data Silo Prevention and Resolution

The first thing to understand is that data silos are not just an IT issue. They have arisen because of business changes, sometimes introduced in haste to meet changes in the business environment. As a result,  they will not be removed or resolved without management action from the top level.
The first step is to recognise that turf wars over data need to stop.  Data ownership needs to be clarified. A rule in data architecture design is the First Normal Form or “FNF” rule.  It states that wherever possible, data is held only once and made accessible through a small number of common access keys. One example is that staff details are held in a single staff member profile accessed by a staff number, with one company department, say HR  given ownership to it.

One technique of implementing FNF is to remove data silos through the integration of databases.  This is the process of having applications work together.  This can be done automatically through data replication, indirect addressing, or at worst manually.

Data replication is where one database is linked to one or more other databases so that when the master database changes, the change is immediately reflected in the secondary database(s).   Linkage can be at item or record level.  The linkage can be made through the intrinsic capabilities of database architectures such as Oracle,  or in simple programmed interactions for a small number of fixed replications.  However, it is better to use middleware for complex integrations.

A second technique is indirect addressing, where a data item contains a reference to a field in another database.  Not all databases support this data type.

Finally, manual replication.  This is not recommended. It takes time, is prone to error, and might be forgotten about entirely.

One common technique is to create dedicated databases for BI and data analytics refreshed automatically from production databases. Data silos are a natural part of business development but can be overcome.