Business Analysis & Data Profiling: Can’t Have One Without the Other

Peanut butter and jelly. Vanilla ice cream and apple pie. Love and marriage. Sonny and Cher. What do these have in common? Each individual component is pretty good on its own, yet the combination of both is far superior. The same can be said for business analysis and data profiling.

When gathering requirements or troubleshooting an issue, the traditional approach has been to conduct business analysis. This can include reviewing historical documentation, interviewing key stakeholders and subject matter experts, understanding the current processes, or looking through system screens. It can also involve a comparison with existing standards and best practices or identifying impacts from policies or regulations. Behind the scenes, code for the relevant applications can be reviewed. Perhaps there are architecture diagrams that show how everything fits together and communicates and there may even be a sandbox environment to test system behavior in specific situations. Once all the relevant information has been gathered, findings are socialized. Any or all of these are still valid activities for understanding the scope of the scenario at hand.

Data profiling takes this many steps further. Digging deeper into the systems and looking at the values in each database can reveal a number of things that may not be apparent from the process documentation or the user interface. A good data profiler will collect and interpret the existence (or non-existence) of attributes, as well as the range and frequency of each value, quantitatively matching the data in each database with purported sources. This will provide a view of patterns in the data that can be interpreted. It may uncover usage that is unexpected or contrary to stated business procedures. Audit history also can provide insight into who performed what updates when and sometimes why. This insight can be combined with the business knowledge to make sense of the results and identify relevant next steps.

Tag-teaming business analysis with data profiling has proven very effective at a financial services client. One work stream in the reference data program investigated business rules for how client data is distributed from the golden source to transaction processing systems, with a view to streamlining duplicates or removing expired rules. Initial conversations proved difficult at determining whether rules were still valid, and there were a few reasons for this. Some stakeholders expressed reluctance to make system changes without fully understanding the ramifications downstream. Many of the transaction processing systems pre-date the golden source system, yet subject matter expertise was often unavailable. Documentation on business process either did not exist or was out of date. Regulatory-related changes also had been implemented in some areas without updates to the business rules. The team then did two separate analyses: a time-series of account requests and a count of open accounts by the system. Once the data profiling was complete, the business analysis team was able to effectively target rules where there were few active accounts or little to no requests for new accounts within recent years. The audit data also helped identify business requestors to better target the interviews and pose more fruitful questions. As a result, the team was able to concurrently analyze two product areas at a time and reduce the time allotted for each audit.

Data profiling is an effective way to evaluate how systems are really being used. The contextual knowledge of how things currently work is just as critical for interpreting the results of the profiling as the data profiling itself. On their own, analysts can gain some knowledge of the current state, but combining business analysis and data profiling paints a better picture.

What business problem are you trying to solve? Have you identified the right people to interview? Does the data line up with the prescribed process?

Contributed by Jace Frey.