Occam’s Razor, a well-known principle stating the simplest solution for a problem is often the best, has been utilized by businesses for decades to solve their most significant and complicated problems. The integration of data analytics – the pursuit of extracting meaning from raw data – into an enterprise’s decision-making process should aid in this effort. Yet, as organizations ramp up their data analytics capabilities, black box algorithms and highly convoluted predictions have been favored over concise and actionable insights.
The process of developing an analytical expression to drive successful business outcomes is very difficult. Traditionally, individuals with high-level degrees (in a STEM field) and a strong knowledge of technical tools (MATLAB, SAS, Stata, etc.) and programming languages (Python or R) spend weeks solving problems with data they spent months collecting, aggregating and transforming. The level of effort needed to implement this approach in big businesses is staggering but, seemingly, necessary in order to extract value from the vast amounts of data large companies are paying millions of dollars a year to store. The output generated through this approach, which ranges from simple linear models to black box machine learning algorithms (neural networks, SVMs, etc.), provides a prediction of what will happen but does not provide increased understanding or insights into decisive actions that can drive business results. Predictive accuracy became the most important metric in analytics because being right was prioritized over providing understanding.
The time has come for this to change. Companies must harness simplicity in order to generate significant business value moving forward. Rather than simply learning what will happen (sales tomorrow will be x), companies need to also understand why it will happen (marketing takes two weeks to influence sales, weather impacts in-person purchases, etc.).
The only way to do the latter is to build simple, interpretable (parsimonious!) models. Simple models deliver results that are as accurate as black box approaches but impact the business much more profoundly. It is time for companies to stop hiding behind their initial approach to predictive modeling and jump head first into the future of machine intelligence.