A lot of people thought I was crazy for leaving one of the hottest, most innovative big data companies of the last 10 years, to join another start-up.
I graduated from UW-Madison with a PhD in databases, and worked for several years as a systems software engineer at Vertica, as deep down in the guts as you can imagine: C++, multi-threaded programming, distributed systems, process management. Towards the end of my stint there, I was leading the analytics team.
The analytics team was responsible for creating a whole slew of analytic plugins for the Vertica database engine. These plugins provided functionality like geospatial capabilities and data mining algorithms such as linear regression, SVM, etc. In the early stages, I spoke to several customers to get some feedback to guide development. The conversations usually went like this:
Me: “We’re thinking of building a library of data mining functions – things like linear regression and support vector machines – to provide predictive analytics. We were hoping to get your thoughts on which algorithms you’d find most useful.”
Customer: “Predictive analytics sounds wonderful! However, how do we tell what algorithms could be used for our business problems?”
Me: “That would be something your data scientist would know.”
Customer: “Our data-what?”
After a couple of conversations like that, we got better at targeting customers that had data scientists working for them, from whom we got the feedback we were looking for. However, this made me realize that even though tools like Vertica and Tableau have solved the problems of capturing, processing and visualizing huge quantities of data, predictive modeling is currently a very human-intensive activity. In addition, from what I can tell, data scientists are a pretty scarce resource!
Enter Nutonian. The first time I had a conversation with Michael Schmidt (the founder of Nutonian), I was very impressed with Eureqa’s ability to automatically build predictive models that are easily understandable by a non-data scientist, like me. The Eureqa core engine is able to automatically discover non-linear relationships in data: essentially a set of mathematical equations that hold true over the range of the data. I realized that this technology has the potential to really disrupt the market by making predictive analysis accessible to the masses. That’s when I decided to join Nutonian, so I could work on really exciting and impactful technology.
Enabling users without a math background to really understand equations would require some very innovative user interfaces and visualizations. It felt like a great opportunity to learn something new and build a product that can disrupt the market. Besides, there is something very satisfying about being able to visually show what you’ve built. This would be in stark contrast to my prior work at Vertica, which was deep in the core of an analytic database – it’s very difficult to demo a SQL prompt!
Stay tuned for next week, when I share some of the interesting projects we’ve been working on in the advanced analytics team.