How Companies Should Approach Data Engineering - Infostrux

Kima: Focus is something we discuss a lot in my new business. We decided to focus on data integration – centralizing the data, modelling and preparing it for reporting. There’s a term that the industry is starting to use, which is data engineering. It’s predicated on the idea that there is another technology layer, the middle layer, where this work happens. It doesn’t happen in the BI reporting tool, it doesn’t happen in the dashboarding level, it happens somewhere else. What is your recommendation for people trying to implement a solution today – should they start from the top of the stack, choose a reporting analytics platform, and work their way down? Or should they focus on the middle layer – the engineering side, which is to centralize all the data together, clean it, implement security around it, and so on? How would you approach it?

Mark: I think there’s a lot of cultural factors inside of organizations that need to be put in place in order to execute on these analytic BI strategies. I think organizations struggle a little bit with this. In the start-up ecosystem when we pick start-ups as investors, if you take a VC or an individual investor, the cream rises to the top because there’s a diligence process – the best companies win, the best founders, the best ideas, and so on. This is what gets funded.

The challenge with a lot of BI projects, if you look at them like a start up inside of a big company, it’s like, ‘Hey, we’re going to launch this initiative to build a data warehouse inside of the organization.’ There are some key things to consider – do you have the skills and the right people doing it? Do you have the team that’s going to execute on the project, and have they been provided the training and the knowledge that they brought in the right people to execute?

It’s a little like diet and exercise – 80% is what you eat, not how much you exercise. In a lot of cases in the analytic world, it’s actually about the people in the culture and setting up the right objectives, and putting the right teams together.

Another important question to ask is: are you solving the right problem? This is a classic challenge with data – people are so focused on buying technology, mashing data together, and making important business decisions. It still seems like it’s a bit of the wild west.

A better approach would be to start with the question: what decisions do we want to make? Let’s sit down and think deeply about the problems that we’re trying to solve, and then from that, execute a process to decide how you’re going to do that. It starts with the data. You have to understand what data is available to you, what problems you want to solve, and do you have the right team and resources to execute.

I’ve talked to many enterprises who want to do something, which then leads to a data exploration exercise. Only then they realize they don’t even have that data. They can’t actually solve that problem because they don’t have access to the data they need. They’re not even creating that data inside the organization, so they need to first acquire that data. If they intend to buy it, then then need to source it and figure out who has the data they are looking for.

So a lot of it is sitting down, setting the right goals and objectives, then putting the right teams on these projects in order to be successful. Obviously, there’s also an evaluation of what tooling you want to use from an analytic perspective, whether you’re going to use Snowflake or some other tool. If you have a sound process in place for accessing the data, choosing the technology becomes easier because you have clarity, and that clarity will then allow you to rule out what you don’t need. You may not need AI, or predictive because you’re not trying to do predictive work, so you don’t need to run around analyzing predictive technology.

So you can eliminate what you don’t need and focus on narrowing it down to maybe 4 or 5 vendors. Then you can ask, which one is the best for us?

Kima: I agree with your answer. The technology stack matters, but you have to start with the why first, and what are you trying to accomplish first.