Why Data Engineering is So Important - Infostrux

On June 18th, Infostrux CEO, Goran Kimovski (Kima) sat down with S. Somasegar (Soma) to discuss Soma’s experience and lessons as a board member and early investor in Snowflake. Kima and Soma discuss the concept and value of the data cloud and why data exchange will be as important as data warehousing. As an active angel investor and seed investor, Soma describes what makes a great investment and what he looks for while investing in both enterprise and consumer segments.

Kima: I want to talk a little bit about data engineering. It’s an emerging discipline and the related agile approach for working with data that everybody seems to call DataOps these days. Do you have views on the importance of data engineering to the success of typical data analytics or data science projects?

Soma: This is what I call a healthy tension, and if approached properly can give a force-multiplier effect. Data analysts and data scientists are focused on building a model because they are trying to get to a particular outcome, and they want to get there as fast as possible.

The data engineer is all about building data pipelines, data ingestion, and data cleansing. They want to build an infrastructure that is going to enable them to operate on their data at scale in the most optimal way. 

The point of convergence between data science and data engineering is where organizations have the opportunity to really have a force-multiplied effect, or have a significant step function in terms of what they are able to do. But there is inherently a little bit of tension because the data scientist is thinking the data engineer is slowing them down. The data engineer is looking at the data scientists and thinking they are running too fast. They need to work together.

Think about the point of convergence where data engineers can help build a robust and scalable infrastructure for what organizations want to do with data. Every company I speak with says, we need more data engineers. It’s a highly sought-after commodity. 

The more data scientists and data engineers understand the value each other brings, they can work together collaboratively. You can do phenomenal things by unlocking the value in your data that you have in your enterprise or organization. 

Kima: We have a view that data engineering is about removing friction, enabling more experiments, enabling more access to data for data analysts and data architects. This allows them to work with the data more efficiently, build algorithms, and develop self-service models for the rest of the organization to access the data.