Are you a cloud engineer or cloud architect who has been pondering on entering the burgeoning data world, wondering if it’s right for you and how to go about the transition?
After being in cloud infrastructure for about 6 years, I decided that data science would be my next career iteration. It made sense — big data was the future. So, I enrolled in a big data program in Toronto at Ryerson University and started learning about everything data.
Initially, my plan was to stop at data science — but as I learned more about my love for data engineering, I could see that I’d be staying on for a bit longer. Now a few years later, I find myself in a data engineering role at Infostrux and loving it.
This article is about my experience and impressions of switching my career from cloud infrastructure to data engineering. My hope is to offer some insight to help you decide if data engineering is for you, and provide some tips to ease the transition if it is a path you decide to take.
Qualities of a Data Engineer
Learning on Your Own
I have asked myself: “Is there a way to learn this stuff alone?” You have probably taught yourself countless technologies if you are a typical DevSecOps engineer or architect. Why would data engineering be any different?
In a way, you would be suitable for saying that… on the other hand, just as you have undoubtedly discovered “undocumented features” while teaching yourself new stacks… you have come to realize that there is a difference between completing a demo installation of a new tool and rolling out a production-grade solution that thousands of users will use. A relentless love of learning and troubleshooting is key.
If this field is for you, you’ll find something so satisfying about wrangling projects through a structured process, which ultimately allows us to untangle the “data” spaghetti and make it worthwhile.
Humility
You won't be an expert overnight. It will take time, experience, constant learning, and application. It is important to accept a newfound entry-level status and constantly strive to better yourself with more and more experience.
You Don't Hate Documentation (don't worry – you don't have to love it)
If you contemplate this field, you must know that you will spend considerable time documenting the current and future data systems and terminology states. Before you even begin to design and implement a solution, good documentation is key.
Transition Tips
So you've decided it’s for you. Here are some suggestions.
Get Into It Gradually
In my opinion, there are only two ways to approach this transition from DevOps/DevSecOps type of work to data: sharp, where you move abruptly from one role to the next without any progression, or gradual, where you develop the transferable skills over time and then make the entire shift.
The sharp approach requires you to front-load your learning and put the learning in this new role above all else.
The gradual approach, however, allows you to make the transition more smoothly at a more digestible pace.
While my skills are transferable, each field has quirks and experience matters. As mentioned, you won’t be an expert overnight.
Don’t Do it Alone: Get a Mentor!
You need to ask people to support you. Whether it’s in your organization or through a colleague in the field, having mentors is imperative.
This may not be too complicated if you work for a larger organization in a consulting capacity. Typically, your employer would rather see you transition into a role within the organization than lose you. Get on the phone/Slack and start having some conversations to shape if transitioning into data is possible and what that would look like.
On the other hand, if you are in a smaller organization, they may not be able to support you in developing the skills and experience you need to make this shift. In this case, I would recommend you start forging relationships with people in the field and formally ask one or more people to become a mentor to you as you plan your strategy and execute it.
It’s as simple as: “You seem like somebody I could learn from, and I’m transitioning into data. Would you mentor me?”
Set Up a Strong Foundation (From the Top Down)
Set yourself up for a good start by building a strong foundation. You may be tempted to go right to tools and learn how to use the latest and greatest in the data engineering realm… but you must resist!
In this field, building a strong foundation means getting a birds-eye view of the field first. Spend time learning architectural concepts of data engineering (security, governance, data architecture principles, ethics, etc). I suggest DAMA-DMBOK2 as your first pick. The DAMA-DMBOK2 book will provide you with a good mental map of all things data and has a wealth of information on methods to classify, and catalog to work with data from an architectural perspective.
Next on my list is, The Data Warehouse Toolkit. This gets you into the thick of the subject with dimensional modeling. Again, this is a great reference book for learning concepts around modeling, but most importantly, techniques for applying those concepts.
Without this overview, I promise you will get lost in the details. However, once you have the overview, you can be strategic with your learning and application and dive into areas without losing footing. The concepts and ideas you learn in that context are tool agnostic and can be applied to almost all data engineering tools and platforms.
Get Certified
Certification is a great way to solidify your learning and understanding of one or more leading data warehousing platforms. Sure, the knowledge you gain of one platform is not necessarily transferable to another. However, either way, you get to deepen your knowledge of infrastructure, which often serves as the means to understand or relate to other platforms.
Get Down and Dirty (With Automation!)
This is an area where you are sure to shine. You will find that setting up automation for data pipelines will feel very familiar when you’re coming from a DevSecOps background. The new addition to your toolkit will be very familiar: SQL. You will use it to describe schemas, models, views, and tables (as part of your automation). You’ll find that your newfound data colleagues will love that you can handle all that infrastructure stuff!
Remember Your Transferable Skills
There are a lot of transferable skills when you are transitioning from DevSecOps to data. You already have:
- A structured approach to solution-finding – you will learn to play with different objects with different inherent properties and relations.
- Familiarity with automation – sure you will have to brush up on SQL, but chances are you will do that in concert with whatever language you used for building your DevSecOps Pipelines.
- People skills – If you have held customer-facing roles and learned a few things about effectively managing the people aspect of a project, that won’t be lost here. It will be used here more than you know: You will be front and center to find the technical means to resolve problems in business “speak."
