Data science and machine learning engineering might get the majority of the hype in the new big data economy impacting almost every industry. It’s no secret that the job market for these roles is hot, with both data scientists and machine learning engineers able to demand extremely lucrative salaries.
For someone interested in implementing data solutions that drive value for a business, however, there’s an adjacent career path that’s quickly gaining recognition: data engineering. According to Dice and Burning Glass’ NOVA platform, “data engineer” was the top tech job in April 2019, its 88% growth year-over-year nearly double the 49% YoY growth for “data scientist.” Though data scientists command slightly higher salaries in the US, earning between $124,770 and $154,336 while the range for data engineers is $98,287 and $130,038, let’s be real: given the median individual wages in the US were less than $60,000 in 2021, data engineering is undeniably a path to financial stability.
Of course, it’s one thing to choose to become a data engineer and another to know how to become a data engineer. For this reason, we’ve put together this step-by-step guide to help you on your way. We’ll start by clarifying something: what exactly does a data engineer do?
What does a data engineer do?
Data engineering is an interdisciplinary field focused on the implementation, assessment, and maintenance of so-called data architectures: the systems, pipelines, and database networks designed to serve an organization’s data needs.
Understandably, a data engineer spends much of their time coding: programming in languages like Python to stand up a data pipeline or data warehouse and debug what’s already been shipped. Time not spent in front of code is usually spent in meetings with various stakeholders to discuss how best to optimize data pipelines for data analytics and business analytics operations.
How to become a data engineer
Step One: Get educated.
To succeed as a data engineer, you need a (*ahem*) very particular set of skills in computer science, data management, data analytics, and machine learning:
Computer science: database querying with SQL; programming languages like Python, R, Java, C, and C++; operating systems like Unix, Linux, macOS, and Windows; software development skills
Data management and data processing: tools for database management like Apache Hadoop, data warehousing, data mining, and cloud computing with Amazon Web Services (AWS)
Data analytics and machine learning: while data engineer doesn’t need to have the same expertise with data analysis and machine learning as a data analyst, business analyst, data scientist, or machine learning engineer, some expertise is crucial to optimize data systems for these purposes
If you’re just getting started, this skill set might seem unattainable: what even is data warehousing? Thankfully, there’s an ever-increasing number of great educational opportunities that can help you master these skills, whatever your background.
Data Engineering Bachelor’s Programs
Who?
High school graduates who have sufficient time and money — or a willingness to take out loans — for a traditional four-year undergraduate degree.
What?
You’d be hard-pressed to find a specific data engineering major at most colleges. For this reason, many interested in going into data engineering major in a related field like computer science, computer engineering, data science, or data analytics. Any of these majors will provide crucial training in the major data engineering skills.
Why?
Traditional four-year undergraduate degrees have historically been considered a reliable way into technical fields. Earning your bachelor’s degree also allows you to pursue a master’s or doctorate later on.
How much?
Total costs for bachelor’s degrees average $102,828 at public universities (in-state) and $218,004 at private universities.
What next?
If you’re interested in a bachelor’s degree that can put you on course to landing a data engineer job, check out our guide to the best data science colleges and universities.
Data Engineering Master’s Programs
Who?
Bachelor’s degree-holders who are looking to transition into data engineering or advance existing data engineering careers.
What?
An individual seeking master’s-level study has several options: they can seek out a specific data engineering program, pursue a general data science master’s, or even pursue a data science master’s with a data engineering concentration. Individuals also have a choice of whether to study in person or online.
Why?
A graduate degree can be a crucial credential, whether you’re looking to transition your career or advance it. In fact, according to a recent Burtch Works study, two-thirds of data scientists hold a master’s degree.
With more and more great online degrees available, many students are able to pursue a master’s without relocating and even while still working.
How much?
Master’s degrees in STEM average $61,200 according to educationdata.org.
What next?
Our guides to data science master’s programs and online data science master’s programs have some great options that can help you move — or move up in — data engineering
Data Engineering Bootcamps
Who?
Bootcamps are great for those looking to land an entry-level job without pursuing a time- and cost-intensive degree.
What?
As with master’s degrees, aspiring data engineers have the option of data engineering-specific bootcamps or data science bootcamps. Either way, they will receive comprehensive training in relevant skills and concepts and, crucially, career services designed to help them earn a job offer.
Why?
Bootcamps are shorter, less expensive, and more flexible than traditional degrees. With added career services, students are able to efficiently pursue an entry-level data engineering position.
How much?
Bootcamps averaged $11,727 in 2020. Some bootcamps offer income-share or payment deferral plans, or even money-back guarantees. As you consider one of these options, make sure to read the small print to ensure you’ll be eligible.
What next?
You can see our picks for data science bootcamps in our DS bootcamp guide.
Data Engineering Certificate, Certifications, and Other Short Courses
Who?
Those contemplating a career in data engineering or who already have some data engineering skills and want to expand their skill set or demonstrate their skills to potential employers.
What?
Offered online, data engineering certificates and other short courses provide either comprehensive or targeted training in data engineering, oftentimes asynchronously
Certification programs culminate in an exam that, if passed, certifies an individual in a particular skill or capacity.
Why?
These programs are often more flexible than bootcamps or degree programs, not to mention less expensive. They also result in a credential that can signal a candidate’s skills to potential employers.
How much?
Certificates and other short courses range from free to several thousand dollars. Some providers, like Coursera and DataCamp, offer educational programs for a flat monthly fee.
What next?
See our rundown of data engineering courses for more.
Step Two: Get experienced.
Education is step one, but to break into data engineering you will also need to show a recruiter or hiring manager that can put your technical skills into practice on real-world data architecture and other data infrastructure. Here are some ways you can build this experience prior to applying:
Internships
One way to gain on-the-job experience is through an internship. Interns are embedded on real teams, where they can support that team’s work while glimpsing into the operations of an actual organization. Most students and recent graduates complete internships during summer breaks, but many companies will offer them year round, especially for recent graduates or students in advanced stages of their degrees. To find data engineering internships, you can look at the career pages of companies that interest you or check out jobs boards like LinkedIn and Indeed.
Independent or Collaborative Projects
A strong portfolio of projects you completed independently or collaborated on can be a crucial differentiator between you and other candidates, as these projects give a glimpse into how you approach problems and leverage your technical skills to solve them. To support portfolio-creation, many bootcamps, bachelor’s, and master’s programs have students complete capstones towards the end of their course of study. These capstones, along with other projects, can then be displayed on a personal website or on GitHub.
Freelancing and Pro Bono Work
It can sometimes be difficult to invent projects out of thin air; in this case, freelancing and pro bono opportunities are key. While many want to start making money right out of the gate, often it can be difficult to build a client base without work examples to show. In this case, pro bono work through a social-good outfit like DataKind, Catchafire, or Statistics Without Borders can be a great way to get started.
Join a Data-Oriented Club or Organization
Colleges and universities generally have robust rosters of student-led special interest groups. While you might not find a data engineering-specific group, joining a data science or data analytics group can help you meet individuals in the industry, build out your professional network, and stay up-to-date with the latest developments in your field.
Step Three: Get polished.
After you’ve matched your education with experience, the next step is to figure out how to best present these to potential employers. By refining your portfolio and resume and figuring out your most compelling personal narrative, you can transform yourself into more than just your skills and expertise: a potential team-member. To learn more about getting polished for the job market, check out our resume and interview tips.
Step Four: Get busy.
In a perfect world, jobs would come to you. Unfortunately, if you want to become a data engineer, there aren’t any shortcuts — you’ll need to get busy. This means researching potential job opportunities, expanding your personal network, and tailoring each application to the job description. But don’t substitute quantity for quality: five well thought-out applications paired with referrals from your network might just be more effective than 100 resumes plastered across LinkedIn.
Step Five: Profit!
In this guide, we’ve covered what a data engineer does and how you can get started down this career path. In the end, it means a lot of work, but if you arm yourself with education and experience and are persistent and patient, there’s a good chance you’ll ultimately be successful. But remember: your first job in data engineering won’t be your last. If you’re interested in learning more about what a data engineering career might have in store for you, check out our guide to the typical data engineering career path.