Published on:
22.3.2022

“It is like playing Lego” – What does a Data Engineer do?

Publikálva:
2022.03.22.

Barnabás Kádár got into coding as a social scientist because he wanted to find a STEM job. Since then, he has become an experienced Data Engineer and now teaches Python programming himself. What does a Data Engineer do, what skills does he need and who should choose this career path? Learn more about Data Engineering through Barni's story!

After graduating at Green Fox as a junior programmer, you started working as a Data Analyst. How did your career develop after that?

I worked for more than 3 years at one of the Big4 companies. My daily work was a mix of consulting, data analysis and programming. Then last summer I changed jobs and now I work as a Data Engineer at a Hungarian IT company called Starschema. It is a medium-sized company of around 240 people specialised in big data and business intelligence solutions. This is a very different corporate environment: This company is explicitly specialised in IT, and my current position is much more technical than the previous one. 

What are your tasks as a Data Engineer?

The client I am currently working for is one of the largest financial asset managers in the world. My job is to organise, tidy up, transform and transfer data and to connect systems. Efficiency is a key factor here because we are managing a significant amount of data. I mainly use Python and SQL programming languages and AWS-based cloud technologies for my tasks. There is a severe lack of Data Engineers in the market, so we recruit recent graduates in economics, social sciences or engineering who have an affinity for this field, and we retrain them. It is also part of my job to teach them Python programming.

What is the difference between a Data Engineer and a Data Scientist?

They are located in different parts of the Big Data chain, but there is a crossover between the two fields. A Data Scientist mostly works with a set of data that has already been collected and organised. He uses statistical and programming tools to analyse them and builds models to help solve specific business problems. A Data Engineer is responsible for setting up the entire data management system. He ensures that the necessary data is available in a usable and structured form. Our work is therefore often a prerequisite for Data Scientists to be able to work with data that is suitable for deeper statistical analysis.

What should you know about the Python programming language?

It is a very human-oriented language that is easy to understand and learn, whether as a first, second, or third programming language. Compared to other languages, you can quickly put together a working program that, among other things, can retrieve data from the internet and then, after analysing it, you can create beautiful visualisations. A general-purpose language like Python may be useful for any IT career, but it is a must for the field of data management.

What do you like most in the field of Data Engineering? 

It gives me a sense of flow when I'm working to tidy up a huge system. It is like playing Lego: putting small colourful pieces together in a nice, regulated order – I can completely immerse myself in it. This knowledge is valuable and is highly appreciated by employers. This is also reflected by the fact that salaries are higher than the average in the field of IT and by the flexible working conditions. You may create stability and a good standard of living with this career, not only in Hungary, but anywhere in the world. I could also work as a freelancer or a digital nomad, which would have been impossible for my pre-Green Fox jobs.

What is the biggest challenge of your job?

What is an advantage is also a disadvantage: There is a high demand for this knowledge, so there is plenty to do. At my workplace, we use an agile methodology. This means that first we estimate how much time each task will take, and then collectively allocate the tasks in a ticketing system. When I have a lot of tickets, I have to prioritise and see which ticket is dependent on another one – and that can be stressful. It is also a challenge to keep your knowledge up-to-date and to improve it. I have spent a lot of time in recent years training to become a Python expert, both at work and in my free time. By the way, continuous self-improvement is essential for all programming languages, as technology is evolving rapidly.

To what extent does this area require soft skills?

Although I work in a team, we have only a few meetings. We have a 15-minute stand-up every day, where we all tell each other in a few minutes what we are working on. If you are an introvert, you may get through a working day with just a few words. However, within Data Engineering you can choose career paths where more emphasis is put on communication, either through customer relations or managing a team.

What type of people would you recommend this career path to?

I would recommend it to people who liked playing Lego as a child, have a basic sense of logic and, for example, have always liked working with numbers. You should also have at least an intermediate level of English knowledge. This is necessary on the one hand because this is the language in which most international companies communicate. On the other hand, it is also essential for keeping your professional knowledge up-to-date, as the latest learning materials are usually available in English.

In hindsight, do you think it was a good decision to choose this field after graduating at Green Fox?

Yes, absolutely. If there had been a specific data-focused course at Green Fox back then, I would have definitely chosen it, because I already knew at that time that I wanted to work with data management. My aim was to gain tangible knowledge that could be objectively measured and would mean the same thing everywhere in the world. This was achieved and many new opportunities opened up for me as a result.

A szerzőről:

About the author

Junior programozó képzések

Zöld út az IT karrierednek

Érdekel

Ismerd meg céges szolgáltatásainkat!

Discover our services for companies!

Tovább olvasnál?

Tovább olvasnál?

Read more?

Read more?