Data Engineer
The Knowledge Center, located in Gdańsk, is a hub for innovation and develop state-of-the-art business and technology solutions to help us navigate the future. And we want to do that together with you.
Team and Project overview:
We aim to build a big data analytics platform for a global shipping company - Hapag-Lloyd AG. In Gdańsk, we established a Knowledge Center. Within it - Data Management Solutions for Analytics (DMSA). A group of top IT experts, designers, and executors, will apply AWS cloud technology in the maritime and shipping industry. We are quality-driven and have a passion for cloud (especially AWS), analytics, and the power of positive changes that technology solutions bring to business globally.
We are looking for passionate Data Engineers who can create insights from this Data by running Analytics on a scale. We use AWS Cloud extensively, but we are open to people who have experience in other cloud solutions in GCP or Azure and want to switch to AWS. Only one programming language is required (Scala / java / Python) and a strong knowledge of SQL.
Responsibilities:
- Building and maintaining complex cloud (AWS) data management systems that combine core data sources into data warehouses and analytical services
- Identifying optimal solutions to efficiently scale the data flow agnostic of on-prem or cloud
- Working closely with Data Analysts, Data Engineers, Data Scientists, and other stakeholders to design and maintain data models
Requirements:
- Knowledge about AWS cloud services (EC2, S3, ECS, EKS, RDS)
- Hands-on experience in Spark (Scala/Python/Java) - we use Spark in all our Data pipelines
- Excellence and proficiency in SQL
- Hands-on experience with CI/CD and version control using GitLab or GitHub
- Experience with containerization technologies (Docker basic knowledge needed)
- Knowledge of Airflow for scheduling and Data orchestration
- Scripting skills: Bash / Python
- Familiarity and flexibility to work with structured, unstructured, and semi-structured datasets on a GB or PB scale
- Expert-level knowledge of data integration and familiarity with common data integration challenges
- Willingness to convert data types, handle errors, and translate between different technology stacks
- ETL pipeline optimization on Software in code and hardware level
We offer: