Big Data Engineer (OpenSearch/PySpark/Kubernetes)

Job Title: Data Engineer (OpenSearch/PySpark/Kubernetes)
Amsterdam - Hybrid - 2 days per week
12 Month Freelance Contract
€100 per hour excl vat

Brookwood is committed to driving innovation through data. We are looking for talented Data Engineers to join our dynamic team and support our migration project from on-premises systems to cloud-based solutions.

Position Overview:
We are seeking two Data Engineers with a strong background in Big Data technologies, particularly PySpark and Kubernetes. The successful candidates will inherit a project focused on migrating data from Hadoop and other on-premises solutions to cloud platforms, specifically AWS and Snowflake. Experience with OpenSearch or ElasticSearch is essential to facilitate efficient data ingestion and management.

Key Responsibilities:

• Participate in the migration of data from on-premises Hadoop systems to AWS and Snowflake.
• Design, develop, and maintain data ingestion pipelines using OpenSearch.
• Collaborate with cross-functional teams to ensure smooth data transition and integration.
• Implement and manage data processing workflows using PySpark and Kubernetes.
• Optimize data storage and retrieval processes to enhance performance and scalability.
• Troubleshoot and resolve data-related issues in a timely manner.
• Document processes and contribute to best practices for data management and migration.

• Experience with OpenSearch or ElasticSearch (years of experience not strictly required).
• Proficiency in Big Data technologies, specifically PySpark.
• Familiarity with Kubernetes for orchestration of containerized applications.
• Knowledge of AWS services and Snowflake is a plus.
• Strong problem-solving skills and the ability to work in a fast-paced environment.
• Excellent communication and collaboration skills.

• Experience with data migration projects.
• Understanding of data modelling and data warehousing concepts.
• Familiarity with agile methodologies and DevOps practices.