AWS Engineer - Terraform, Kubernetes, Snowflake, Java, Python
Our client is seeking a Software Engineer to build and optimize our data platforms leveraging AWS and key technologies like CDAP, Snowflake, and Databricks. You will design and implement robust and scalable data pipelines, ETL, and analytics systems in the cloud.
Responsibilities:
- Develop and enhance data pipelines, ETL processes using CDAP on AWS infrastructure.
- Build data integration flows to migrate large datasets into Snowflake data warehouse.
- Implement AWS infrastructure-as-code solutions for deployment automation.
- Instrument data pipelines and leverage monitoring for performance tuning and reliability.
- Work with data scientists to optimize data workflows and models on Databricks.
- Follow security best practices for access control, encryption, auditing across data platforms.
- Participate in architecture reviews and technology selections.
- Continuously monitor and improve data platforms for scalability and costs.
Requirements:
- 5+ year's experience in Back End development or data engineering.
- Hands-on experience with AWS services like S3, EC2, EMR.
- Proficiency in SQL and experience with CDAP, Spark, Kafka.
- Experience building scalable ETL processes and workflows.
- Strong programming ability with Python, Java and unit testing.
- Infrastructure-as-code expertise with CI/CD pipelines.
- Ability to communicate complex topics clearly.
Nice to Have:
- Experience with Snowflake, Databricks, and GCP or Azure.
- Knowledge of streaming data architectures
- Data security and compliance implementation.
- Machine Learning Operations (MLOps) experience
Contract role inside IR35 - hybrid - Central London based
Candidate must be eligible to work in the UK
By applying to this job you are sending us your CV, which may contain personal information. Please refer to our Privacy Notice to understand how we process this information. In short, in order to supply you with work finding services, we will hold and process your personal data, and only with your express permission we will share this personal data with a client (or a third party working on behalf of the client) by email or by upload to the Client/third parties vendor management system. By giving us permission to send your CV to a client, this constitutes permission to share the personal data that would be necessary to consider your application, interview you (Phone/video/face to face) and if successful hire you.
Scope AT acts as an employment agency for Permanent Recruitment and an employment business for the supply of temporary workers. By applying for this job you accept the Terms and Conditions, Data Protection Policy, Privacy Notice and Disclaimers which can be found at our website.
Posted Date: 28 Oct 2024
Reference: JSDATA/AWS ENGINEER
Employment Business: Scope AT Limited
Contact: Vig Patel