AWS Data Engineer - Kinesis

We have an exciting job opportunity for AWS Data Engineer - Kinesis role at Telford, UK - Permanent.

Title: AWS Data Engineer - Kinesis

Location: Telford, UK

Duration: Permanent

Job Description:

• Design, develop, and maintain Real Time data streaming pipelines using AWS Kinesis for processing and analysing high-velocity data streams.
• Collaborate with stakeholders to gather requirements and design efficient data streaming architectures for Real Time analytics, monitoring, and alerting.
• Build and optimize Kinesis Data Streams, Kinesis Data Firehose, and Kinesis Data Analytics to process Real Time data from various sources.
• Develop and implement ETL processes to transform, enrich, and aggregate streaming data for downstream consumption in data lakes or data warehouses such as AWS S3, Redshift, and DynamoDB.
• Integrate Kinesis with other AWS services like Lambda, S3, and EMR for Real Time data ingestion, processing, and storage.
• Optimize streaming applications for performance and scalability, ensuring efficient resource usage and minimal latency.
• Develop and maintain robust error handling, data validation, and retry mechanisms within Kinesis pipelines to ensure data consistency and reliability.
• Implement security best practices for data streaming, including encryption in transit and at rest, access control using IAM roles and policies, and data retention policies.
• Monitor and troubleshoot AWS Kinesis streams using CloudWatch metrics, alarms, and logs, ensuring high availability and minimal downtime.
• Collaborate with DevOps teams to automate the deployment and scaling of streaming pipelines using AWS CloudFormation, Lambda, and CI/CD tools.
• Work closely with data analysts and data scientists to ensure Real Time data is processed and delivered for immediate insights and reporting.
• Implement data partitioning, sharding, and key management strategies to handle high-throughput data streams and ensure scalability.
• Optimize the cost of Real Time data processing by tuning the configurations of Kinesis and associated AWS services like Lambda and Firehose.
• Stay current with AWS updates and best practices for streaming data, recommending improvements to the architecture as new features and services become available.
• Provide guidance and mentorship to junior engineers in Real Time data processing, ensuring best practices in streaming technologies and AWS cloud services.
• Create and maintain detailed technical documentation for data pipelines, including data flow diagrams, stream processing logic, and operational guidelines