Senior Data Engineer (AWS, Python, AI) Centurion
Active Recruitment · Centurion, Gauteng · Hybrid
Overview
Active Recruitment is looking for a hands-on Senior Data Engineer to join a financial services team in Centurion. You will own the full data engineering lifecycle, from ingestion and transformation to platform architecture and production deployment on AWS. This is a great fit for someone with 7+ years of experience who enjoys building scalable pipelines, coaching other engineers, and working with AI enabled tooling in a hybrid environment.
About the role
This is a hands-on senior role where you will design, build and run complex data pipelines on AWS. You will be part of an Agile team delivering data engineering outcomes aligned to sprint goals and programme commitments.
What you will do:
- Design and implement scalable ingestion and transformation pipelines that support batch and incremental processing.
- Build in error handling, monitoring and alerting to keep pipelines robust.
- Implement data validation and reconciliation mechanisms to maintain confidence in data assets.
- Design AWS solutions aligned to the data platform strategy, contributing to the evolution of the cloud based data lakehouse.
- Provide technical guidance and oversight to less experienced data engineers.
- Support analytics, BI and data science teams with clarity on data structures, availability and pipeline behaviour.
- Review code and configuration, and help define shared standards, patterns and best practices.
- Promote responsible use of AI within governance frameworks, using only enterprise approved AI tooling.
- Apply version control, automated testing, CI and CD and structured releases.
- Diagnose and resolve data incidents, perform root cause analysis and implement corrective actions.
- Communicate technical progress, constraints and decisions clearly to business and technical stakeholders.
Requirements
- 7+ years professional experience in data engineering or related roles.
- Advanced Python for data pipeline development, automation and scripting.
- Advanced SQL for complex query writing, performance optimisation and stored procedures.
- Mid to advanced AWS skills covering S3, Glue, Lambda, Redshift, RDS, IAM, CloudWatch and Athena.
- Experience with data warehousing and data modelling including star and snowflake schemas.
- Experience with ETL and ELT pipeline development including batch, incremental and near real time patterns.
- Experience with data lakehouse architecture on cloud native platforms.
- Experience with CI and CD and Git including version control, automated testing and deployment pipelines.
- Power BI experience supporting BI teams, data model integration and semantic layer understanding.
- Experience ingesting data from relational databases such as SQL Server or Oracle, cloud storage, APIs, SFTP, MongoDB and file based sources.
- Demonstrated delivery of production ready end to end data pipelines and cloud data platforms.
- Experience working within Agile or SAFe delivery environments.
- Experience coaching or guiding other engineers.
- Highly valued: Apache Spark or PySpark, dbt, Apache Airflow or AWS Step Functions, Terraform or CloudFormation, Kafka or Kinesis, machine learning workflow support, and Azure or GCP exposure.
- Experience in financial services, insurance, investment management or other regulated data environments is preferred.
Responsibilities
- Contribute to data engineering outcomes aligned to sprint goals and programme level commitments.
- Take part in Agile ceremonies including planning, estimation, prioritisation and continuous improvement.
- Decompose data features into implementable tasks with reliable effort estimates.
- Ensure outputs meet agreed functional, performance and data quality expectations.
- Design and implement scalable ingestion and transformation pipelines across multiple systems and data domains.
- Build error handling, monitoring and alerting into pipelines.
- Implement data validation and reconciliation mechanisms.
- Design solutions aligned to the data platform strategy and target architecture.
- Assess the impact of design choices on security, performance, cost and supportability.
- Identify integration points, upstream and downstream dependencies and risks early in the delivery lifecycle.
- Provide guidance and technical oversight to less experienced data engineers.
- Support analytics, BI and data science teams with clarity on data structures, availability and pipeline behaviour.
- Review code and configuration and help define shared standards, patterns and best practices.
- Optimise and simplify existing pipelines and data models.
- Promote responsible use of AI to improve engineering productivity and solution quality.
- Validate all AI assisted outputs and retain accountability for correctness, compliance and production readiness.
- Apply version control, automated testing, CI and CD and structured releases.
- Contribute to operational readiness, deployment pipelines and monitoring.
- Diagnose and resolve data incidents, perform root cause analysis and implement corrective actions.
- Build solutions with privacy, access control and regulatory considerations embedded by design.
- Identify and mitigate data risks and support remediation of audit findings and compliance gaps.
- Maintain accurate documentation of pipelines, data models and operational considerations, and keep work tracking up to date in Jira.
- Adhere to delivery governance, change management and release processes.
- Communicate technical progress, constraints and decisions clearly to business and technical stakeholders.
- Collaborate across technology and analytics teams and contribute positively to team culture.
Job details
- Organisation
- Active Recruitment
- Location
- Centurion, Gauteng
- Job type
- Full time
- Work arrangement
- Hybrid
- Experience level
- Senior (7+ years)
- Category
- IT & Software Development
- Application Method
- Apply manually
Success Guide
This Senior Data Engineer role is a strong opportunity if you already have solid AWS and Python experience under your belt and want to work on real production pipelines in a regulated environment. Before you apply, update your CV to clearly show the data pipelines you have built end to end, not just the tools you have used. Recruiters for senior data roles look for evidence of ownership, scale and delivery.
In your application, highlight your experience with S3, Glue, Redshift and Lambda, and mention any work with data lakehouses or streaming. If you have coached junior engineers before, add that too. Prepare for questions on pipeline design, data quality and how you handle incidents. If you have used AI tools responsibly in your workflow, that is worth mentioning since this role values it.
How we help you prepare
This employer takes applications on their own website, so you'll submit this one yourself — but SPAAN NOW gets everything ready for you first. Here's how it works:
- Our AI tailors your CV to match this job's keywords and requirements, so it stands out to recruiters and screening systems.
- A custom cover letter is written specifically for this role — no generic templates.
- We gather the documents this job needs: CV. SPAAN NOW pulls these from your profile and packages them together, ready to attach.
- Once your application is prepared, we show you exactly how and where to apply — just follow the link to the employer and submit your ready-made CV, cover letter and documents.
- Mark it as applied under Applications to keep track, and if you're shortlisted, use AI Interview Prep to practise questions commonly asked for roles like this.
We do the hard part — tailoring and gathering everything — so applying yourself only takes a minute.
Spaan Talk
Nobody has said anything yet. Be the first.