Job description
Role overview
Senior Data Engineer required to support the development of an Azure Databricks platform for a leading education business. Responsibilities include designing and developing data pipelines, optimizing cloud workflows, and ensuring data quality and security.
Key responsibilities
- Design and maintain data pipelines using Databricks and Lakehouse
- Optimize data workflows for performance and scalability
- Implement CI/CD pipelines for notebook deployment
- Develop Python/Pyspark notebooks with governance best practices
- Monitor and tune data processing systems
Requirements
- 4+ years in data engineering or related field
- Strong Databricks and Azure cloud platform experience
- Proficiency in Python, Pyspark, and CI/CD tools
- Data governance and security expertise
- Problem-solving with complex data architectures
Benefits
- Work with cutting-edge cloud data technologies
- Collaborative environment in education sector
- Professional development opportunities
- Competitive salary and benefits package
Responsibilities
- Design and maintain data pipelines using Databricks and Lakehouse
- Optimize cloud workflows for performance and scalability
- Implement CI/CD pipelines for notebook deployment
Requirements
- 4+ years in data engineering or related field
- Strong Databricks and Azure cloud platform experience
- Proficiency in Python, Pyspark, and CI/CD tools
Benefits
- Work with cutting-edge cloud data technologies
- Collaborative environment in education sector
Keywords
DatabricksPythonPysparkCI/CDAzuredata pipelinescloud engineeringdata governanceLakehouse architectureETL