Lead Data Engineer (m/f/d) · for the BMP Migration Tool

ID: 10352

Our client is a service company that operates within an international corporate group.

Full remote

2–3 days per year on site

Must haves

• Databricks
• Lakeflow
• SQL

Nice to have

• Unity Catalog
• PySpark

Your Tasks

• Provide hands-on technical leadership for the BMP data migration.
• Lead the profiling and assessment of source data.
• Define and implement data mappings and transformation rules.
• Develop and maintain data pipelines using Databricks, PySpark, Spark SQL and Delta Lake.
• Establish and oversee data quality checks and monitoring.
• Lead source-to-target reconciliation and resolve data discrepancies.
• Guide testing, CI/CD deployment and code quality practices.
• Prepare runbooks and support cutover activities

Requirements

• Strong hands-on experience delivering data migrations and building production-grade data pipelines.
• Extensive practical experience with Databricks , including Unity Catalog, Delta Lake, Lakeflow Connect, Lakeflow Declarative Pipelines and Lakeflow Jobs.
• Strong PySpark and Spark SQL skills.
• Experience with data profiling, source-to-target mapping, reconciliation and data quality checks.
• Familiarity with DQX , Databricks Data Quality Monitoring and Lakebridge.
• Experience with Databricks Asset Bundles, CI/CD and automated testing.
• Ability to prepare runbooks, troubleshoot migration issues and support cutover activities.
• Proven technical leadership, including guiding implementation and coordinating delivery.
• Required knowledge of: Databricks, Unity Catalog, Lakeflow Connect, Lakeflow Declarative Pipelines, Lakeflow Jobs, Delta Lake, PySpark, Spark SQL, DQX, Databricks Data Quality Monitoring, Lakebridge, Databricks Asset Bundles, CI/CD, Lakehouse Federation and AI/BI Dashboards.
• Professional English communication skills; willingness to work in a hybrid setup.
• Availability to start as soon as possible.