emagine Polska is looking for a skilled Databricks Data Engineer to join their team in a fully remote capacity. This role focuses on building and maintaining robust data pipelines and core data products, ensuring they are sustainable and easily manageable by the engineering team.
Key responsibilities
- Build bronze, silver, and gold pipelines using PySpark and SQL according to established patterns.
- Implement core data products that include data contracts, comprehensive tests, and clear documentation.
- Ensure pipelines are idempotent and re-runnable, with the ability to handle replays, backfills, and late-arriving data.
- Deploy solutions through CI/CD pipelines using Databricks Asset Bundles, incorporating unit tests and data-quality checks.
- Optimize performance and costs through Delta techniques, liquid clustering, and appropriate compute sizing.
Requirements
- Proven experience working with Databricks and building scalable data pipelines.
- High proficiency in PySpark and SQL development.
- Strong understanding of CI/CD processes and automated deployment workflows.
- Ability to implement rigorous data quality checks and testing frameworks.
- Familiarity with Delta Lake, data optimization, and Oracle database environments.
What we offer
- A fully remote working environment allowing for flexibility and focus.
- Opportunity to work on complex data engineering projects with modern technologies.
- Collaboration within a professional team focused on high-quality documentation and engineering standards.
- Engagement on a long-term project basis with clear technical objectives.