Databricks Data Architect

Our client is building a unified Master Data Management platform that consolidates data from many applications into a single source of truth. On top of this platform an AI layer is being built - either as a separate layer over the unified data, or embedded directly into the ETL processes to deliver clean, connected, and enriched data.

The project domain is Occupational Health & Safety, Incident Management, Risk Management, and global regulatory frameworks. The platform must provide trustworthy, connected, and explainable data for incident management, risk assessment, and compliance with global regulatory requirements.

This is a hands-on architecture role. The person will own the platform design and also build reference pipelines and data models, additionally define AI/RAG patterns together with the engineering team.

Tech stack:
Databricks (Spark, PySpark, SQL, Delta Lake, Unity Catalog, Lakeflow), ETL/ELT, knowledge graphs / GraphFrames, semantic layer (Unity Catalog metric views), AI Search / Vector Search, RAG / LLM tooling, cloud infrastructure (AWS / Azure / GCP).

Requirements:

• Strong production experience with Databricks Lakehouse architecture, including Spark, PySpark, SQL, Delta Lake, Unity Catalog, and workflow orchestration

• Hands-on experience designing and building ETL/ELT pipelines for batch and incremental ingestion, cleansing, normalization, deduplication, and enrichment

• Practical experience with MDM: golden records, survivorship/merge rules, trust ranking, identity resolution, duplicate detection, SCD, and exception workflows

• Strong data modeling skills for analytical, operational, and semantic consumption patterns

• Experience designing a semantic layer with shared business definitions, governed metrics, reusable dimensions, and consistent entity definitions

• Experience with data quality and observability: pipeline SLAs, schema drift, CDC, data contracts, dead-letter handling, and source-to-master reconciliation

• Experience implementing data governance and security: Unity Catalog lineage, RBAC/ABAC, row/column-level security, PII handling, and regulatory traceability

• Ability to translate business requirements from product, compliance, and engineering stakeholders into scalable data architecture

Nice to have:

• Experience with Databricks Lakeflow Connect, Lakeflow Spark Declarative Pipelines, and Lakeflow Jobs

• Experience with Unity Catalog metric views or comparable semantic-layer technologies

• Experience with knowledge graphs, graph analytics (e.g. GraphFrames), or graph-based entity resolution - linking people, organizations, locations, incidents, hazards, controls, regulations, assets, and corrective actions

• Experience building AI/RAG solutions over enterprise data using AI Search / Vector Search, embeddings, metadata filtering, retrieval evaluation, and source-grounded generation with citations

• Experience with ML-based data enrichment, classification, anomaly detection, or entity matching

• Experience in regulated domains such as occupational health and safety, incident management, risk, compliance, ESG, insurance, healthcare, or industrial operations

We offer:

• Projects for such clients as PayPal, Wargaming, Xerox, Philips, Adidas and Toyota

• Competitive compensation that depends on your qualification and skills

• Career development system with clear skill qualifications

• Flexible working hours aligned to your schedule

• Options to work remotely