About the role
Job Description
* Design and build scalable cloud-native data platforms from greenfield to production * Design and implement near-real-time ingestion pipelines using event-driven patterns * Define and enforce data platform standards including Data Lake/Lakehouse principles, medallion architecture, and data contracts * Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability * Introduce best practices for code quality, testing, and CI/CD across data pipelines * Drive adoption of AI tooling and agentic workflows within the data engineering team * Ensure data quality, observability, and reliability across pipelines and platforms * Design and deliver self-service tooling and microservices that abstract platform complexity * Collaborate directly with the Customer’s scientific and engineering teams * Monitor platform performance, reliability, and cloud infrastructure costs
Qualifications
* 5+ years of experience in data engineering * 3+ years of production experience with Databricks and Delta Lake * Strong expertise in medallion architecture and event-driven asynchronous pipeline design * Advanced proficiency in Python, PySpark, and SQL * Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization * Strong AWS expertise including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway * Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles * Practical experience with Git workflows, pull request reviews, and automated testing * Experience configuring secure CI pipelines using OIDC or service principals * Practical MLflow experience including experiment tracking, model registration, and reproducibility practices * Strong understanding of data quality, lineage, provenance, schema contracts, and versioning * Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX * Experience with monitoring, observability, and cloud cost optimization * Upper-Intermediate English level or higher
WILL BE A PLUS
* Experience packaging workloads in Docker containers and running batch jobs on AWS * Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy * Experience building feature sets and train/test splits that avoid data leakage * Experience with regulated or high-security environments including data residency requirements * Experience disabling or restricting AI features in secured environments
Additional Information
PERSONAL PROFILE
* Strong analytical mindset * Ownership and accountability * Ability to communicate complex technical concepts clearly * Proactive approach to problem-solving * Attention to scalability and maintainability * Collaborative and team-oriented attitude