We are looking for a Senior Data Engineer/Architect with extensive hands-on experience in the Databricks ecosystem and exceptional communication skills to join our flagship project: a cutting-edge Data Platform for the life sciences industry. This platform supports industry leaders such as Pfizer, Moderna, and Novartis in developing innovative RNA-based solutions, leveraging data-driven research, cloud computing, and advanced AI capabilities. This role offers a unique opportunity for an experienced data engineer to take the next step in their career, playing a key technical leadership role in the rapidly evolving world of AI-driven solutions

Responsibility

  • Design, develop, and optimize data pipelines and workflows within the Databricks platform.

  • Take part and lead architecture discussions with engineering, product managers and data scientists to implement advanced analytics solutions that drive business insights.

  • Build, optimize, and fine-tune Databricks workflows to improve performance and reliability.

  • Work closely with data scientists and analysts to ensure the quality of data.

  • Ensure the integrity, accuracy, and security of data across all processing stages.

  • Implement data ingestion from various sources into Databricks, ensuring data quality and reliability.

  • Participate in daily Scrum ceremonies and collaborate with team members in the US and Europe with required online presence from 9 AM to 5 PM EST



Requirements

  • Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.

  • 10+ years of experience in the software development industry, preferably in data engineering, data warehousing or data analytics companies and teams.

  • 5+ year of experience with the DataBricks ecosystem.

  • Expert level of Python and Typescript.

  • Expert level of understanding and hands-on experience with Lake House architecture.

  • Expert level of experience with Spark/Glue and Delta tables/Iceberg.

  • Experienced in designing and implementing complex, scalable data pipelines/ETL processes using Databricks.

  • Skilled in cloud-based data storage and processing technologies, particularly AWS services such as S3, Step Functions, Lambda, and Airflow.

  • Familiar with CI/CD practices, version control (Git), automated testing, and Agile environments.

  • Experience with the Agile development process in a distributed engineering team.

  • Ability to articulate ideas clearly, present findings persuasively, and build rapport with clients and team members.

  • Experience working in US-led high-tech companies and startups.

Nice to Have

  • DataBricks certifications

  • AWS or Azure DevOps or SA certifications

  • Knowledge of basic DevOps and MLOps principles

  • Experience in working with Data Scientists and ML Developers

  • Experience in management and lead developer roles in technology services companies