We are looking for a Senior Data Engineer/Architect with extensive hands-on experience in the Databricks ecosystem and exceptional communication skills to join our flagship project: a cutting-edge Data Platform for the life sciences industry. This platform supports industry leaders such as Pfizer, Moderna, and Novartis in developing innovative RNA-based solutions, leveraging data-driven research, cloud computing, and advanced AI capabilities. This role offers a unique opportunity for an experienced data engineer to take the next step in their career, playing a key technical leadership role in the rapidly evolving world of AI-driven solutions
Responsibility
Design, develop, and optimize data pipelines and workflows within the Databricks platform.
Take part and lead architecture discussions with engineering, product managers and data scientists to implement advanced analytics solutions that drive business insights.
Build, optimize, and fine-tune Databricks workflows to improve performance and reliability.
Work closely with data scientists and analysts to ensure the quality of data.
Ensure the integrity, accuracy, and security of data across all processing stages.
Implement data ingestion from various sources into Databricks, ensuring data quality and reliability.
Participate in daily Scrum ceremonies and collaborate with team members in the US and Europe with required online presence from 9 AM to 5 PM EST
Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
10+ years of experience in the software development industry, preferably in data engineering, data warehousing or data analytics companies and teams.
5+ year of experience with the DataBricks ecosystem.
Expert level of Python and Typescript.
Expert level of understanding and hands-on experience with Lake House architecture.
Expert level of experience with Spark/Glue and Delta tables/Iceberg.
Experienced in designing and implementing complex, scalable data pipelines/ETL processes using Databricks.
Skilled in cloud-based data storage and processing technologies, particularly AWS services such as S3, Step Functions, Lambda, and Airflow.
Familiar with CI/CD practices, version control (Git), automated testing, and Agile environments.
Experience with the Agile development process in a distributed engineering team.
Ability to articulate ideas clearly, present findings persuasively, and build rapport with clients and team members.
Experience working in US-led high-tech companies and startups.
Nice to Have
DataBricks certifications
AWS or Azure DevOps or SA certifications
Knowledge of basic DevOps and MLOps principles
Experience in working with Data Scientists and ML Developers
Experience in management and lead developer roles in technology services companies
