Data Engineer

What you'll do

• Build scalable data product architecture, capable of supporting both internal and external data consumers
• Responsible for modernizing data frameworks and integrations with Databricks and BigQuery
• Upgrade and reduce toil for developers on Apache Airflow
• Develop and optimize data transformations using Apache Spark (PySpark/Scala)
• Build procedures and guidelines to help teams operate with data
• Identify bottlenecks in development lifecycle and find solutions to improve them
• Work directly with data teams and FinOps teams to drive cross-team efforts
• Implement data governance, access control, and auditing using Databricks Unity Catalog
• Build and integrate automated, reusable data validation suites using data quality frameworks (Great Expectations or similar)
• Implement monitoring and anomaly detection systems for data quality, reliability and performance
• Develop and manage REST APIs to support secure data access, automation, and integration
• Collaborate with data scientists, analysts, and software engineers to deliver governed, reusable data assets
• Implement monitoring, logging, and alerting for data workflows
• Optimize cost and performance of cloud-based data infrastructure

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience)
• 2+ years of experience in data engineering or a related role
• Strong hands-on experience with Databricks, Apache Spark and BigQuery or Snowflake
• Proven experience with Modern table formats such as Delta Lake and Iceberg
• A deep understanding of the data lifecycle and how teams operate with data
• Hands-on experience implementing data governance and metadata management using Databricks Unity Catalog
• Experience managing and extending Apache Airflow (custom operators, plugins, infrastructure)
• Experience with Kubernetes
• Solid experience with AWS cloud services, especially S3 and data-related services
• Experience with data validation and data quality principles and working with SLA systems
• Proficiency in Python and SQL
• Experience with data modeling, data lakes, and lakehouse architectures
• Strong understanding of distributed systems and big data processing

Tech stack