Senior Data Engineer - Big Data

data engineeringapache sparketleltgcpawsbigquerys3gcsemrairflowkafkaflinkhivetrinodata warehousedata lakemedallion architecturelakehouseapache icebergdelta lakestreamingbatch processing

Key details

Salary

Not specified

Employment type

Full-time

Seniority

Senior

Years experience

5-10

Location

Netanya, Israel

Full job description

Teads is seeking a Senior Data Engineer to join their Core Data Platform team in Netanya, Israel. The role involves designing, building, and maintaining scalable ETL/ELT data pipelines using Apache Spark, managing cloud data infrastructure on GCP or AWS, and enhancing data warehouse and data lake solutions. The candidate will collaborate with data scientists, analysts, and engineers to ensure data reliability and accessibility. Requirements include 5+ years of data engineering experience with large-scale batch and streaming pipelines, hands-on experience with Apache Spark and distributed processing frameworks, production experience with cloud platforms (BigQuery, S3, GCS, EMR, AirFlow), Kafka streaming platforms, and strong knowledge of data warehouse and lake concepts including Medallion Architecture. Nice to have experience with lakehouse table formats like Apache Iceberg or Delta Lake. The position offers a hybrid working model (3 days in office), parking, mentorship, pet-friendly office, happy hours, and a stocked kitchen.

What you'll do

Design, build, and maintain ETL/ELT data pipelines using Apache Spark to ingest, process, and transform large-scale datasets
Architect and manage data infrastructure on Google Cloud Platform or Amazon Web Services including BigQuery, S3, GCS, EMR, and AirFlow
Improve and manage data warehouse and data lake solutions ensuring data quality, consistency, and accessibility
Collaborate with cross-functional teams to understand data needs and implement solutions supporting new product features and business initiatives
Implement monitoring, alerting, and logging systems to maintain data pipeline health and ensure data accuracy

Requirements

5+ years of data engineering experience building and operating production data pipelines at scale (TB+ datasets, hourly/daily batch or streaming workloads)
Hands-on production experience with Apache Spark and distributed data processing frameworks such as Flink, Hive, or Trino
Strong understanding of large-scale batch and streaming pipelines including performance tuning and troubleshooting
Production experience building and operating data solutions on GCP or AWS including cloud-native services such as BigQuery, Dataproc, GCS, S3, EMR, or Redshift
Production experience with Kafka or Kafka-compatible streaming platforms including development, operation, and troubleshooting of real-time data pipelines
Strong understanding of data warehouse and data lake concepts including Medallion Architecture (Bronze, Silver, Gold) and data platform best practices
Nice to have: Production experience with a lakehouse table format (Apache Iceberg or Delta Lake), understanding schema evolution and table maintenance

Tech stack

Apache SparkScalaPythonJavaGoogle Cloud PlatformAmazon Web ServicesBigQueryS3GCSEMRAirFlowKafkaFlinkHiveTrinoApache IcebergDelta Lake

Benefits

Hybrid working model (3 days per week in the office)Nearby parking availableMentorship program and internal learning toolsPet friendly officeHappy hoursFully stocked kitchen