AdTechTalent
Engineering2 months agoHybrid

Epsilon

Lead Data Engineer

sparkscalapythonsqlawsdatabricksemrbig datadistributed computingai-assisted developmentadtechprogrammaticairflowkafkarest apinosql

Key details

Salary

$98K – $182K

Employment type

Full-time

Seniority

Senior

Years experience

5-10

Location

Chicago, United States

Full job description

The Epsilon Activation Delivery Platform team seeks a senior engineer to build and maintain high-scale data pipelines and integrations with major advertising and CTV platforms. The role reports to the Engineering Director and collaborates with data engineers, product managers, and partner teams. Responsibilities include writing scalable code using Spark (Scala/Python) and SQL, leveraging AI-assisted engineering tools, optimizing data connectors, resolving production issues, mentoring engineers, and maintaining automated cloud processing routines. Requirements include a bachelor's degree in a related field, 5+ years experience with big data pipelines, expertise in Apache Spark, Scala/Python, SQL, cloud platforms (AWS preferred), and strong problem-solving and communication skills. Preferred experience includes AdTech/MarTech, orchestration tools like Airflow, high-concurrency REST API integrations, streaming technologies (Kafka), and NoSQL databases. The position is full-time, based in Chicago, Illinois. Salary range is $98,000 to $182,000 annually. Benefits include flexible time off, paid holidays and sick time, family leave and assistance, comprehensive health coverage, 401(k), tuition assistance, commuter benefits, professional development, and more.

What you'll do

  • Write robust, scalable, and maintainable code using Spark (Scala/Python) and SQL
  • Build and maintain core framework and data pipelines
  • Leverage AI-assisted engineering tools like Cursor and Amazon Q Developer
  • Build, maintain, and optimize high-concurrency data connectors for external publishers and ad networks
  • Fix production issues, tune SQL queries, and solve performance bottlenecks in distributed environments
  • Participate in rigorous code reviews and enforce engineering standards
  • Mentor mid-level and junior engineers
  • Build and maintain automated production processing routines integrated with scheduled cloud infrastructure

Requirements

  • Bachelor's degree in Computer Science, Computer Engineering, or related field
  • 5+ years of professional experience building and maintaining big data pipelines
  • Deep hands-on expertise in Apache Spark and distributed computing concepts
  • Strong programming proficiency in Scala and/or Python
  • Fluent SQL skills with ability to refactor code and tune queries for massive datasets
  • Experience working within cloud environments, preferably AWS
  • Experience with managed platforms like Databricks or EMR
  • Ability to solve production issues autonomously and own problems end-to-end
  • Excellent communication skills to collaborate with internal partners and translate business requirements
  • Preferred: Experience in AdTech/MarTech, programmatic advertising, identity resolution, or audience activation
  • Preferred: Familiarity with orchestration tools such as Airflow
  • Preferred: Experience building and optimizing high-concurrency data integrations with external REST APIs
  • Preferred: Exposure to streaming technologies like Kafka or NoSQL databases

Tech stack

Apache SparkScalaPythonSQLAWS EMRDatabricksCursorAmazon Q DeveloperAirflowKafkaREST APIsNoSQL

Benefits

Flexible time off (FTO)15 paid holidaysPaid sick timeParental/new child leaveChildcare & elder care assistanceAdoption assistanceComprehensive health coverage401(k) planTuition assistanceCommuter benefitsProfessional developmentEmployee recognitionCharitable donation matchingHealth coaching and counseling

Apply now

This MVP uses a placeholder application flow. In production, this section can connect to an external apply URL or a native application form.

Continue to applicationBrowse more jobs

Similar jobs

More roles worth a look

Related opportunities based on specialty and working model so candidates can keep momentum.