AdTechTalent
Engineering7 days agoHybrid

Magnite

Data Engineer II

apache sparkpysparkpythondatabrickskafkasqlawsterraformjenkinsgithub actionsgolangkubernetesdata engineeringdata pipelinesdistributed systemsetlstreamingbatch processing

Key details

Salary

$120K – $135K

Employment type

Full-time

Seniority

Mid-level

Years experience

0-2

Location

New York, US

Full job description

Data Engineer II role focused on building and optimizing large-scale data pipelines using Apache Spark and Python, processing billions of events in streaming and batch modes. Collaborate with senior engineers to improve data pipelines, ensure operational performance and uptime, and provision infrastructure with Terraform and AWS. Required skills include hands-on experience with Apache Spark (PySpark), strong Python skills, data pipeline design, familiarity with Kafka, SQL, AWS, Terraform, Jenkins or Github Actions, GoLang, and Kubernetes. The role supports career growth and offers a hybrid work schedule with two remote days per week. Salary range is $120,000 to $135,000 based on location and experience. Benefits include healthcare, time off, equity, 401k, parental leave, and more.

What you'll do

  • Build and optimize data pipelines using Apache Spark and Python processing billions of events in streaming and batch modes
  • Collaborate with senior data engineers to improve Audience data pipelines with innovative solutions
  • Own operational performance of builds ensuring uptime and recovery strategies
  • Collaborate with SRE teams to provision infrastructure using Terraform and AWS, with future Kubernetes on-premise
  • Proactively identify and solve impactful problems

Requirements

  • Hands-on experience with Apache Spark (mainly PySpark) and production pipeline deployment
  • Strong Python skills, especially with Databricks and Spark
  • Data pipeline design knowledge from event data sourcing to queryable data layers
  • Familiarity with event-based data architecture such as Kafka
  • SQL experience for data analysis and transformations
  • Hands-on experience with AWS and Terraform for IaaS
  • Experience writing Jenkins or Github Actions pipelines for build and integration
  • Knowledge of GoLang for distributed service development
  • Kubernetes experience
  • Willingness and eagerness to learn and grow

Tech stack

Apache SparkPySparkPythonDatabricksKafkaSQLAWSTerraformJenkinsGithub ActionsGoLangKubernetesRustJavaSpring BootECSFargateLuigiAirflowDockerALBHAProxyRDSMariaDBPostgresRedisMemcacheSNSSQSJavaScriptTypeScriptReactLinux

Benefits

Comprehensive Healthcare Coverage from Day OneGenerous Time OffHoliday Breaks and Quarterly Wellness DaysEquity and Employee Stock Purchase PlanFamily-Focused Benefits and Parental Leave401k Retirement Savings Plan with Employer MatchDisability and Life InsuranceCell Phone SubsidyMonthly Fitness and ReimbursementAnnual performance-based bonus

Apply now

This MVP uses a placeholder application flow. In production, this section can connect to an external apply URL or a native application form.

Continue to applicationBrowse more jobs

Similar jobs

More roles worth a look

Related opportunities based on specialty and working model so candidates can keep momentum.