AdTechTalent
Engineering8 days agoHybrid

Magnite

Senior Big Data Engineer

big dataapache kafkahadoopmapraerospikesparklinuxrocky linuxpostgresqlmysqlscylladbbashpythonterraformansiblepuppetgrafanaprometheusgraphitekubernetescephtcp/ipdnsload balancinginfrastructure as codemonitoringautomationdevopscloudaws

Key details

Salary

$170K – $180K

Employment type

Full-time

Seniority

Senior

Years experience

3-5

Location

New York, US

Full job description

Senior Big Data Engineer role on Infrastructure Engineering team responsible for designing, building, maintaining, and modernizing enterprise big data platforms for large-scale data processing and analytics. Requires extensive experience with distributed data platforms such as Apache Kafka, Hadoop/MapR, Aerospike, and Linux-based infrastructure. Responsibilities include managing distributed technologies, building scalable and resilient data infrastructure, capacity planning, performance tuning, troubleshooting, automating infrastructure deployment using Infrastructure as Code, collaborating with engineering teams, implementing monitoring and alerting, and participating in production support and on-call rotations. Requires 4+ years supporting large-scale Linux infrastructure and administering distributed Big Data platforms, strong Linux system administration, experience with relational databases (PostgreSQL, MySQL), shell scripting, monitoring platforms, networking fundamentals, and working in 24x7 production environments. Preferred qualifications include Kubernetes, Ceph, Terraform, Puppet, Ansible, AWS, and CI/CD familiarity. Salary range in NY is $170,000 – $180,000 plus bonus, equity, and comprehensive benefits. Hybrid schedule with remote work on Monday and Friday, in-office Tuesday through Thursday.

What you'll do

  • Design, deploy, administer, and optimize enterprise Big Data platforms
  • Manage and support distributed technologies including Apache Kafka, Hadoop / MapR, Aerospike
  • Build highly available, scalable, and resilient data infrastructure across multiple data centers
  • Perform capacity planning, performance tuning, troubleshooting, and root cause analysis for production environments
  • Automate infrastructure deployment, configuration management, and operational workflows using Infrastructure as Code and scripting
  • Collaborate with Software Engineering, Data Engineering, SRE, and Database teams to support new applications and platform requirements
  • Implement monitoring, alerting, and observability solutions to ensure platform health and availability
  • Participate in production support and on-call rotations for critical infrastructure

Requirements

  • 4+ years of experience supporting large-scale Linux infrastructure
  • 4+ years of experience administering distributed Big Data platforms
  • Strong experience with Apache Kafka, Hadoop (MapR, Cloudera, Apache Hadoop, or similar), Aerospike
  • Strong Linux system administration experience (RHEL, Rocky Linux, CentOS)
  • Experience with relational databases such as PostgreSQL and MySQL
  • Experience troubleshooting distributed systems in production environments
  • Strong knowledge of networking fundamentals including DNS, TCP/IP, load balancing, and security
  • Experience with shell scripting (Bash, Python, or similar)
  • Experience with monitoring platforms such as Grafana, Prometheus, Graphite, or similar
  • Experience working in highly available, 24x7 production environments
  • Ability to work independently in a fast-paced production environment
  • Passion for automation, operational excellence, and continuous improvement
  • Preferred: Experience with Kubernetes and containerized workloads
  • Preferred: Experience with Ceph or other distributed storage platforms
  • Preferred: Experience with Infrastructure as Code tools such as Terraform or Spacelift
  • Preferred: Experience with configuration management tools such as Puppet or Ansible
  • Preferred: Experience with cloud platforms (AWS)
  • Preferred: Familiarity with CI/CD pipelines and DevOps practices

Tech stack

Apache KafkaHadoopMapRAerospikeSparkRocky LinuxPostgreSQLMySQLScyllaDBBashPythonTerraformAnsiblePuppetGrafanaPrometheusGraphiteKubernetesCephTCP/IPDNSLoad Balancing

Benefits

Comprehensive Healthcare Coverage from Day OneGenerous Time OffHoliday Breaks and Quarterly Wellness DaysEquity and Employee Stock Purchase PlanFamily-Focused Benefits and Parental Leave401k Retirement Savings Plan with Employer MatchDisability and Life InsuranceCell Phone SubsidyMonthly Fitness and ReimbursementAnnual performance-based bonus

Apply now

Ready to take the next step in your career? Click the button below to continue to the application process.

Continue to applicationBrowse more jobs

Similar jobs

More roles worth a look

Related opportunities based on specialty and working model so candidates can keep momentum.