AdTechTalent
Engineering1 month agoHybrid

Epsilon

Lead Software Engineer

scalaapache sparkhadooppythonsqlbig datadatabrickselk stackdockerkubernetesawsgcpdata pipelinesdata engineeringagilementoring

Key details

Salary

Not specified

Employment type

Full-time

Seniority

Lead

Years experience

10+

Location

Bengaluru, India

Full job description

Lead Engineer role in Epsilon's Attribution/Forecasting Product Development team. Requires over 8 years of experience in Scala and Apache Spark for building scalable big data solutions on-premises and cloud (AWS, GCP). Responsibilities include developing and optimizing data processing solutions, building and maintaining workflows, data integration, troubleshooting, and enhancing attribution and forecasting engines. Requires strong knowledge of Hadoop ecosystem, SQL, Python, GIT, unit and integration testing, and Agile methodologies. Experience mentoring junior staff is expected. Advantageous skills include Databricks, ELK stack, data visualization tools, Docker, and Kubernetes.

What you'll do

  • Build and scale services in on-premises and cloud environments
  • Implement and optimize advanced data processing solutions using Scala, Spark, and Hadoop
  • Collaborate with multi-functional teams to deploy scalable big data solutions
  • Build, schedule, and maintain complex workflows
  • Perform data integration and transformation tasks
  • Troubleshoot issues and document processes
  • Communicate technical concepts to technical and non-technical stakeholders
  • Enhance attribution and forecasting engines to meet evolving business needs
  • Work within Agile methodologies (SCRUM)
  • Mentor junior staff

Requirements

  • Over 8 years of experience in Scala programming
  • Extensive use of Apache Spark for scalable big data solutions
  • Experience with on-premises and cloud environments (AWS, GCP)
  • Performance tuning of Spark jobs
  • Implementing scalable, fault-tolerant data pipelines with monitoring and alerting
  • Hands-on experience with Python for infrastructure modules
  • Deep understanding of Hadoop ecosystem (HDFS, YARN, MapReduce)
  • Proficient in writing efficient SQL queries for large data volumes
  • Experience building, scheduling, and maintaining DAG workflows
  • Familiarity with data warehousing concepts and technologies
  • Experience with source control systems like GIT
  • Developing and implementing unit and integration tests
  • Strong collaboration and communication skills
  • Ability to work under pressure and meet deadlines
  • Experience mentoring junior staff
  • Advantageous: experience with Databricks, ELK stack, analytics, data visualization tools, Docker, Kubernetes

Tech stack

ScalaApache SparkHadoopPythonSQLGITDatabricksDelta LakeELK StackTableauPower BIGrafanaDockerKubernetesAWSGCP

Benefits

Employee well-being focusCollaborative work environmentOpportunities for growth, learning, and career advancementInnovation-driven cultureWork-life balance and flexibilityDiversity and inclusion commitment

Apply now

Ready to take the next step in your career? Click the button below to continue to the application process.

Continue to applicationBrowse more jobs

Similar jobs

More roles worth a look

Related opportunities based on specialty and working model so candidates can keep momentum.