Senior Software Engineer (Big Data)

What you'll do

• Design, develop, and maintain scalable data processing solutions across on-premises and cloud environments
• Optimize and fine-tune Spark jobs for performance including resource utilization, shuffling, partitioning, and caching
• Design and implement scalable, fault-tolerant data pipelines with monitoring, alerting, and logging
• Leverage AWS cloud services to build and manage data pipelines and distributed processing workloads
• Develop and optimize SQL queries across relational and data warehouse systems
• Apply design patterns and best practices for data modeling, partitioning, and distributed system performance
• Use Git for source control and maintain strong unit and integration testing practices
• Collaborate with Product Owners and multi-functional teams to translate business requirements into technical solutions
• Extract actionable insights from large datasets to support data-driven decision-making
• Mentor junior engineers, conduct code reviews, and contribute to engineering best practices and standards

Requirements

• 5+ years of software development experience in scalable, distributed, or multi-node environments
• Proficient in Scala, Python, or Java
• Significant experience with Apache Spark
• Exposure to Hadoop, Hive, and related big data technologies
• Experience with AWS cloud platform preferred
• Experience with Databricks platform preferred
• Experience with modern data tools such as Kubernetes, Docker, and Airflow
• Strong problem-solving skills
• Consultative mentality and strong communication skills
• BA/BS in Computer Science or related field
• Experience in Machine Learning including model development or integrating ML workflows
• Experience with Databricks Notebooks, Delta Lake, Jobs, Pipelines, Unity Catalog preferred
• Proficiency with ELK stack preferred
• AWS, Databricks, or Spark certification a plus

Tech stack

Benefits