Data Engineering (Palantir, Spark, PySpark, Python)- W2 Role

Technical Lead -- Data Engineering (Palantir, Spark, PySpark, Python)

W2 role OPT with 7 years exp is fine ++Onsite work :One or two days in a month++

Job Summary

We are looking for a hands-on Technical Lead in Data Engineering to drive the design, development, and delivery of scalable data solutions in a large retail enterprise. This role requires strong expertise in Palantir Foundry, Spark/PySpark, SQL, and Python, along with the ability to lead engineering teams and partner with business stakeholders across supply chain, merchandising, and store operations.

Key Responsibilities

• Lead the design and implementation of scalable data platforms and pipelines using PySpark, Spark, and Python
• Drive adoption and best practices for Palantir Foundry (data pipelines, ontology, workflows, and operational applications)
• Architect and optimize high-performance data processing solutions for large-scale datasets
• Provide technical leadership and mentorship to data engineers, ensuring code quality and best practices
• Collaborate with cross-functional teams (business, analytics, data science) to translate requirements into scalable solutions
• Design robust data models, ETL/ELT frameworks, and data integration strategies
• Ensure data quality, governance, security, and compliance across enterprise data platforms
• Lead performance tuning, troubleshooting, and optimization of data pipelines
• Drive CI/CD implementation, code reviews, and release management
• Stay current with emerging data engineering technologies and recommend improvements

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or related field
• 8 years of experience in data engineering, with at least 2--3 years in a technical leadership role
• Strong hands-on experience with:
  • Python & PySpark
  • Apache Spark
  • Advanced SQL
• Experience with Palantir Foundry or similar modern data platforms
• Deep understanding of data engineering principles (ETL/ELT, data modeling, distributed systems)
• Experience designing and managing large-scale data architectures
• Strong leadership, communication, and stakeholder management skills

Preferred Qualifications

• Experience in retail domain (supply chain, inventory, merchandising, store analytics)
• Experience with cloud platforms (Azure, AWS, or GCP)
• Familiarity with orchestration tools (Airflow, Azure Data Factory, etc.)
• Experience with real-time/streaming data pipelines
• Exposure to DevSecOps practices

Key Skills

• Technical Leadership
• Data Engineering Architecture
• PySpark & Spark
• Python Programming
• SQL Optimization
• Palantir Foundry
• Data Modeling & Warehousing
• CI/CD & DevOps

Flexible work from home options available.