Senior Azure Databricks Expert (Pune)

Job Title: Senior Azure Databricks Expert Job Summary This role involves designing, building, and optimizing enterprise-grade data platforms on Microsoft Azure. The ideal candidate will bring robust technical expertise in Azure Databricks, Spark, and data engineering, along with the ability to lead teams, mentor junior engineers, and drive best practices across global, cross-functional teams. Key Responsibilities - Provide technical leadership for Azure Databricks-based data transformation initiatives - Design, develop, and maintain scalable data pipelines using Azure Databricks - Build and optimize ETL/ELT solutions for enterprise data platforms - Lead migration and modernization efforts from SQL Server and legacy systems to Azure - Develop and optimize data processing workflows using Apache Spark, PySpark, and SQL - Improve performance of Spark workloads through deep understanding of Spark architecture (drivers, executors, partitions, stages, tasks) - Work with Azure Data Lake Storage Gen2 (ADLS Gen2) and Delta Lake for reliable data storage solutions - Optimize Databricks clusters, jobs, and workloads for performance and cost efficiency - Implement data governance and secure access using Unity Catalog and Azure security best practices - Collaborate with architects, DevOps teams, and business stakeholders to deliver scalable solutions - Troubleshoot and resolve production issues related to data pipelines and platform performance - Support CI/CD pipelines and Infrastructure as Code (IaC) implementations - Mentor junior engineers and provide technical guidance and best practices Must-Have Skills - Minimum 3+ years of hands-on experience with Azure Databricks - Strong expertise in Apache Spark, PySpark, and SQL - Experience in building and optimizing ETL/ELT pipelines - Hands-on experience with ADLS Gen2 and Delta Lake - Strong understanding of Spark architecture and performance tuning - Experience in migration from SQL Server or other relational databases - Practic .