Experience : 5 + Years
Location : Hyderabad
Job Description
We are seeking an experienced Azure Databricks + SQL Developer / Big Data Engineer to design, develop, and maintain scalable data solutions on Azure. The role will focus on building efficient ETL/ELT pipelines, optimizing SQL queries, and leveraging Databricks and other Azure services for advanced data processing, analytics, and data platform engineering. The ideal candidate will have a strong background in both traditional SQL development and modern big data technologies on Azure.
Key Responsibilities
- Develop, maintain, and optimize ETL/ELT pipelines using Azure Databricks (PySpark/Spark SQL).
- Write and optimize complex SQL queries, stored procedures, triggers, and functions in Microsoft SQL Server.
- Design and build scalable, metadata-driven ingestion pipelines for both batch and streaming datasets.
- Perform data integration and harmonization across multiple structured and unstructured data sources.
- Implement orchestration, scheduling, exception handling, and log monitoring for robust pipeline management.
- Collaborate with peers to evaluate and select appropriate tech stack and tools.
- Work closely with business, consulting, data science, and application development teams to deliver analytical solutions within timelines.
- Support performance tuning, troubleshooting, and debugging of Databricks jobs and SQL queries.
- Work with other Azure services such as Azure Data Factory, Azure Data Lake, Synapse Analytics, Event Hub, Cosmos DB, Streaming Analytics, and Purview when required.
- Support BI and Data Science teams in consuming data securely and in compliance with governance standards.
Required Skills & Experience
- 5–9 years of overall IT experience with at least 4+ years in Big Data Engineering on Microsoft Azure.
- Proficiency in Microsoft SQL Server (T-SQL) — stored procedures, indexing, optimization, and performance tuning.
- Strong experience with Azure Data Factory (ADF), Databricks, ADLS, PySpark, and Azure SQL Database.
- Working knowledge of Azure Synapse Analytics, Event Hub, Streaming Analytics, Cosmos DB, and Purview.
- Proficiency in SQL, Python, and either Scala or Java with debugging and performance optimization skills.
- Hands-on experience with big data technologies such as Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, and Elastic Search.
- Strong understanding of file formats such as Delta Lake, Avro, Parquet, JSON, and CSV.
- Solid background in data modeling, data transformation, and data governance best practices.
- Experience designing and building REST APIs with practical exposure to Data Lake or Lakehouse projects.
- Ability to work with large and complex datasets, ensuring data quality, governance, and security standards.
- Certifications such as DP-203: Data Engineering on Microsoft Azure or Databricks Certified Developer (DE) are a plus.