Senior Specialist - Data Engineering at Vivantify Technology Solutions India Pvt. Ltd.

About this role

Location: Pune Experience: 7–11 Years About the Role We are seeking an experienced Data Engineer with 8+ years of expertise in Big Data Engineering, Databricks, Apache Spark, PySpark, Scala, and SQL . The role focuses on building and delivering scalable data solutions using Databricks, Delta Lake, Lakehouse architecture, AWS, and Power BI in a fast-paced Agile environment. Job Description The role involves designing, developing, and supporting modern data engineering solutions using Databricks and Apache Spark , with strong emphasis on PySpark, Scala, Python, SQL, and data modeling. The candidate will work with Delta Lake and Lakehouse architecture to build scalable distributed data processing solutions. The position requires hands-on experience with AWS Big Data services , including S3, Glue, EMR, DMS, Athena, RDS, Redshift, and Kinesis, as well as experience managing EMR clusters and working with relational databases such as PostgreSQL, Oracle, and SQL Server. The candidate will also contribute to data orchestration, streaming, CI/CD, source control, data cataloguing, and Power BI solutions. Experience with Agile at Scale, Continuous Integration, Test-Driven Development, and Infrastructure as Code practices will support successful delivery. Key
Design and develop scalable data engineering solutions using Databricks, Apache Spark, PySpark, Scala, and Python .
Build and maintain data pipelines using Delta Lake and Lakehouse architecture .
Develop complex SQL queries and implement effective data models across relational databases.
Work with AWS Big Data services and manage EMR clusters and associated services.
Develop data visualization and reporting solutions using Power BI .
Implement data orchestration, streaming, CI/CD, and source control practices.
Work with data cataloguing solutions such as Unity Catalog .
Collaborate with stakeholders in Agile environments and take ownership of data engineering deliverables.
Required Skills
Databricks
Apache Spark, PySpark, and Scala
Python
SQL and Data Modelling
Power BI
Delta Lake and Lakehouse architecture

AWS cloud platform
AWS services: S3, Glue, EMR, DMS, Athena, RDS, Redshift, Kinesis
EMR cluster management
Data orchestration tools

Kafka / streaming frameworks
RDBMS: PostgreSQL, Oracle, SQL Server
Complex SQL queries, particularly in PostgreSQL
Git and CI/CD pipelines
Azure DevOps / GitHub
Unity Catalog
Distributed computing principles
Agile, Continuous Integration, Test-Driven Development, and Infrastructure as Code practices
Preferred / Good to Have
Data warehousing concepts
Experience supporting Machine Learning pipelines in Databricks


MLflow experiment tracking and model registry
Databricks Mosaic AI, Vector Search, and embeddings pipelines
Generative AI concepts including LLMs, embeddings, and RAG

Terraform or CloudFormation
Debezium or other CDC tools
Jenkins-based workflow orchestration
Mandatory Skills Databricks Candidate Profile The ideal candidate will have 8+ years of Data Engineering experience , including strong hands-on expertise in Databricks, Spark, Python, SQL, AWS, and modern data architecture. The candidate should possess strong analytical and problem-solving abilities, communicate effectively with stakeholders, and demonstrate ownership while working in a fast-paced Agile environment. Benefits Package
Competitive salary and benefits package
Opportunities for professional growth and development
A collaborative and inclusive work environment
The opportunity to work on exciting and challenging projects with leading clients
Originally posted on Himalayas