• 3+ years of experience as a Data Engineer
• Extensive experience with Databricks platform, including Delta Lake, Unity Catalog and Databricks SQL.
• Expertise in designing and implementing ETL pipelines.
• Comprehensive understanding of the end-to-end data engineering lifecycle, including data ingestion, transformation, storage, and retrieval.
• Advanced proficiency with Python, including libraries and frameworks commonly used in data engineering, such as Pandas, NumPy, PySpark, etc..
• Extensive experience with SQL databases, including advance querying and optimization techniques.
• Experience with data modeling tools and best practices.
• Proficiency in CI/CD pipelines and automation for data engineering workflows.
• Proficient in file processing techniques, managing large datasets, and experienced with various file formats such as CSV, JSON, Parquet, and Avro.
• Expertise in diverse data architectures, including data lakes, data warehouses, data marts, and data mesh.
• Highly skilled in transparently communicating project status, including updates on progress, challenges encountered, and upcoming steps.
• Strong analytical and problem-solving skills.
• In-depth knowledge of data governance, data quality management, and data lineage.

Main Responsibilities

• Design, develop, and maintain data pipelines to extract, transform, and load (ETL or ELT) data from various sources into data warehouses (Snowflake) or data lakes (Azure).
• Optimize data pipelines for performance, reliability, and scalability.
• Create and maintain data models, schemas, and database structures.
• Ensure data integrity, consistency, and security.
• Handle data quality issues, data lineage, and metadata management.
• Monitor and troubleshoot ETL/ELT jobs.
• Automate data workflows using tools like Stonebranch, Apache Airflow, or custom scripts.
• Profile and optimize SQL queries.
• Document data pipelines, processes, and workflows.
• Follow best practices for data engineering, version control, and code reviews.

• Work within an agile team to design, develop and deliver business value through data.
• Trouble shoot and solve critical production issues.
• Estimate work and meet the team’s commitments.