As a Data Engineer, you’ll be responsible for designing, building, and maintaining data pipelines, databases, and data warehouses. Your work ensures that data is collected, processed, and made accessible for analysis and reporting. You’ll work with multidisciplinary teams following an agile methodology.

• 3+ years of experience as a Data Engineer
• Extensive experience with Databricks platform, including Delta Lake, Unity Catalog and Databricks SQL.
• Expertise in designing and implementing ETL pipelines.
• Comprehensive understanding of the end-to-end data engineering lifecycle, including data ingestion, transformation, storage, and retrieval.
• Advanced proficiency with Python, including libraries and frameworks commonly used in data engineering, such as Pandas, NumPy, PySpark, etc..
• Extensive experience with SQL databases, including advance querying and optimization techniques.
• Experience with data modeling tools and best practices.
• Proficiency in CI/CD pipelines and automation for data engineering workflows.
• Proficient in file processing techniques, managing large datasets, and experienced with various file formats such as CSV, JSON, Parquet, and Avro.
• Expertise in diverse data architectures, including data lakes, data warehouses, data marts, and data mesh.
• Highly skilled in transparently communicating project status, including updates on progress, challenges encountered, and upcoming steps.
• Strong analytical and problem-solving skills.
• In-depth knowledge of data governance, data quality management, and data lineage.

Main Responsibilities

• Design, develop, and maintain data pipelines to extract, transform, and load (ETL or ELT) data from various sources into data warehouses (Snowflake) or data lakes (Azure).
• Optimize data pipelines for performance, reliability, and scalability.
• Create and maintain data models, schemas, and database structures.
• Ensure data integrity, consistency, and security.
• Handle data quality issues, data lineage, and metadata management.
• Monitor and troubleshoot ETL/ELT jobs.
• Automate data workflows using tools like Stonebranch, Apache Airflow, or custom scripts.
• Profile and optimize SQL queries.
• Document data pipelines, processes, and workflows.
• Follow best practices for data engineering, version control, and code reviews.

• Work within an agile team to design, develop and deliver business value through data.
• Trouble shoot and solve critical production issues.
• Estimate work and meet the team’s commitments.