Job Description
Job DescriptionWe are looking for a skilled Data Engineer to join our team in Malvern, Pennsylvania. In this role, you will be instrumental in designing, developing, and maintaining robust data integration processes using Python and Azure Synapse Analytics. By collaborating with cross-functional teams, you will ensure the delivery of high-quality data solutions that empower business insights and decision-making.
Responsibilities:
• Design and implement data integration workflows using Python (PySpark) and Azure Synapse Analytics to support data extraction, transformation, and loading processes.
• Develop and optimize data storage solutions such as data warehouses and lakehouses, employing best practices in data modeling, including star schemas, facts, and dimensions.
• Extract and transform data from diverse sources, including APIs, database tables, and structured files, ensuring seamless data integration.
• Leverage Azure Synapse Analytics features, such as Notebooks and Pipelines, to create scalable, high-performance data solutions.
• Contribute to the adoption and implementation of advanced data management concepts, including data lakes, delta lakes, and data cataloging.
• Collaborate with data architects to define and implement efficient data models aligned with organizational needs.
• Conduct data quality assessments and implement validation procedures to maintain data integrity and reliability.
• Monitor and troubleshoot data pipelines to ensure optimal performance and resolve any technical issues.
• Document data engineering processes, workflows, and transformations to facilitate knowledge sharing and operational continuity.
• Ensure compliance with data governance policies and implement security measures to protect sensitive information.• Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent work experience.
• Proven expertise in data engineering with hands-on experience in Python (PySpark) for data integration tasks.
• Proficiency in using Azure Synapse Analytics tools, including Notebooks, Pipelines, and Linked Services.
• Strong SQL skills, including the ability to write complex queries and optimize query performance.
• Familiarity with version control systems, such as Git or Azure DevOps.
• Knowledge of data modeling techniques and experience in designing scalable data architectures.
• Excellent problem-solving and analytical skills with a strong attention to detail.
• Effective communication and teamwork abilities, with a collaborative mindset.