Job Summary
Responsible for identification, assessment, and design of specific data engineering solutions, infrastructure, and systems to support data-driven decision-making and analysis. They enable the organization to effectively manage, process, and analyze data, thereby supporting data-driven decision-making and unlocking valuable insights from data.
Key Requirements
- 3+ years’ experience with SparkSQL, Python, and PySpark for data engineering workflow
- Strong proficiency in dimensional modeling and star schema design for analytical workloads
- Experience implementing automated testing and CI / CD pipelines for data workflows
- Familiarity with GitHub operations and collaborative development practices
- Ability to optimize engineering workflow jobs for performance and cost efficiency
- Experience with cloud data services and infrastructure (AWS, Azure, or GCP)
- Proficiency with IDE tools such as Visual Studio Code
- Experience with Databricks platform is a plus
Key Accountabilities
Designs, develops, and validates data processes. Develops data pipelines and supports their implementation, ensuring data solutions align with business objectives. Looks ahead to understand future technology options for the business. Serves as a technical expert in specific processes or product areas, conducting process reviews and initiating changes to contribute to continuous improvement of services, efficiency, and quality. Researches external primary data sources, selects relevant information, evaluates key technology themes, and makes recommendations to inform policy and / or product development in the IT area.
Key Responsibilities
Design and implement ETL / ELT pipelines using Spark SQL and Python within Databricks Medallion architectureDevelop dimensional data models following star schema methodology with proper fact and dimension table design, SCD implementation, and optimization for analytical workloadsOptimize Spark SQL and DataFrame operations through partitioning strategies, clustering, and join optimizations to maximize performance and minimize costsBuild comprehensive data quality frameworks with automated validation checks, statistical profiling, exception handling, and data reconciliation processesEstablish CI / CD pipelines incorporating version control and automated testing (unit, integration, smoke tests, etc.)Implement data governance standards, including row-level and column-level security policies for access control and complianceCreate and maintain technical documentation such as ERDs, schema specifications, data lineage diagrams, and metadata repositoriesAt Zurich, we like to think outside the box and challenge the status quo. We focus on the positives and ask, "What can go right?"
We are an equal opportunity employer who values the uniqueness of each employee—it's what makes our team great!
Join us as we explore new ways to protect our customers and the planet.
Location(s) : ID - Head Office - MT Haryono#J-18808-Ljbffr