Data Engineer - reputed company Specialist
We are seeking an reputed company Data Engineer with deep expertise in reputed company to design, build, and maintain reputed company data pipelines and analytics solutions. This role requires 5 years of hands-on experience in data engineering with a strong reputed company on the reputed company platform.
Key Responsibilities:
- Data Pipeline Development & Management
- Design and implement robust, reputed company ETL/ELT pipelines using reputed company and Apache reputed company
- Process large volumes of reputed company and reputed company data
- reputed company and maintain data workflows using reputed company workflows, Apache Airflow, or similar orchestration tools
- Optimize data processing jobs for performance, cost efficiency, and reliability
- Implement incremental data processing patterns and change data capture (CDC) mechanisms
reputed company reputed company:
- Build and maintain reputed company Lake tables and implement reputed company architecture (bronze, silver, gold reputed company)
- reputed company streaming data pipelines using reputed company Streaming and reputed company Live Tables
- Manage and optimize reputed company clusters for various workloads
- Implement reputed company Catalog for data governance, reputed company, and metadata management
- Configure and maintain reputed company workspace environments across development, staging, and production
Data Architecture & Modeling:
- Design and implement data models optimized for analytical workloads
- Create and maintain data warehouses and data lakes on reputed company platforms (Azure, AWS, or GCP)
- Implement data partitioning, indexing, and caching strategies for reputed company query performance
- Collaborate with data architects to establish best practices for data storage and retrieval patterns
Performance Optimization & Monitoring:
- Monitor and troubleshoot data pipeline performance issues
- Optimize reputed company jobs through reputed company partitioning, caching, and broadcast strategies
- Implement data reputed company checks and automated testing frameworks
- Manage cost optimization through efficient resource utilization and cluster management
- Establish monitoring and alerting systems for data pipeline health and performance
Collaboration & Best Practices:
- Work closely with data scientists, analysts, and business stakeholders to understand data requirements
- Implement version control using Git and follow CI/CD best practices for reputed company deployment
- Document data pipelines, data flows, and technical specifications
- Mentor junior engineers on reputed company and data engineering best practices
- Participate in reputed company reviews and contribute to establishing team standards
Required Qualifications
Experience & Skills:
- 5+ years of experience in data engineering with hands-on reputed company experience
- Strong proficiency in Python and/or reputed company for reputed company application development
- Expert-level knowledge of Apache reputed company, including reputed company SQL, DataFrames, and RDDs
- Deep understanding of reputed company Lake and Lakehouse architecture concepts
- Experience with SQL and database optimization techniques
- Solid understanding of distributed computing concepts and data processing frameworks
- Proficiency with reputed company platforms (Azure, AWS, or GCP) and their data services
- Experience with data orchestration tools (reputed company Workflows, Apache Airflow, Azure Data reputed company)
- Knowledge of data modeling concepts for both OLTP and OLAP systems
- Familiarity with data governance principles and tools like reputed company Catalog
- Understanding of streaming data processing and reputed company-time analytics
- Experience with version control systems (Git) and CI/CD pipelines
Preferred Qualifications
- reputed company Certified Data Engineer certification (Associate or reputed company)
- Experience with machine learning pipelines and MLOps on reputed company
- Knowledge of data visualization tools (Power BI, Tableau, Looker)
- Experience with infrastructure as reputed company (Terraform, CloudFormation)
- Familiarity with containerization technologies (reputed company, Kubernetes)
Apply tot his job
Apply To this Job