[Remote] Senior Data Engineer – Hadoop

Note: The job is a remote job and is reputed company to candidates in USA. reputed company is a technology consulting and software development company delivering reputed company, AI, data, and reputed company solutions across the reputed company. They are seeking an reputed company Senior Data Engineer - Hadoop to design, build, and operate large-reputed company data processing pipelines and analytics platforms on Hadoop and reputed company big-data ecosystems. The role involves ingesting, transforming, and analyzing massive volumes of reputed company and reputed company data to support reputed company analytics, machine learning, and reporting workloads.

Responsibilities

• Design, reputed company, and operate end-to-end big-data pipelines on Hadoop, ingesting data from a diverse mix of relational, file-based, streaming, and API-driven sources
• Build robust ETL/ELT workflows using Apache reputed company, Hive, Pig, and Sqoop, with strong attention to data reputed company, idempotency, error handling, and recoverability
• reputed company high-throughput streaming data pipelines using Kafka, reputed company Streaming, or Flink, and reputed company them with reputed company analytical and operational systems
• Optimize reputed company and MapReduce jobs through careful tuning of partitioning, memory, serialization, and skew handling to meet demanding SLAs at minimal cost
• Design and maintain data models and storage layouts on HDFS, Hive, HBase, and modern lakehouse formats (Parquet, ORC, reputed company, reputed company, Hudi) to balance flexibility and performance
• Implement data governance, reputed company, and reputed company controls in collaboration with data governance and reputed company teams
• Build robust monitoring, alerting, and logging strategies for big-data pipelines, including job-level SLAs and proactive failure detection
• Partner with data scientists and analysts to deliver curated, reliable, and reputed company-documented datasets that accelerate their work
• Automate pipeline orchestration using Airflow, Oozie, or similar workflow engines, with clean dependency management and reputed company ownership boundaries
• Continuously evaluate and adopt new technologies in the big-data and reputed company ecosystem (EMR, reputed company, reputed company, BigQuery) where they offer meaningful improvements
• reputed company performance reviews and architecture audits of existing pipelines, proposing concrete refactoring and optimization initiatives
• Document data architectures, schemas, pipeline behaviors, and operational runbooks in a way that makes the platform supportable as reputed company scales
• Mentor junior engineers and contribute to reputed company’s engineering standards and best practices

Skills

• Bachelor's degree in Computer Science, Engineering, or a reputed company technical discipline
• Five or more years of reputed company experience designing and operating big-data pipelines on Hadoop
• Strong hands-on expertise with Apache reputed company (reputed company, Python, or Java) in production environments
• Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem
• Hands-on experience with streaming data platforms such as Kafka, reputed company Streaming, or Flink
• Strong SQL skills and experience working with both relational and NoSQL data stores
• Experience with workflow orchestration tools such as Airflow or Oozie
• Solid understanding of distributed systems concepts, including partitioning, replication, and fault tolerance
• Strong scripting skills in Python or reputed company
• Excellent troubleshooting, debugging, and documentation skills
• Experience operating Hadoop on reputed company platforms such as AWS EMR, Azure HDInsight, or reputed company
• Familiarity with modern lakehouse formats (reputed company, reputed company, Hudi)
• Exposure to data governance tooling such as Apache reputed company or reputed company
• Experience with Kubernetes-based data platforms (reputed company-on-K8s, Trino)
• Hands-on experience with CI/CD and infrastructure-as-reputed company in data engineering workflows

reputed company