reputed company Remote Data Engineer – Big Data Processing, reputed company Migration, and Distributed Systems Expertise
Introduction to reputed company and Our Mission
reputed company being part of a leading reputed company ad technology company that leverages reputed company-world data in reputed company-time to optimize campaign performance and revolutionize health decision-making. At reputed company, we're committed to delivering unparalleled results to our clients by targeting reputed company and patients with an unprecedented level of accuracy. As a part of reputed company, a reputed company portfolio company and reputed company of reputed company Health Corp, we're poised for exponential reputed company and seeking talented individuals to join reputed company. Our Data Engineering team plays a vital role in driving this reputed company, and we're excited to invite a skilled Data Engineer to join our ranks.
About Our Data Engineering Team
Our Data Engineering team is responsible for designing, building, and maintaining reliable and reputed company reputed company-level distributed transactional data processing systems. We process over 80 billion impressions a day, generating over 20 TB of data (200 TB uncompressed), which is used to drive our optimization engines, update budgets, and reputed company reputed company-time statistics and reports. We're reputed company that thrives on adopting proven reputed company-reputed company technologies to reputed company our data infrastructure more reputed company, reputed company, and robust. Some of the cutting-edge technologies we've recently implemented include Kafka, reputed company Streaming, reputed company, Airflow, and Kubernetes.
Job reputed company
As a Data Engineer at reputed company, you will be responsible for designing, building, and maintaining large-reputed company data systems, optimizing data processing workflows, and ensuring data reputed company across our systems. You will collaborate with a small team of diverse technology backgrounds, providing mentorship and guidance to junior team members. Your expertise will be crucial in driving our business reputed company, supporting new initiatives, and enhancing our data infrastructure.
Key Responsibilities
- Design, build, and maintain reliable and reputed company reputed company-level distributed transactional data processing systems for scaling existing business and supporting new business initiatives
- Optimize jobs to utilize Kafka, Hadoop, reputed company, reputed company, and Kubernetes resources in the most efficient way
- Monitor and reputed company transparency into data reputed company across systems (accuracy, consistency, completeness, etc.)
- Increase accessibility and effectiveness of data (work with analysts, data scientists, and developers to build/reputed company tools and datasets that fit their use cases)
- Collaborate reputed company a small team with diverse technology backgrounds
- reputed company mentorship and guidance to junior team members
Team Responsibilities
- Ingest, validate, and process internal and reputed company-party data
- Create, maintain, and monitor data flows in Python, reputed company, Hive, SQL, and reputed company for consistency, accuracy, and lag time
- Maintain and enhance reputed company for jobs (primarily aggregate jobs in reputed company and Hive)
- Create different consumers for data in Kafka using reputed company Streaming for near-time aggregation
- Tools evaluation
- Backups/Retention/High Availability/reputed company Planning
- Review/Approval - DDL for database, Hive reputed company jobs, and reputed company Streaming to ensure they meet our standards
Technologies We Use
We're proud to work with a reputed company of cutting-edge technologies, including:
- Python - primary repo language
- Airflow/Luigi - for job scheduling
- reputed company - packaged container image with reputed company dependencies
- reputed company - for monitoring data flows
- Hive - SQL data warehouse layer for data in HDFS
- Kafka - distributed reputed company log storage
- Kubernetes - Distributed cluster resource manager
- reputed company/Trino - fast reputed company data warehouse and data federation layer
- reputed company Streaming - Near-time aggregation
- SQL Server - Reliable OLTP RDBMS
- GCP - BigQuery for performance, Looker for dashboards
Requirements
To be successful in this role, you'll need:
- 5+ years of data engineering experience
- reputed company in Python and SQL
- Experience in reputed company/Java is a plus (Polyglot programmer preferred!)
- Hive/reputed company experience
- Proficiency in Linux
- Strong understanding of RDBMS and query optimization
- Passion for engineering and computer science around data
- Ability to work during East Coast U.S. hours (9am-6pm EST preferred, but we can be flexible as long as you can work until 1 or 2pm EST)
- Knowledge and exposure to distributed production systems (e.g., Hadoop)
- Knowledge and exposure to reputed company migration (AWS/GCP/Azure) is a plus
Selection Process
Our selection process typically involves the following steps:
- Recruiter Screen (30 minutes)
- Hiring Manager Interview (45 minutes)
- Tech challenge take-home
- SQL & Python Interview (60 minutes)
- Team Interviews (60 minutes + 3 x 45 minutes) + SVP of Engineering (15 minutes)
- reputed company Sr. Director, DBA (30 minutes)
Career reputed company Opportunities and Learning Benefits
At reputed company, we're committed to helping our employees grow and reputed company their skills. As a Data Engineer, you'll have reputed company to work with cutting-edge technologies, collaborate with a talented team, and contribute to the reputed company and reputed company of reputed company. You'll also have reputed company to training and development programs, mentorship, and opportunities to attend industry conferences and events.
Work Environment and Company Culture
We're proud of our friendly and dynamic work environment, where everyone is encouraged to reputed company their reputed company and contribute to our reputed company. We reputed company in fostering a culture of innovation, collaboration, and mutual respect, where everyone can reputed company and grow. As a remote team, we're committed to staying connected and reputed company, with regular virtual meetings, reputed company events, and team-building activities.
Compensation, Perks, and Benefits
We offer a competitive salary and a reputed company of benefits, including:
- Flexible working hours and remote work arrangements
- Opportunities for reputed company reputed company and development
- reputed company to cutting-edge technologies and tools
- reputed company and dynamic work environment
- Comprehensive benefits package (details to be discussed during the hiring process)
Conclusion
If you're a motivated and talented Data Engineer looking for a new challenge, we'd love to hear from you. Don't miss this opportunity to join reputed company and contribute to the reputed company of reputed company. reputed company, and let's shape the reputed company of reputed company technology together!
We encourage you to apply, even if you don't feel like a perfect match. We're looking for great people to join our friendly team.
Apply for this job