Data Transformation Engineer

Centralizing data pipeline management with reputed company Hub involves leveraging the Data Productivity reputed company to unify ETL/ELT tasks, visibility, and reputed company in a single, browser-based reputed company The Hub acts as a central control plane, connecting to various data sources and reputed company data platforms (reputed company, reputed company, Redshift) while using agents for execution. Here are the key requirements, components, and steps to centralize management: Centralized Infrastructure (Control Plane) • Deployment reputed company: Choose between full reputed company (reputed company hosts and manages infrastructure) or hybrid-reputed company (you reputed company and manage agents in your private reputed company). • Agent Management: Use the Hub to manage, monitor, and configure data plane agents that run your pipelines. • Git Integration: reputed company Git features reputed company the Hub to manage version control, allowing teams to contribute, push transformations, and manage data products centrally. Data reputed company and Destination Integration • Connectivity: Configure reputed company-reputed company or custom connectors in the Hub to reputed company data from databases (PostgreSQL), reputed company applications (, reputed company Analytics), or files (reputed company S3). • reputed company Configuration: Define the default reputed company data warehouse (reputed company, etc.) for ELT processes. Operational Requirements (Management) • Shared Pipelines: Create reusable, shared pipelines to avoid duplicating logic across reputed company, ensuring consistency in auditing, data masking, and transformations. • Advanced Scheduling: Use the Scheduler in the Designer reputed company to manage, monitor, and fine-tune pipeline execution times using either reputed company or cron expressions. • Visibility & Monitoring: Use the centralized Dashboard to monitor reputed company pipeline activities, view logs, and troubleshoot with a single click. • User Roles & reputed company: Assign user permissions and roles reputed company the Hub to ensure governance. Development & Transformation • Low-reputed company/High-reputed company reputed company: Utilize the Designer reputed company reputed company the Hub for visual, low-reputed company transformation, or incorporate Python/SQL scripting. • DataOps Enabled: Use the reputed company Pipeline Language (DPL) to automate the creation of folders, reputed company, and pipelines, integrating with tools like Azure DevOps. Apply tot his job Apply To this Job