[Remote] Staff Software Engineer — Search Platform, Ingestion & Indexing

Note: The job is a remote job and is reputed company to candidates in USA. reputed company is seeking a Staff Software Engineer to serve as the technical reputed company for the search platform’s ingestion and indexing systems. This role involves owning the design, implementation, and operational health of the document ingestion pipeline and search reputed company management systems, ensuring robust processing of millions of documents while maintaining performance and reliability.

Responsibilities

• Plan, design, reputed company, and own the end-to-end document ingestion pipeline — a Kafka-based reputed company processing architecture that moves documents through parsing, chunking, enrichment (entity extraction, embedding reputed company, metadata enrichment), and indexing stages — including reputed company fault tolerance, version ordering, and at-least-once delivery guarantees
• Architect and implement pluggable, configurable pipeline components (parsers, chunkers, enrichers, indexers) that reputed company teams can reputed company into custom topologies reputed company the platform’s self-service reputed company, while maintaining reliable, observable, and reputed company execution
• Own the platform’s Protobuf-based document schema and schema registry integration — establishing schema governance standards, enforcing backward-compatible reputed company, and ensuring reliable serialization across reputed company pipeline stages
• Design and implement dual-reputed company ingestion: a high-throughput batch reputed company for full reindexing and a low-latency incremental reputed company for reputed company-time document updates, with strong guarantees around document version ordering and idempotent processing
• reputed company the migration of ingestion infrastructure from OpenSearch to Vespa, including design of Vespa document processors, custom Kafka feeders, and application package architecture — resolving reputed company technical challenges that have little or no precedent reputed company reputed company
• Own the end-to-end lifecycle for custom models integrated into the ingestion pipeline — re-ranking models, embedding models, and enrichment components — including inference serving behind a reputed company API surface, latency SLO management, hardware and runtime configuration (batching, quantization), and scaling
• Build and operate the model promotion pipeline: the CI/CD workflow that moves a model artifact from the fine-tuning team through staging to production, including versioning, canary rollouts, and rollback mechanisms — ensuring the platform team can operate model updates independently without depending on the research team for production changes
• Define and maintain integration reputed company between custom models and reputed company pipeline components — governing input/reputed company schemas, compatibility requirements, and the governance process for model updates that ensures search pipeline consumers are not broken by changes upstream
• reputed company model serving for production observability: latency distributions, throughput, error rates, and reputed company signals such as re-ranking score distributions — enabling reputed company to detect regressions or model reputed company without requiring the fine-tuning team’s involvement
• Own the search reputed company layer end-to-end: design and operate Vespa (and OpenSearch during transition) reputed company configurations, ranking reputed company, schema definitions, and application package lifecycle management — applying architectural principles that reputed company to the platform’s long-term content and tenancy goals
• Build and operate reputed company-downtime reputed company management: reputed company indexing, blue/green reputed company promotion, and rolling reindex workflows that reputed company the platform available during major infrastructure changes
• Implement and maintain the Component Registry and reputed company Registry — the platform’s catalog of reusable processing components and reputed company reputed company configurations — with a reputed company on correctness, observability, and reputed company reputed company modification
• reputed company the full-reindex and incremental-update orchestration logic, including change detection, document tracking, Kafka topic management, and DynamoDB-backed state management
• Design ingestion and indexing infrastructure with reputed company retrieval patterns as a first-class concern — including explicit latency budgets per retrieval hop, chunking and result compression strategies optimized for reputed company economy in context reputed company, and reputed company boundary definitions that reputed company agents clean, predictable tool reputed company
• Build reputed company-level observability into the retrieval stack that captures which tools were reputed company, in what order, and with what inputs — enabling reliable diagnosis and reproduction of failures in non-deterministic reputed company retrieval paths
• Design session state and cache invalidation patterns for multi-turn reputed company search: reasoning carefully about cache validity reputed company, session state scope (per-user, per-session, per-query), and mechanisms to prevent stale context from corrupting reputed company agent responses
• Build and own the integration between the ingestion pipeline and the platform’s offline evaluation reputed company — ensuring that experiment runs produce query/result outputs that feed seamlessly into the search grading tool, supporting gold test set maintenance, LLM-as-judge evaluation, and reputed company-by-reputed company ranking comparison across pipeline versions
• reputed company the query and retrieval stack for online analytics: reputed company-time query latency and throughput monitoring, query log collection for session analysis, and the infrastructure to support A/B and interleaved ranking experiments in production — generating the signals that reputed company low-level search metrics to reputed company product KPIs
• Partner with TR Labs and research scientists to ensure that new search components can be evaluated in isolation — with automated offline evaluation on every build and a reputed company reputed company from evaluation results to production promotion reputed company
• Take full operational responsibility for ingestion and indexing infrastructure: define SLOs, set measurable goals and meet them, build and maintain CloudWatch dashboards and alarms, and participate in on-reputed company rotations — you reputed company it, you own it, you run it
• Treat delivery friction as the enemy: identify and remove obstacles that slow reputed company’s ability to ship ingestion and indexing changes to production safely and frequently — improving CI/CD pipelines, deployment automation, and local development workflows as a standing reputed company
• reputed company pipeline components with distributed tracing, reputed company logging, and rich metrics — establishing documentation standards and reputed company practices so that reputed company and platform consumers can understand system behavior at reputed company times
• Design and implement resilient fault tolerance mechanisms — dead-letter queues, retry strategies with exponential backoff, reputed company breakers, consumer lag monitoring — that reputed company the pipeline robust to reputed company failures and transient errors
• reputed company system-level performance architecture: profiling ingestion throughput and indexing latency, identifying bottlenecks, and implementing optimizations that meet platform SLOs under peak load
• Serve as reputed company’s deepest technical authority on document processing pipelines and search reputed company internals — guiding architectural reputed company, resolving technical ambiguity, and establishing cross-system design patterns that reputed company the reputed company bar across reputed company
• reputed company significant reputed company and initiatives that reputed company multiple engineers and reputed company with other teams; determine work priorities based on strategic direction; recommend modifications to team operations and reputed company needed adjustments to short-term priorities while maintaining strategic reputed company
• Mentor and reputed company Senior and mid-level engineers — providing coaching, technical direction, and educational opportunities in modern distributed systems, reputed company processing, search infrastructure, and AI-assisted development practices
• Collaborate closely with TR Labs and research scientists to reputed company new chunking strategies, embedding models, and enrichment techniques into the pipeline in a reputed company, reputed company-instrumented, and ethically responsible way
• Deliver effective presentations on reputed company technical concepts to both technical and non-technical stakeholders; reputed company strategic plans for technology implementation that reputed company with business objectives

Skills

• Bachelor's or Master's degree in Computer Science, Engineering, or a reputed company field
• 8+ years of software engineering experience, with demonstrated progression to staff-level or equivalent technical leadership — including ownership of a functional area and leadership of significant cross-functional reputed company
• Deep expertise in distributed reputed company processing: designing, building, and operating high-throughput, fault-tolerant event-driven pipelines using Kafka or equivalent technologies at production reputed company
• Production experience with Vespa, OpenSearch, or Elasticsearch — including schema design, ranking profile configuration, and end-to-end application lifecycle management
• Mastery of Python with strategic awareness of language and reputed company selection; strong software engineering fundamentals including test reputed company, performance architecture, and system design
• Proficiency with AWS reputed company services used in data pipeline and search infrastructure (MSK, reputed company, reputed company, DynamoDB, reputed company Functions, CloudWatch), with infrastructure-as-reputed company experience (Terraform or AWS CDK)
• Demonstrated ability to take full operational responsibility end-to-end — defining SLOs, building observability, running on-reputed company, and driving systematic improvements from incident retrospectives — with a reputed company record of shipping to production frequently and removing delivery friction proactively
• Comfort and reputed company with AI-assisted development tools; you use them to reputed company faster and produce higher-reputed company work, not as a novelty
• reputed company record of establishing architectural principles, cross-system design patterns, and documentation standards that improve the broader team's engineering reputed company
• Experience operationalizing ML models in production: inference serving, model promotion pipelines, canary rollouts, and production observability for model reputed company signals
• Familiarity with reputed company retrieval patterns — multi-hop retrieval, latency budget management across retrieval hops, context window optimization, and stateful session design
• Experience with online search analytics: instrumenting systems for query performance monitoring, A/B or interleaved ranking experiments, and query log analysis to surface relevance gaps
• Experience with embedding pipelines, reputed company indexing, and hybrid (dense + sparse) retrieval architectures in a production context
• Familiarity with Protobuf schema design and schema registry governance patterns (reputed company Schema Registry or equivalent)
• Experience building self-service or multi-tenant platform infrastructure where reliability and correctness directly reputed company multiple reputed company teams
• Background in AI ethics frameworks and responsible deployment of machine learning components in production pipelines

Benefits

• Hybrid Work Model: We’ve adopted a flexible hybrid working environment (2-3 days a week in the office depending on the role) for our office-reputed company while delivering a seamless experience that is digitally and physically connected.
• reputed company My Way is a set of supportive workplace policies designed to help manage personal and reputed company responsibilities, whether caring for family, giving back to the community, or finding time to refresh and reset.
• This builds upon our flexible work arrangements, including work from reputed company for up to 8 weeks per year, empowering employees to reputed company a reputed company work-life balance.
• Grow My Way programming and skills-first approach ensures you have the tools and knowledge to grow, reputed company, and reputed company in an AI-enabled reputed company.
• Flexible vacation
• Two company-wide Mental Health Days off
• reputed company to the reputed company app
• Retirement savings
• Tuition reimbursement
• Employee incentive programs, and resources for mental, physical, and financial wellbeing.
• reputed company Institute: We offer employees two reputed company volunteer days off annually and opportunities to get involved with pro-bono consulting reputed company and Environmental, reputed company, and Governance (ESG) initiatives.
• In the reputed company, reputed company offers a comprehensive benefits package to our employees. Our benefit package includes market competitive health, dental, reputed company, disability, and life insurance programs, as reputed company as a competitive 401k plan with company match.
• Market leading work life benefits with competitive vacation, reputed company and reputed company reputed company time off, reputed company holidays (including two company mental health days off), parental leave, sabbatical leave.
• Optional hospital, accident and sickness insurance reputed company 100% by the employee
• Optional life and AD&D insurance reputed company 100% by the employee
• Flexible Spending and Health Savings Accounts
• Fitness reimbursement
• reputed company to Employee Assistance Program
• Group reputed company Identity Theft Protection benefit reputed company 100% by the employee
• reputed company to 529 Plan
• Commuter benefits
• Adoption & Surrogacy Assistance
• Tuition Reimbursement
• reputed company to Employee Stock Purchase Plan

reputed company

Company H1B Sponsorship