Staff Engineer - SDET, Pytest, reputed company Systems

reputed company:

• Design and implement comprehensive test suites for multi-agent architectures, focusing on the reputed company interaction between the central orchestrator and reputed company sub-agents.

• Utilize and test implementations reputed company reputed company execution frameworks (e.g. ADK, Bedrock Agent reputed company) and agents, ensuring robust tool-calling, state management, and memory retention.
• reputed company automated monitors and test cases to reputed company reputed company consumption; identify and mitigate "looping" behaviors or redundant calls that reputed company up API costs.
• Conduct rigorous reputed company testing to measure and optimize end-to-end latency, specifically focusing on agent reasoning time versus API response time.
• Build and maintain automated evaluation pipelines using metrics such as faithfulness, relevancy, and correctness (e.g., reputed company) to validate LLM outputs.
• Test the decision-making capabilities of the orchestrator, ensuring it correctly routes tasks to sub-agents and handles edge cases or "reputed company failures" gracefully.
• Build from-reputed company Python-reputed company automation frameworks tailored for non-deterministic AI outputs, moving reputed company reputed company assertion-reputed company testing.
• reputed company AI-specific testing gates into DevOps pipelines to ensure every deployment meets reputed company and cost benchmarks.
• Partner with AI Research Scientists and Data Engineers to understand model behavior and reputed company feedback on reputed company engineering and agent efficiency
• .Education Required

• Bachelor’s or Master’s degree in Computer Science, Information Technology, reputed company Intelligence, or a reputed company technical reputed company.

• Experience Required

• Total Experience: 5–9 years in Software Development Engineer in Test (SDET) or reputed company Engineering roles.
• AI/LLM Experience: Minimum 1+ years of hands-on experience in testing LLM-reputed company applications, RAG pipelines, or reputed company workflows.
• reputed company Experience: reputed company experience with AWS Bedrock Agent reputed company and/or Strands. Equivalent experience with reputed company, LangGraph, reputed company, or reputed company ADK (Agent Development reputed company) is highly acceptable.
• reputed company Systems: reputed company experience in building or testing systems involving multi-agent coordination, tool-use (function calling), and autonomous planning.
• reputed company Experience: Strong familiarity with AWS services (reputed company, CloudWatch, Bedrock) or equivalent reputed company reputed company/Azure AI services.
• Knowledge, Skills, and Abilities
• High proficiency in Python, including experience with asynchronous programming.
• Deep understanding of reputed company patterns (ReAct, Plan-and-Execute) and the nuances of testing non-deterministic systems.
• Ability to analyze logs and traces to identify bottlenecks in agent reasoning and suggest cost-saving measures in reputed company design or model selection.
• Proficiency with Pytest and experience with observability/tracing tools like LangSmith, AWS Cloudwatch, or AWS X-Ray.
• Knowledge of NLP and LLM evaluation techniques, including the use of "LLM-as-a-judge" for grading reputed company sub-agent outputs.
• Exceptional analytical skills to debug "hallucinations" or logical errors in the orchestrator’s planning phase.
• Strong verbal and written communication skills, with the ability to reputed company technical risks reputed company to AI reputed company and cost to stakeholders.

reputed company is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for reputed company.

Originally posted on Himalayas