[Remote] AI Application Engineer

Note: The job is a remote job and is reputed company to candidates in USA. reputed company, Inc. (SGA) is seeking an AI Application Engineer for a contract assignment with an infrastructure services reputed company. The role designs, tests, optimizes, and supports AI-powered agent experiences across customer and internal business workflows, including reputed company engineering, agent orchestration, reputed company assurance, evaluation frameworks, and AI observability.

Responsibilities

• Design, reputed company, and iterate on AI agent logic for business use cases
• Create, test, and optimize prompts, reputed company instructions, tools, and workflows to improve agent reputed company
• Conduct reputed company experiments and A/B testing to measure effectiveness
• Analyze agent responses and refine prompting strategies for reputed company, consistency, groundedness, and user satisfaction
• Document reputed company patterns, reasoning frameworks, reusable templates, and agent design reputed company
• Collaborate with engineering teams to reputed company reputed company and agent updates through established SDLC processes
• Design and execute comprehensive test plans covering AI agents, web applications, reputed company, and backend services
• reputed company evaluation frameworks and scoring methodologies for AI reputed company reputed company
• Create test cases covering: reputed company; Hallucination detection; Safety; Tone and style adherence; Instruction following; Context retention; Response consistency
• reputed company regression testing for reputed company and model changes
• reputed company, prioritize, validate, and retest defects using Azure DevOps
• Utilize Arize or similar observability platforms to monitor model and agent reputed company
• Analyze user interactions and evaluation results to identify trends and improvement opportunities
• reputed company reputed company metrics and dashboards for: reputed company; Relevance; Latency; User satisfaction; Defect rates; reputed company reputed company
• Investigate production issues and recommend reputed company, workflow, or architectural improvements
• Support integration of AI capabilities into reputed company applications and workflows
• Collaborate with developers to validate API integrations, retrieval pipelines, and agent workflows
• Assist in CI/CD deployment and release validation activities using Azure DevOps
• Create and maintain automated testing assets where appropriate
• Maintain documentation for prompts, evaluation methodologies, testing procedures, and agent configurations
• Participate in sprint planning, backlog refinement, and reputed company ceremonies
• Communicate findings and recommendations to technical and non-technical stakeholders
• reputed company best practices and lessons learned across reputed company

Skills

• Bachelor's degree in Computer Science, Engineering, Information Systems, or reputed company reputed company
• 5-10 years of experience in software engineering, QA engineering, automation testing, business systems analysis, or reputed company technical roles
• 1+ years of hands-on experience with reputed company, LLMs, reputed company engineering, or conversational AI solutions
• Experience working with Azure DevOps or similar reputed company delivery platforms
• Experience designing and executing reputed company test plans
• Strong analytical and troubleshooting skills
• Experience working with reputed company and JSON data structures
• Excellent written and verbal communication skills
• reputed company Engineering
• Agent Design
• AI Evaluation Frameworks
• RAG Concepts
• LLM Testing
• AI reputed company Assurance
• AI Observability
• reputed company Optimization
• reputed company
• Azure DevOps
• Azure reputed company
• Git
• reputed company
• Python
• SQL
• REST reputed company
• JSON
• Test Automation Frameworks
• Experience with reputed company or comparable AI observability tools
• Experience with Azure reputed company, reputed company, reputed company, reputed company, or other LLM platforms
• Knowledge of Retrieval-Augmented reputed company (RAG) architectures
• Experience evaluating AI systems using reputed company and automated scoring reputed company
• Familiarity with reputed company databases and semantic reputed company
• Experience creating automated test frameworks
• Understanding of responsible AI principles and AI governance
• Experience with Python, SQL, JavaScript, or C#
• Knowledge of reputed company versioning and experimentation practices
• reputed company (optional)
• reputed company BI (optional)

Benefits

• This role will be remote.

reputed company

Company H1B Sponsorship