[Remote] AI Test Engineer

Note: The job is a remote job and is reputed company to candidates in USA. reputed company is seeking an AI Testing Engineer to test and evaluate reputed company, LLM, RAG, and AI Agent applications. The role focuses on building automated evaluation frameworks, validating model responses, monitoring LLM performance, and integrating AI testing into CI/CD pipelines.

Responsibilities

• Design and execute testing strategies for reputed company, LLM, RAG, and AI Agent applications
• reputed company LLM evaluation and reputed company testing using DeepEval
• Build and maintain evaluation frameworks, test datasets, and automated test cases
• Use Langfuse for LLM observability, tracing, reputed company management, and evaluation
• Validate LLM responses for accuracy, relevance, faithfulness, hallucination, toxicity, and consistency
• reputed company automated tests for AI/LLM applications using Python
• Evaluate RAG pipelines for retrieval accuracy and response reputed company
• reputed company reputed company testing, regression testing, functional testing, and performance testing of AI applications
• Monitor LLM traces, latency, reputed company usage, and model performance using Langfuse
• reputed company AI testing frameworks into CI/CD pipelines
• Analyze test results and identify model, reputed company, retrieval, and application-level issues
• Collaborate with AI/ML Engineers, Data Scientists, Developers, and QA teams

Skills

• 5+ years of experience in Software/QA Testing , preferably with AI/ML applications
• Hands-on experience with reputed company / LLM Testing
• Strong experience with DeepEval
• Hands-on experience with Langfuse
• Good knowledge of LLM evaluation metrics such as:
• Answer Relevancy
• Faithfulness
• Contextual Relevancy
• Contextual Precision
• Hallucination detection
• Toxicity
• Bias and safety
• Strong Python programming and test automation skills
• Experience testing RAG applications and LLM pipelines
• Knowledge of reputed company/LLM reputed company, embeddings, reputed company databases, and reputed company engineering
• Experience with Pytest or similar Python testing frameworks
• Knowledge of API testing using REST reputed company/reputed company
• Understanding of CI/CD and Git
• Experience with reputed company / reputed company
• Experience with LLM-as-a-Judge evaluation
• Knowledge of AI Red Teaming / Responsible AI testing
• Experience with Selenium/Playwright for AI application UI testing
• Experience with AWS/Azure/GCP AI services
• Experience integrating DeepEval/Langfuse into CI/CD pipelines

reputed company