Senior SQA Engineer – LLM

Location

Pakistan

Posted

25 days ago

Salary

0

Seniority

Senior

Bachelor Degree6 yrs expEnglishJUnitMongoDBNoSQLSpringSpring BootSpringBoot

Job Description

Senior SQA Engineer – LLM

TechSurge Private Limited

• Design and own the end-to-end QA strategy for the Conversational Banking Platform, covering functional, regression, performance, security, and AI-specific evaluation. • Build and maintain golden datasets, eval suites, and LLM-as-judge frameworks to validate conversational quality across intents, languages, and tenants. • Define the tenant onboarding QA gate, the certification checklist every new business unit must pass before going live. • Establish regression strategies for prompt changes, model upgrades, retrieval index updates, and guardrail policy changes. • Use Langfuse traces to drive evaluation: mine production failures, convert them into test cases, and close the loop with engineering. • Test NeMo Guardrails configurations against jailbreaks, prompt injection, off-topic drift, and false-positive over-blocking. • Validate governance and compliance behaviors: data residency, PII handling, regulated-product disclosures, and off-limits topics. • Build automated test harnesses for Spring AI services, including tool-calling validation, RAG groundedness, and integration with Cosmos DB and MongoDB data layers. • Partner with the Platform team on quality metrics, SLOs, and the platform eval scorecard. • Coach feature engineers and tenant teams on writing their own evals, making platform-grade quality self-service over time.

Job Requirements

  • 6+ years in software QA, with at least 1–2 years testing LLM-based, RAG, or conversational AI systems in production.
  • Hands-on experience with LLM observability and evaluation tools such as Langfuse, LangSmith, Arize, or Phoenix.
  • Working knowledge of eval frameworks such as Ragas, DeepEval, Promptfoo, or TruLens — including metrics like faithfulness, groundedness, answer relevance, and context precision.
  • Practical understanding of how to test non-deterministic systems: golden datasets, semantic similarity, LLM-as-judge, and statistical regression detection.
  • Experience testing guardrail or policy frameworks (NeMo Guardrails, Guardrails AI, or similar).
  • Solid foundation in API testing, automation frameworks (e.g., pytest, JUnit, Karate, RestAssured), and CI/CD integration.
  • Familiarity with Spring and Spring Boot applications and JVM-based services.
  • Comfortable writing queries against NoSQL stores (MongoDB, Cosmos DB) for test data setup and trace inspection.
  • Strong written communication : able to produce clear test plans, defect reports, and tenant readiness assessments.
  • Good to Have Experience in banking, financial services, or another regulated industry.
  • Exposure to multi-tenant platforms: understanding how shared infrastructure changes the testing problem.
  • Familiarity with red-teaming, adversarial prompt testing, and prompt injection defense.
  • Working knowledge of vector databases, embedding models, and retrieval evaluation.
  • Experience with multi-language conversational systems.
  • Performance and load testing experience for AI workloads (token throughput, latency percentiles, cost per conversation).
  • Contributions to open-source eval or AI testing tooling.
  • Experience working with compliance, risk, or audit teams on AI assurance.

Related Categories

Related Job Pages

More QA Engineer Jobs

LEARN Behavioral logo

Behavior Analyst Specialist

LEARN Behavioral

A national organization dedicated to nurturing the potential of children and young adults with autism and special needs.

QA Engineer25 days ago
Part TimeTeam 1,001-5,000H1B No Sponsor

Title: Behavior Analyst Specialist (Part-time) Location: Worcester United States Business Unit Behavioral Concepts Category Behavior Analyst Minimum USD $70.00/Per Hr. Maximum USD $84.00/Per Hr. Job Description: Overview We're looking for... Bright, Collaborative, Big-hearted, and... Analytical clinicians to join us. Does this sound like you? If so, we'd love to talk! See why joining our team could be the perfect fit for you: https://youtu.be/ehqCfFFy21M Who We're Looking For BCI is hiring a part-time Board Certified Behavior Analyst Specialist (BCBA) to join them in providing high-quality, evidence-based, contemporary ABA therapy to children with autism. Our clinicians are committed to delivering individualized, person-centered care rooted in the principles of assent-based treatment. We consider parents and caregivers our partners on our mission to nurture the potential of every child in our care-setting them up for success in school and life. And, we have reasonable billable expectations for our team. Allow Us to Introduce Ourselves For over 20 years, BCI has provided evidence-based, contemporary ABA therapy to help children and young adults with autism find success. Our goal is to empower our clients to build the skills needed to live a happy and fulfilling life. As a Behavior Analyst Specialist at BCI, you'll not only have access to experienced local clinical leadership and support, but you'll also be part of LEARN Behavioral-a collective group of ABA providers delivering collaborative care to communities from coast-to-coast. LEARN employs over 500 BCBAs and offers services in 18 states, and our average clinical leader has been with the organization for 10 years. What We Offer LEARN Perks - Free, on-demand CEUs on our proprietary LEARNing Lab - Yearly professional development stipend (can be used for conferences, licensure, recertification, etc.) - Monthly clinical forums (live CEUs) - Person-centered care and assent-based treatment programming - Support from multiple specialty teams including: Feeding Intervention Support Team, Functional - Analysis Support Team, and High-Risk Review Team - Experienced scheduling, training, and insurance teams - DEI, Neurodiversity, and other specialty groups that foster a diverse and inclusive workplace Additional Benefits - Plenty of promotional and leadership opportunities - Comprehensive wellness benefits, including Talkspace, care.com, and LEARN Perks (discounts) - 28 days of total paid time off for new, full-time BCBAs - Full-time and part-time benefits available including: - Medical - Vision - Dental - 401(k) with discretionary match starting at year 1 - Accident benefit, short-term disability, life/AD&D Insurance, and more What You Have - Master's degree or higher - BCBA certification - Minimum of two years of experience working as a BCBA - Certification, registration, and/or license as required by local statutes to deliver behavior treatment - Exceptional professional, interpersonal, and communication skills (written and vocal) - Commitment to our five values: partnership, integrity, curiosity, client-centered, and excellence Bonus if you have an interest in research, specifically functional analysis, verbal behavior, feeding, and other topics! What You'll Be Doing - Provide oversight and supervision for your team and clients - Mentor and support behavior technicians (BTs) (note: all our BTs go through competency-based training) - Write reports and conduct clinical reviews with funding sources - Attend monthly regional meetings for ongoing training and supervision - Conduct regular parent/caregiver trainings for family members - Treat the safety of clients and others involved in each case as a top priority - Consult with clients and provides continuous program direction and maintenance - Analyze data/behavior and makes data-based decisions - Guide ongoing implementation of teaching procedures - Ensure program directives from senior clinical team are implemented accurately and timely - Deliver individualized, person-centered care rooted in the principles of assent-based treatment

Massachusetts
$70 - $84 / hour

Role Description RRI is seeking experienced part-time Quality Control Reviewers to audit commercial insurance loss control surveys. - Reviews reports for quality, including technical/grammatical accuracy and appropriate report formats, assuring client requirement guidelines are met. - Provides technical training to field consultants, as needed. - Communicates report review findings, both positive and negative, with all field consultants for continued improvement of report quality. - Keeps field management informed of field consultant capabilities and issues. Qualifications - Minimum of 8 years’ experience in loss control, preferably having experience report reviews. - Knowledge of current NFPA codes and standards. - Good written and verbal communication skills including exceptional writing ability. - Solid time management skills, organizational skills, and computer skills. - Designations such as CSP, ARM or related loss control management certifications are preferred. Requirements - This is an excellent opportunity for individuals who want to set their own schedules and work independently in a growing segment of a vital industry.

United States
QA Engineer25 days ago
Full TimeRemoteTeam 10,001+Since 1982H1B Sponsor

• Ensure high-quality delivery across complex customer programmes. • Provide end-to-end test leadership. • Act as Programme Test Manager on large or complex initiatives. • Own the programme test strategy and oversee quality gates and assurance activities. • Produce and maintain all key test artefacts, ensuring testing is carried out to robust, agreed standards throughout the lifecycle.

United Kingdom
Blend360 logo

Data QA Analyst

Blend360

Optimizing business performance through people, data, tech & analytics

QA Engineer25 days ago
Full TimeRemoteTeam 501-1,000H1B Sponsor

• Validate data accuracy and ensure production readiness by working closely with the Data Engineering Manager and Analytics Engineering Lead across 18 data domains. • Design and build a reconciliation framework to systematically compare legacy pipeline outputs against new pipeline outputs, identifying discrepancies and gaps. • Execute structured acceptance testing for each pipeline prior to promotion to production environments. • Validate identity resolution accuracy through rigorous analysis of match rates, false positives, and false negatives. • Document end-to-end data lineage across all 18 domains to support auditability, transparency, and regulatory compliance. • Build and maintain automated regression test suites to enable continuous quality assurance as pipelines evolve.

Argentina