Abascus Insights is a technology company working to improve people's lives by "harnessing the healthcare data explosion" through intelligent integration software. Founded in 2017 b
Senior AI Systems Quality Engineer
Location
Massachusetts
Posted
9 days ago
Salary
0
Seniority
Senior
Job Description
Senior AI Systems Quality Engineer
Abacus Insights
• Build and ship production-grade, automated validation frameworks, test harnesses, and evaluation pipelines across the AI lifecycle (design → deploy). • Design and evolve an AI testing platform integrated with Databricks and MLflow, enabling repeatable testing, traceability, and auditability. • Create large-scale, scenario-based test suites (hundreds to thousands of cases) to validate agentic workflows end-to-end, including edge cases, long-tail scenarios, and failure modes. • Validate orchestration behavior (tool use, memory, decision logic) and stress-test non-deterministic system behavior before production. • Embed quality by design: define system contracts, guardrails, and safe-degradation patterns at key boundaries. • Define measurable quality signals for LLM systems (grounding, hallucinations, relevance, latency, cost) and integrate them into CI/CD pipelines as automated quality gates. • Ensure AI validation runs automatically on model, prompt, and code changes—enabling continuous quality enforcement. • Build reusable libraries and components so teams can adopt consistent AI quality practices quickly. • Own aspects of AI release readiness, including defining go/no-go criteria based on measurable quality thresholds. • Partner with AI, platform, security, and delivery teams to translate mission needs into clear quality criteria, tradeoffs, and confidence levels.
Job Requirements
- 7+ years of software engineering experience, primarily in backend or platform systems.
- Proven experience designing and implementing AI testing automation in production environments, not just executing tests.
- Demonstrated ability to build custom validation, evaluation, or testing frameworks for complex, distributed systems.
- Strong proficiency in Python and/or TypeScript within modern AI engineering stacks.
- Hands-on experience with AI-powered systems, including LLM-based or agentic workflows and non-deterministic behavior.
- Experience designing or contributing to AI testing at scale, including regression frameworks, long-tail evaluation, and large test coverage.
- Deep understanding of CI/CD integration, including embedding automated tests and quality gates into deployment pipelines.
- Solid understanding of AWS cloud-native architectures.
- Track record of engineering for quality, reliability, governance, and safety as core system design principles.
- Working knowledge of security, privacy, and operational risk in regulated or mission-critical environments, including failure modes and recovery.
- Experience with AI testing methodologies, including evaluation of non-deterministic outputs, drift detection, bias/fairness testing, and robust regression strategies.
- Proven ability to establish measurable trust thresholds for AI systems, including defining and operationalizing success metrics such as query accuracy, hallucination limits, explainability, and PHI-safe behavior as enforceable release criteria.
- Experience working with domain experts to define correctness and real-world validation scenarios, enabling large-scale, business-relevant test coverage that reflects true production use cases rather than engineering-only perspectives.
Benefits
- Unlimited paid time off – recharge when you need it
- Work from anywhere – flexibility to fit your life
- Comprehensive health coverage – multiple plan options to choose from
- Equity for every employee – share in our success
- Growth-focused environment – your development matters here
- Home office setup allowance – one-time support to get you started
- Monthly cell phone allowance – stay connected with ease
Related Guides
Related Categories
Related Job Pages
More QA Engineer Jobs
• Proactively acquire new customers from mid-market and enterprise companies • Develop a go-to-market strategy for QA services • Identify market opportunities, target industries and decision-makers • Build a sustainable sales funnel through CRM, networking, events and social selling • Advise customers on suitable solutions, prepare proposals and conduct negotiations • Represent the company at trade shows and events • Monitor trends, competitors and markets to identify new business opportunities • Work closely with Business Development, Marketing/Communications and stakeholders
Senior Manual QA Engineer
TeramindTeramind was founded in 2014 to deliver workforce monitoring software and insider threat prevention with a user-centric approach to security. In addition to mon
• Be the quality champion: plan, design, and run test cases for web and desktop applications. • Perform hands-on testing — functional, regression, integration — to make sure nothing slips through. • Drive and improve our QA strategy as we evolve our CI/CD pipeline. • Collaborate directly with developers to turn business requirements into testing requirements. • Spot issues early, drive resolutions, and elevate our testing standards. • Share your knowledge with the team and mentor others in QA best practices.
Senior QA Engineer
TeramindTeramind was founded in 2014 to deliver workforce monitoring software and insider threat prevention with a user-centric approach to security. In addition to mon
• Be the quality champion: plan, design, and run test cases for web and desktop applications. • Perform hands-on testing — functional, regression, integration — to make sure nothing slips through. • Drive and improve our QA strategy as we evolve our CI/CD pipeline. • Collaborate directly with developers to turn business requirements into testing requirements. • Spot issues early, drive resolutions, and elevate our testing standards. • Share your knowledge with the team and mentor others in QA best practices.
Senior QA Automation, Advanced English
CapcoCapco, a Wipro company, is a management & technology consultancy dedicated to the financial services & energy industries
• Design, develop, and maintain comprehensive test automation frameworks. • Lead testing efforts across multiple projects. • Collaborate closely with developers and other stakeholders. • Identify, analyze, and prioritize software defects. • Mentor and guide junior and mid-level QA professionals. • Provide leadership in Agile ceremonies. • Analyze test results and produce reports. • Drive continuous improvement initiatives.
