Role Description The Test Engineering team is the quality assurance backbone of the CMS Medicare Systems Solutions (CMSS) portfolio, responsible for protecting the stability and integrity of FISS, MCS, DME, CWF, and Crosscutting programs through every software release. The Supervisor, Test Engineering owns the end-to-end test strategy for this environment, ensuring that every change, enhancement, and modernization effort is validated against the full depth of legacy system behavior before it reaches production. In this role, the Supervisor applies an expert understanding of how the test engineering function integrates with development, architecture, operations, and business teams to protect system stability and support CMS's modernization objectives. The Supervisor identifies and solves technical and operational testing problems, particularly those arising at the intersection of legacy mainframe systems and modern integration layers. The Supervisor develops, coaches, and directs test engineers and functional testers to adopt consistent regression and automation standards, adapting test plans and priorities to address resource and operational challenges. Key Responsibilities - Mainframe & Regression Testing Leadership - Develop and implement a comprehensive testing strategy grounded in Agile/DevOps practices and automation-first principles. - Own and continuously mature the regression testing strategy for COBOL-based mainframe systems. - Maintain and expand the regression test suite to reflect changes introduced by enhancements and modernization activities. - Establish and enforce regression baseline standards. - Lead mainframe-specific test planning for batch processing, JCL job streams, VSAM file handling, and claims adjudication logic. - Coordinate regression test execution across all 12 Medicare Administrative Contractors (MACs). - Ensure regression test assets are version-controlled and traceable to requirements and business rules. - Test Planning & Execution - Create, maintain, and manage test execution plans spanning multiple testing phases, environments, and release cycles. - Coordinate and oversee automated and manual testing activities across integrated teams and subcontractors. - Ensure test environments, data sets, and automation assets support scalable, repeatable testing. - Participate in requirements and user story reviews and design walk-throughs to ensure testability. - Systems Stability & Release Readiness - Serve as the quality gate for every production release. - Develop and maintain release-specific test plans. - Identify and escalate systemic risks to system stability. - Lead defect triage and resolution analysis. - Maintain production readiness checklists and sign-off documentation for each release. - Test Automation & Modernization Support - Design and maintain automation frameworks that extend coverage into mainframe testing. - Support modernization initiatives by developing integration and regression test strategies. - Promote automation-first principles while recognizing where manual testing is required. - Identify, evaluate, and implement AI-enabled testing tools. - Ensure test automation assets are integrated into CI/CD pipelines. - Quality Management & Reporting - Develop, maintain, and enforce Quality Management Plan (QMP) standards. - Track and analyze test metrics, defect trends, and execution results. - Identify, communicate, and mitigate risks. - Collaborate with the Delivery Support Services team. - Team & Stakeholder Leadership - Supervises and assigns work for a team of test automation engineers and business analyst professionals. - Conduct performance monitoring and formal evaluations. - Build partnerships with development, architecture, operations, and business teams. - Collaborate effectively across distributed teams. - Foster a culture of quality, regression discipline, and continuous improvement. Qualifications - 6+ years of combined software testing, test automation, technical analysis, or business analysis experience. - 5+ years of direct management experience leading teams of 10+ personnel. - Demonstrated expertise in mainframe testing environments, including COBOL-based systems. - Proven ability to design, manage, and continuously mature large-scale regression testing programs. - Hands-on experience with test automation tools such as Selenium, Cucumber, Postman, JUnit, Jira, TestRail, or similar. - Experience developing automation frameworks across API, UI, E2E, and performance layers. - Experience producing test metrics, dashboards, and release readiness assessments. - Strong oral and written communication and presentation skills. - Ability to work within Agile/DevOps environments while managing legacy system release cycles. - Experience in multivendor, multisystem program environments. - Active Public Trust Required. Preferred Qualifications - Bachelor's degree. - Experience with healthcare claims processing or federal healthcare programs. - Prior experience supporting major modernization or transformation initiatives. - Experience with CI/CD pipelines and DevSecOps environments. - Relevant test automation or quality certifications. - Experience applying AI/ML-based testing tools or predictive analytics. Company Description Peraton is a next-generation national security company that drives missions of consequence spanning the globe and extending to the farthest reaches of the galaxy. As the world’s leading mission capability integrator and transformative enterprise IT provider, we deliver trusted, highly differentiated solutions and technologies to protect our nation and allies. Peraton operates at the critical nexus between traditional and nontraditional threats across all domains: land, sea, space, air, and cyberspace. The company serves as a valued partner to essential government agencies and supports every branch of the U.S. armed forces. Target Salary Range $112,000 - $179,000. This represents the typical salary range for this position. EEO EEO: Equal opportunity employer, including disability and protected veterans, or other characteristics protected by law.