• Define the testing standards and patterns for AI at Roofr — establishing how product teams validate AI behaviour when building on top of the application foundation • Build and own Roofr's LLM eval framework — selecting and extending the right tooling (e.g. Promptfoo, DeepEval, Braintrust) and designing the methodology that measures whether our AI integrations and agent outputs are performing correctly, consistently, and safely • Integrate quality gates into CI/CD pipelines so that regressions in AI behaviour are caught before they reach production • Design and implement human-in-the-loop review processes for AI outputs where automated evaluation isn't sufficient • Embedded on the AI Platform team — ensuring quality is designed into the integration architecture from day one, not bolted on after the fact • Work horizontally across the testing organization — coaching QA engineers and developers on AI eval patterns, embedding best practices into team workflows, and actively raising the quality bar across engineering • Stay close to the evolving AI quality landscape — new eval techniques, benchmarking approaches, and tooling like Ragas, Arize Phoenix, or LangSmith — and bring the best of it to Roofr

• Lead quality activities across all phases of development • Establish and implement quality management across greenfield and brownfield projects • Support projects from proposal through closeout using structured problem-solving • Lead quality oversight of project partners • Establish, implement, and continuously improve the Quality Management System (QMS) • Lead performance-based audits and corrective action programs • Monitor regulatory developments and industry standards • Ensure quality requirements are defined and flowed down across project activities • Oversee QA requirements for software and digital tools affecting licensing or quality • Establish and implement the vendor qualification program • Define, track, and report quality performance metrics • Build and lead the quality organization • Support implementation of digital tools and process improvements

Role Description The Senior QA Analyst is responsible for working with cross-functional subject matter experts to document, deploy and monitor business and application test cases. This position will require the crafting and deployment of testing strategies and the presenting of strategic findings to leadership to move the organization to action. Those successful in this role will be strategic thinkers with the ability to negotiate skillfully with stakeholders to find common ground and acceptable alternatives that satisfy the needs of multiple end users. Additionally, you will help drive overall strategy and prioritization of work efforts. What you'll Do - Collaborate with leaders and stakeholders across the company to understand the business, product and system requirements. - Thoroughly test software to ensure proper operation and freedom from defects utilizing knowledge of application architecture. - Maintain documentation of test results, assist in debugging and communicate to business stakeholders. - Utilize necessary analytical skills to develop and maintain knowledge of in-house/vendor systems, formulas, rate specifications, manual rules, and state filing information to identify unusual, inconsistent or error conditions. - Maintain a test repository and/or performance test results; identify and resolve problems in areas of test development/management; devise improvements to current procedures and develop models of possible future configurations. - Provide detailed, insightful, constructive, and actionable feedback based on testing outcomes; produce reporting and documentation for all testing efforts, results, activities, data, logging and tracking. - Create visibility and deepen collaboration during development to ensure more reliable releases. - Enhance tools, processes, and frameworks to bring insurance products to market. Qualifications - Bachelor’s degree or equivalent required. - 4+ years experience developing, executing and monitoring application and business test cases. - 4+ years experience in P&C insurance industry or insurtech preferred. - 2+ years experience creating test automation suites on web services preferred. - Proven expertise in reviewing/designing test cases, test strategies, and test plans for multiple application releases based upon decomposed technical and user requirements. - Strong experience and comfort within Salesforce, a Policy Administration System (PAS), GSuite and JIRA (or similar task management software). - Experience using Java and Python, and tools such as Postman, Percy, and Playwrite. - Experience with accessibility testing, cross browser/platform testing, performance and security testing. - Strong analytical techniques and technical communication skills with ability to communicate ideas clearly, concisely and effectively. - Proven experience working in cross-functional teams and communicating with colleagues in both business and technical roles. - Positive “can do” attitude and a desire to contribute to a growing, energetic working team; experience working within a fast paced agile software development environment. - High level of detail orientation, with the ability to prioritize competing demands and manage time effectively. - Ability to adjust and change direction with ease. - Ability to build trust and rapport cross functionally in a remote, dispersed environment. Interview Process At Obie, transparency is core to our interview process. We focus on open communication and a positive experience; no trick questions—just honest conversations about your skills and aspirations. To help you prepare, please familiarize yourself with these resources before moving forward: - Our company’s blog to learn more about the market we serve and the information our customers value. - Check out our LinkedIn to see how people interact with our brand and product. - Press and podcast features like Inc. 5000, TechCrunch, Innovate State, Uncontested Investing Podcast, Insurtech Amplified Podcast. You'll first talk to Skylar on a 30-minute call to determine whether we’re a good fit for one another. You’ll learn more about Obie and have the chance to ask about our company, team, culture, and product. Next, you'll have a 45-minute video interview with the hiring manager, Denise Pickering, to assess your technical skills for the role. This role requires strong planning and analytical skills as well as collaboration and communication skills, so we'll also present you with a short exercise to complete to provide you a chance to flex those muscles. You will be able to complete this exercise on your own time, so please be thorough and use the resources available to you. We'll be looking to see your problem solving abilities, writing and communication skills, empathy, general demeanor, and ability to leverage key QA strategies. More on this when you get there! Since you'll be joining a close-knit team, we'll ask you to meet other people in the company so we get to know each other a bit better. Moving Forward with Obie Obie is leading the future of real estate investment protection, securing over $25 billion in coverage for thousands of policyholders nationwide. Backed by $39 million in funding and recognized as a top workplace by Built In and Inc., we’re committed to driving progress for our customers and team. Through advanced digital tools, a trusted agent network, and strategic partnerships, Obie provides real estate investors the insurance coverage they need to move forward confidently. Benefits - Comprehensive health, dental, vision and life insurance. - Flexible paid time off. - Retirement savings plan with company match. Obie values transparency and data driven decision making, enabling us to have a consistent and equitable approach to compensation. We are informed by a benchmarking tool which provides real time compensation data allowing us to regularly check our salary ranges against other venture backed companies and startups within our industry. We've provided the starting base salary range for this role below. Your salary is dependent upon many factors, including years of relevant experience, expertise, etc. $90,000 - $130,000 USD per year. During your first interview, you can expect to have an open conversation with your recruiter regarding your compensation expectations.

• Work closely with various desktop (Linux, Windows, macOS), iOS, and web technologies. • Collaborate with developers on exploratory testing of the framework’s platform-specific features. • Conduct integration testing of important user scenarios with particular attention to the corner cases. • Search for and pinpoint bugs, submitting the results to our issue tracker and performing regression tests for code fixes. • Take part in product planning, including creating plans for testing the proposed changes. • Work on the automation of integration tests from the point of view of both Compose API users and the end users of applications built with Compose Multiplatform.