Job Closed
This listing is no longer active.
AI Query Evaluation Specialist – Copilot Competitive Intelligence
Location
Washington
Posted
83 days ago
Salary
$80K - $95K / year
Seniority
Senior
Job Description
AI Query Evaluation Specialist – Copilot Competitive Intelligence
BPCS, Comprehensive marketing solutions, ltd.
• Review and analyze real user query logs to identify queries with clear intent and strong representativeness of English-speaking markets • Curate and maintain high-quality datasets used to evaluate AI systems such as Microsoft Copilot, ChatGPT, and Gemini • Annotate queries across multiple evaluation dimensions, including: • Need for web search or external information retrieval • Presence or likelihood of personally identifiable information (PII) • Requirement for domain-specific or professional expertise • Additional attributes relevant to AI response quality • Ensure annotations are consistent, structured, and aligned with evolving evaluation guidelines • Use tools such as Excel to organize, review, and summarize evaluation outputs • Identify trends in query patterns and provide feedback to improve dataset coverage and quality • Maintain a high level of attention to detail, documentation quality, and evaluation integrity
Job Requirements
- Strong English reading comprehension with the ability to interpret subtle differences in user intent
- Demonstrated analytical thinking and logical reasoning skills
- Experience working with structured data or annotation workflows
- Familiarity with tools such as Microsoft Excel or similar data analysis tools
- Strong user empathy and understanding of how diverse users formulate queries
- Curiosity and familiarity with modern AI tools (e.g., Copilot, ChatGPT, Gemini)
- High attention to detail with a track record of delivering consistent, high-quality work
- Reliable, proactive, and adaptable in fast-changing environments
- Prior experience in data annotation, content evaluation, or dataset curation for AI or search products (preferred)
- Experience with AI evaluation, search relevance, or linguistic analysis (preferred)
- Basic statistical or data analysis knowledge (preferred)
- Demonstrated ability to quickly learn and interpret unfamiliar domains (preferred)
- Fluency in English plus at least one additional language: Japanese, Korean, French, Chinese, German, or Italian (preferred)
Benefits
- Medical, dental, and vision coverage
- Flexible Spending Account
- 401k program
- Competitive PTO offerings
- Parental Leave
- Opportunities for professional growth and development
Related Guides
Related Categories
Related Job Pages
More Artificial Intelligence Jobs
Senior Creature AI Designer – Contract
ArenaNetWe make the games we want to play a reality, and infuse them with innovation, hand-crafted detail, and creative passion.
• Collaborate with Content Design, Technical Design, Creature Design, Art, and Engineering to deliver quality AI behaviors • Work with Systems and Product Leads to support development timelines and goals for content completion • Craft elegant solutions that create opportunities for players to engage with AI in understandable and intentional ways • Maintain vision and direction for the AI behaviors needed across the game • Use Unreal Engine 5 tools and scripting systems to prototype and stress-test AI behaviors • Work with the AI Engineering team to prioritize work • Iterate on features based on player feedback and design reviews
• Help to define and support suitable quality frameworks, metrics, and strategies of Data Services projects. • Perform detailed task analysis, defining key quality drivers and necessary skills, develop efficient and onboarding methodologies for our Data Services Vendors. • Provide feedback to team/stakeholders. • Support client quality escalations and collaborate with the team for root cause analysis. • Support quality improvement plans for underperforming locales and vendors. • Support data stakeholders and provide timely answers to their queries. • Evaluate vendors and offer feedback with training on error trends. • Support and help drive a robust auditor program with training. • Collaborate with Solutions Architects to develop skill evaluation solutions and visualize quality metrics. • Stay updated on AI industry trends and contribute to new opportunities.
Oil & Gas Consultant – AI Discovery Support
WorkanaThe largest platform for hiring top remote talent from Latin America.
• Join occasional client-facing meetings with Xenon7’s CEO and potential oil and gas clients. • Provide subject-matter expertise on oil and gas operations, digital transformation, and AI use cases. • Help identify where AI agents, automation, or analytics could add value across upstream, downstream, HSE, maintenance, production, or capital project workflows. • Support discovery conversations by asking relevant industry-specific questions and helping qualify client needs. • Advise Xenon7 internally on industry context, client pain points, and solution positioning.
Director - Revenue Cycle Automation & AI
Mayo ClinicHeadquartered in Rochester, Minnesota, Mayo Clinic is a nonprofit medical institution ranked first in more specialties than all other hospitals in America. The
Role Description The Director, Revenue Cycle Automation & AI leads Mayo Clinic's automation and AI portfolio across Revenue Cycle operations. This leader owns strategy, delivery, and value realization for a mature, multi-year program that has delivered significant FTE savings and process improvement. The role sets direction for agentic automation, RPA, and AI-enabled solutions in Revenue Cycle; partners deeply with Revenue Cycle operational leadership; and ensures the pillar continues to scale without disruption as the broader program expands. The Director reports to the Revenue Cycle Chair and may have Revenue Cycle Managers and other Revenue Cycle staff as direct reports in a multi-site, multi-specialty academic medical center. Assures the integrity and stability of revenue and billing data and provides direction for complex business decision making for operations. Provides leadership in a team environment, teams functional and technical activities and changes. - Participates in establishing the strategic direction of work teams or service lines incorporating an awareness of the internal and external environment. - Provides direction and interprets revenue cycle key performance indicators with significant institutional impact. - Recognized and relied upon to synthesize various subject matter expert inputs to formulate solutions and implement change for multi-disciplinary or complex technical issues. - Coordinates the development of programs and processes for Revenue Cycle operational areas. - Interprets and implements billing rules, regulatory compliance, policies and regulations. - Stays abreast of emerging issues and risks and plans accordingly. - Identifies strategies that lead to improved financial performance and follows through on their implementation while preserving Mayo standards and minimizing financial and legal risks. - Directs managers, reviews and interprets results of variable-sensitive business models and/or compliance or business operations. - Participates in establishing an environment that leads to efficient Revenue Cycle operations and adheres to complicated and ambiguous billing rules and regulations (all payers). - Responsible for recruitment, development and performance management within areas of responsibility, translating the departmental vision into meaningful and effective results. - Spearheads change and leads others in implementation. - Promotes effective change management practices. - Initiates, leads and facilitates institutional workgroups or complex projects. - Manages and prioritizes limited resources across multi-disciplinary, multi-site teams to maximize efficiency. - Handles ambiguous situations in a productive and professional manner and leads others through such situations successfully. - Travel required as assignments warrant. Qualifications - Own the end-to-end Revenue Cycle automation and AI portfolio strategy, roadmap, and delivery outcomes, in partnership with Revenue Cycle operational leaders. - Lead a multi-disciplinary delivery team of developers, architects, and analysts to design and deploy UiPath and AI-enabled solutions across Revenue Cycle workflows. - Sponsor and govern agentic automation initiatives (e.g., UiPath Maestro, LLM-based agents with human-in-the-loop) that extend the program beyond traditional RPA. - Drive business case development, ROI measurement, and value realization reporting for Revenue Cycle initiatives, leveraging standardized rate cards and value levers. - Manage vendor and platform relationships (UiPath and AI tooling) for Revenue Cycle use cases, coordinating with enterprise architecture and procurement. - Serve as executive-level interface to Revenue Cycle leadership, representing the automation program in governance forums, steering committees, and strategic reviews. - Develop, coach, and retain talent across the Revenue Cycle pillar; model leadership behaviors that reinforce Mayo Clinic's primary value. - Partner across pillars to share reusable components, accelerators, and architecture patterns that benefit the broader enterprise program. Benefits - Medical: Multiple plan options. - Dental: Delta Dental or reimbursement account for flexible coverage. - Vision: Affordable plan with national network. - Pre-Tax Savings: HSA and FSAs for eligible expenses. - Retirement: Competitive retirement package to secure your future. Equal Opportunity All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender identity, sexual orientation, national origin, protected veteran status or disability status. Learn more about the "EOE is the Law". Mayo Clinic participates in E-Verify and may provide the Social Security Administration and, if necessary, the Department of Homeland Security with information from each new employee's Form I-9 to confirm work authorization.




