Mindrift logo
Mindrift

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid. Project time expectations: Tasks are estimated to require around 10–20 hours per week during active phases, based on project requirements; This is an estimate, not a guaranteed workload, and applies only while the project is active. Note: Rates vary based on expertise, skills assessment, location, project needs, and other factors. Higher rates may be offered to highly specialized experts. Lower rates may apply during onboarding or non-core project phases. Payment details are shared per project.

Senior Software Engineer - Testing AI Coding Agents

Location

Worldwide

Posted

12 hours ago

Salary

$50 / hour

Seniority

Senior

Job Description

Senior Software Engineer - Testing AI Coding Agents

Mindrift

Role Description Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based. What this opportunity involves: - Building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. - Creating challenging tasks and evaluation criteria within realistic simulated environments: - Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history. - Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent. - Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient. - Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust. What this is NOT: - Not data labeling. - Not prompt engineering. - Not writing code from scratch - the agent writes most of the code; you guide and evaluate. Qualifications - 5+ years in software development. - Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis. - Experience writing tests (functional, integration). - English proficiency - B2+. Requirements - Deep understanding of where models fail and what scenarios reveal the difference between a good and a bad solution. - Ability to create tasks that genuinely challenge the best models. - Writing tests that accept all correct solutions and reject incorrect ones. Benefits - Compensation up to $50/hr equivalent, depending on level and pace. - Tasks are estimated at ~20 hours each; you set your own schedule. Effort Estimate Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Related Job Pages

More Software Engineer Jobs

Newell Brands logo

Manager, Global Automation Operations – Engineering

Newell Brands

We are focused on delighting consumers by lighting up everyday moments.

Software Engineer13 hours ago
Full TimeRemoteTeam 10,001+H1B Sponsor

• Lead the implementation and continuous improvement of warehouse operations, processes, and performance across distribution sites • Optimize workflows, improve productivity, and ensure consistent execution of operational standards • Partner with Operations, Engineering, IT, and Analytics teams to analyze processes • Validate workflows and process changes to ensure safety • Analyze operational data to identify bottlenecks and inefficiencies • Design, analyze, and optimize warehouse layouts and workflows using industrial engineering principles • Develop and implement process improvements to reduce waste, cost, and cycle times using Lean or Six Sigma methodologies

United States
Mindrift logo

Senior Software Engineer

Mindrift

Apply → Pass qualification(s) → Join a project → Complete tasks → Get paid. Project time expectations: Tasks are estimated to require around 10–20 hours per week during active phases, based on project requirements; This is an estimate, not a guaranteed workload, and applies only while the project is active. Note: Rates vary based on expertise, skills assessment, location, project needs, and other factors. Higher rates may be offered to highly specialized experts. Lower rates may apply during onboarding or non-core project phases. Payment details are shared per project.

Software Engineer13 hours ago

Role Description Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based. What this opportunity involves: - Building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks. - Creating challenging tasks and evaluation criteria within realistic simulated environments: - Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history. - Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent. - Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient. - Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust. What this is NOT: - Not data labeling. - Not prompt engineering. - Not writing code from scratch - the agent writes most of the code; you guide and evaluate. Qualifications - 5+ years in software development. - Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis. - Experience writing tests (functional, integration). - English proficiency - B2+. Requirements - Deep understanding of where models fail and what scenarios reveal the difference between a good and a bad solution. - Ability to create tasks that genuinely challenge the best models. - Writing tests that accept all correct solutions and reject incorrect ones. Benefits - Compensation up to $40/hr equivalent, depending on level and pace. - Tasks are estimated at ~20 hours each; you set your own schedule. - Effort estimate: Tasks for this project are estimated to take 20 hours to complete, depending on complexity.

Worldwide
$40 / hour
ServiceNow logo

Senior Staff Software Engineer

ServiceNow

ServiceNow provides cloud-based services that automate enterprise information technology operations. As an employer, ServiceNow offers a challenging, collaborat

Software Engineer13 hours ago
Full TimeRemoteTeam 29,000Since 2004

Company Description It all started when engineer Fred Luddy wrote code that automated a tedious task for his coworker, Phyllis. She cried tears of joy. That moment inspired Fred to build a company that could do that for everyone-freeing people from busywork so they could focus on meaningful work. Today, ServiceNow is the AI control tower for business reinvention. Our ServiceNow AI platform brings together any AI, any data, and any workflow- helping 85% of the Fortune 500® work smarter, faster, and better. We're building an AI-native culture where technology and talent are unstoppable together. And we're just getting started. Join us to put AI to work for people. Job Description We are looking for a software engineer and technical lead with deep software design and architecture skills, excellent coding ability, and the technical judgment to build reliable production systems at enterprise scale. ServiceNow's core platform is built on a powerful and highly scaled Java service that supports some of the world's largest enterprise workflows. In this team, we are extending that platform with modern Python and Java microservices that complement the main platform architecture and enable new AI-native capabilities. These services are designed to run in Kubernetes-based environments while integrating deeply with the broader ServiceNow platform. We are focused on production-grade services that bring together enterprise data, AI model integrations, scalable APIs, evaluation workflows, and reliable deployment patterns. You will own the design, development, and operation of containerized backend services running in Kubernetes environments. You will apply strong distributed systems and production reliability judgment to build scalable, maintainable services, and you will lead technical design across ambiguous, high-impact problem spaces. Experience working on AI, GenAI, ML-powered, or data-intensive products will help you be especially successful in this role. Responsibilities - Designing, architecting, and implementing Java and Python microservices deployed on Kubernetes - Designing APIs, service boundaries, data models, and integration patterns for AI-enabled products - Building reliable distributed systems that handle concurrency, queueing, retries, fairness, backpressure, and failure recovery - Integrating frontier AI SDKs such as Anthropic, Google, and OpenAI into production software systems - Applying prompt engineering, structured outputs, model evaluation, and production observability to GenAI use cases - Partnering with research, product, security, and infrastructure teams to ship enterprise-grade services - Raising the engineering bar through architecture reviews, code reviews, mentoring, and technical direction Qualifications To be successful in this role, you have: - 8+ years of professional software engineering experience, with strong fundamentals in data structures, algorithms, distributed systems, APIs, and backend service design - Experience in leveraging or critically thinking about how to integrate AI into work processes, decision-making, or problem-solving (including using AI-powered tools, automating workflows, analyzing AI-driven insights, or exploring AI's potential impact) - Demonstrated experience as a technical lead, Staff Engineer, or equivalent senior individual contributor responsible for architecture, technical direction, code quality, and mentoring - Strong backend engineering skills in Java, Python, or equivalent with hands-on experience designing, building, and operating production services - Hands-on experience building or operating containerized applications deployed in Kubernetes, OpenShift, or similar orchestration environments - Experience designing scalable APIs, asynchronous processing systems, queues, event-driven services, or data-processing pipelines - Familiarity with cloud-native infrastructure, service observability, logging, monitoring, reliability engineering, and production troubleshooting - Ability to make pragmatic architecture decisions across performance, reliability, security, maintainability, and delivery speed - Ability to lead other engineers through code reviews, design reviews, and technical mentorship - Strong communication skills and the ability to partner across product, research, engineering, infrastructure, and security Nice to Have - Prior experience working on AI/ML products, collaborating with research teams, or translating advanced AI/ML capabilities into production software - Knowledge of NLP, search, and knowledge extraction - Experience with multi-modal systems such as document ingestion and processing, image processing, and voice - Working experience with frontier AI SDKs such as Anthropic, Google, or OpenAI - Familiarity with prompt engineering, structured outputs, tool calling, agentic design patterns, Model Context Protocol, or AI-assisted development workflows - Experience with observability tools such as Prometheus, Grafana, Instana, or equivalent monitoring/logging platforms - Knowledge of MLOps or applying machine learning models to production use cases - Experience with infrastructure backends such as PostgreSQL, Redis, Kafka, RabbitMQ, S3-compatible object storage, or equivalent technologies - Published work, patents, conference papers, or open-source contributions related to AI systems, knowledge systems, search, retrieval, or large-scale data processing. Compensation is based on the geographic location in which the role is located and is subject to change based on work location. For positions in this location, we offer a base pay of $190,900 - $334,100, plus equity (when applicable), variable/incentive compensation and benefits. Sales positions generally offer a competitive On Target Earnings (OTE) incentive compensation structure. Please note that the base pay shown is a guideline, and individual total compensation will vary based on factors such as qualifications, skill level, competencies, and work location. We also offer health plans, including flexible spending accounts, a 401(k) Plan with company match, ESPP, matching donations, a flexible time away plan and family leave programs. Compensation is based on the geographic location in which the role is located and is subject to change based on work location. Additional Information Work Personas We approach our distributed world of work with flexibility and trust. Work personas (flexible, remote, or required in office) are categories that are assigned to ServiceNow employees depending on the nature of their work and their assigned work location. Learn more here . To determine eligibility for a work persona, ServiceNow may confirm the distance between your primary residence and the closest ServiceNow office using a third-party service. Equal Opportunity Employer ServiceNow is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, national origin, age, disability, gender identity, veteran status, or any other category protected by law. In addition, all qualified applicants with arrest or conviction records will be considered for employment in accordance with legal requirements. Accommodations We strive to create an accessible and inclusive experience for all candidates. If you require a reasonable accommodation to complete any part of the application process, or are unable to use this online application and need an alternative method to apply, please contact globaltalentss@servicenow.com for assistance. Export Control Regulations For positions requiring access to controlled technology subject to export control regulations, including the U.S. Export Administration Regulations (EAR), ServiceNow may be required to obtain export control approval from government authorities for certain individuals. All employment is contingent upon ServiceNow obtaining any export license or other approval that may be required by relevant export control authorities. From Fortune. ©2026 Fortune Media IP Limited. All rights reserved. Used under license.

California
$190.9K - $334.1K / year
Boeing logo

Lead Java Software Engineer - Developer (Virtual)

Boeing

A leading global aerospace company and top U.S. exporter, Boeing develops, manufactures and services commercial airplanes, defense products and space systems for customers in more than 150 countries. Our U.S. and global workforce and supplier base drive innovation, economic opportunity, sustainability and community impact. Boeing is committed to fostering a culture based on our core values of safety, quality and integrity.

Software Engineer13 hours ago
Full TimeRemoteTeam 10,001+Since 1916H1B Sponsor

Job Description At Boeing, we innovate and collaborate to make the world a better place. We're committed to fostering an environment for every teammate that's welcoming, respectful and inclusive, with great opportunity for professional growth. Find your future with us. The Boeing Company is looking for a Lead Software Engineer - Developer (Virtual) to join the Advanced Ground Architecture team. This position will focus on supporting the Boeing Defense, Space & Security (BDS) Software Engineering organization. The Advanced Ground Architecture (AGA) software team is dynamic group of software engineers creating the future of Ground support with the extensibility and adaptability to be used across ALL Boeing programs. The software team is executing this vision through modern software technologies (Java, ReactJS, python, CI/CD pipelines) and methodologies (Scaled Agile). AGA is looking for self-motivated high performers to lead and execute the large scope of Java development needed for the program's vision. The ideal candidates will provide Java software development leadership for the Advanced Ground Architecture program, specifically the System Planning and Scheduling team. These positions will be responsible for accelerating complex java development, shepherding peer reviews for code merge requests at a faster cycle time, providing sub system leadership on key components, mentoring early and mid career software developers in AGA to be higher performers, and assisting P6 and Chief Engineering on architecture and software system engineering duties and decisions. These responsibilities are in addition to monitoring overall cost and schedule performance to meet or exceed expectations. Position Responsibilities: - Oversees the design, development, test, and maintenance of non-embedded software throughout the end-to-end lifecycle that meets industry, customer, safety, and regulation standards - Oversees the review, analysis, and translation of customer requirements into initial design of software products - Reviews the development, maintenance, enhancement and optimization of software products and functionalities for systems integrations - Oversees development, documentation and maintenance of architectures, requirements, algorithms, interfaces and designs for software products - Oversees debugging and resolution of issues identified to ensure the reliability and efficiency of software products - Oversees research and implementation of current and emerging technologies, tools, frameworks and changes in regulations relevant to software development - Leads development, establishment, monitoring and improvement of software processes, tools and key performance metrics that align projects with organizational goals and industry benchmarks - Reviews planning, execution and documentation of software research and development projects - Consults on software domains, system-specific issues, processes and regulations - Oversees software project management activities and software supplier management functions - Oversees tracking and evaluation of software team and supplier performance to ensure product and process compliance to project plans and industry standards This position has been identified as a virtual opportunity and will not require the selected candidate to relocate. Basic Qualifications (Required Skills/Experience): - Bachelor's Degree - Active Top Secret Security Clearance - Experience working with both Windows and Linux operating systems - 10+ years of experience with Java, C++, Angular/React.js, HTML, and CSS Languages - 10+ years of experience in software configuration management tools - 10+ years of experience in developing software products in a cloud environment Preferred Qualifications (Desired Skills/Experience): - Bachelor of Science degree from an accredited course of study in engineering, engineering technology (includes manufacturing engineering technology), chemistry, physics, mathematics, data science, or computer science and 14+ years of related work experience OR Bachelor's Degree and 18+ years of directly related work experience OR 22+ years of related, relevant experience - Experience with the specification format requirements, guidelines, and change control processes. Understanding of relationship between specifications/standards and aircraft/product certification systems - Experience with host environments, including processors, operating systems and networks. Ability to build, integrate and troubleshoot software components - Experience with deriving and allocating requirements (e.g., system, sub-system, software) and show traceability throughout the product lifecycle - Experience with software architecture and design methodologies. Ability to synthesize software architectures and designs that meet requirements, characteristics and constraints - Experience with software development processes in compliance with established internal and industry standards, guidelines, and best practices in the development, testing, and deployment of software - Experience with software development and testing tools, capability and usage. Examples include compilers, linkers, debuggers, data analysis tools, graphical user interface builders, post processing tools, requirements management tools, and web authoring tools - Experience with the implementation, validation, and verification processes, practices, and guidelines to ensure software development quality, performance, and safety Drug Free Workplace: Boeing is a Drug Free Workplace where post offer applicants and employees are subject to testing for marijuana, cocaine, opioids, amphetamines, PCP, and alcohol when criteria is met as outlined in our policies. CodeVue Coding Challenge: To be considered for this position you will be required to complete a technical assessment as part of the selection process. Failure to complete the assessment will remove you from consideration. Pay & Benefits: At Boeing, we strive to deliver a Total Rewards package that will attract, engage and retain the top talent. Elements of the Total Rewards package include competitive base pay and variable compensation opportunities. The Boeing Company also provides eligible employees with an opportunity to enroll in a variety of benefit programs, generally including health insurance, flexible spending accounts, health savings accounts, retirement savings plans, life and disability insurance programs, and a number of programs that provide for both paid and unpaid time away from work. The specific programs and options available to any given employee may vary depending on eligibility factors such as geographic location, date of hire, and the applicability of collective bargaining agreements. Pay is based upon candidate experience and qualifications, as well as market and business considerations. Summary Pay Range: $184,450 - $286,350 Applications for this position will be accepted until Jul. 13, 2026 Export Control Requirements: This position must meet U.S. export control compliance requirements. To meet U.S. export control compliance requirements, a "U.S. Person" as defined by 22 C.F.R. §120.62 is required. "U.S. Person" includes U.S. Citizen, U.S. National, lawful permanent resident, refugee, or asylee. Export Control Details: US based job, US Person required Education Bachelor's Degree or Equivalent Required Relocation This position offers relocation based on candidate eligibility. Security Clearance This position requires an active U.S. Top Secret Security Clearance (U.S. Citizenship Required). (A U.S. Security Clearance that has been active in the past 24 months is considered active). This position requires ability to obtain program access, for which the U.S. Government requires U.S. Citizenship only. Visa Sponsorship Employer will not sponsor applicants for employment visa status. Shift This position is for 1st shift Equal Opportunity Employer: Boeing is an Equal Opportunity Employer. Employment decisions are made without regard to race, color, religion, national origin, gender, sexual orientation, gender identity, age, physical or mental disability, genetic factors, military/veteran status or other characteristics protected by law.

Virginia
$184.5K - $286.4K / year