At Cohere, our mission is to build machines that understand the world, and to make them safely accessible to all.
Data Annotation Specialist, Safety
Location
United States + 1 moreAll locations: United States | Canada
Posted
3 days ago
Salary
$45 / hour
Seniority
Mid Level
Job Description
Data Annotation Specialist, Safety
Cohere
Role Description We are on a mission to build machines that understand the world and make them safely accessible to all. Data quality is foundational to this process. Machines (or Large Language Models to be exact) learn in similar ways to humans: by way of feedback. By labeling, ranking, auditing, prompting, red teaming, and correcting output, you will improve Large Language Model’s performance for iterations to come, thus having a lasting impact on Cohere’s tech. Cohere is looking for Data Annotation Specialists with backgrounds and skills in Trust & Safety, Content Moderation, AI model evaluation, or related fields. This role is best suited for candidates who bring: - Strong contextual judgment - Cultural and bias sensitivity - Experience applying nuanced guidelines to complex or ambiguous content Successful candidates will be highly detail-oriented, comfortable evaluating safety risks across different user intents and scenarios, and able to make consistent, well-reasoned decisions with a high degree of independence after onboarding. IMPORTANT CONTEXT ON THIS ROLE: In this position, you will be asked to engage with human-generated and model-generated tasks which will sometimes mean intentional exposure to explicit content. Your annotations on these explicit tasks will be used to prevent the Large Language Model from generating unintentional, adversarial, toxic, or unsafe outputs. The types of explicit content you may be exposed to may include but are not limited to those of a sexual, violent, or psychologically disturbing nature. Please Note: This is a part-time, remote, independent contractor position available within Canada or the United States. We seek candidates who are able to commit to 16 hours per week minimum at a 45 CAD/hour or 45 USD/hour contract rate, depending on your location, consisting of 30/hour base pay plus 15/hour hazard pay. This role is BYOD 💻 - Bring Your Own Device (laptop). 12 months contract with potential for extension. As a Data Annotation Specialist on safety task, you will: - Evaluate and improve model safety: - Label, rank, audit, and refine human- and model-generated text to improve safety, quality, and policy alignment, including content that may be sexual, violent, or psychologically disturbing. - Apply nuanced safety judgment: - Assess model outputs against detailed safety guidelines, rubrics, and style standards, making consistent decisions across ambiguous, sensitive, and context-dependent cases. - Create prompts and safety test cases: - Write realistic prompts, user scenarios, and adversarial examples that help evaluate model behavior across safety categories and uncover unsafe, evasive, over-refusing, or policy-inconsistent responses. - Support quality and calibration: - Identify annotation inconsistencies or unclear guidelines, and provide actionable feedback on recurring edge cases, model failures, and opportunities to improve data quality. - Work with precision and independence: - Complete annotation tasks with strong attention to detail, while being comfortable working independently in a globally distributed, asynchronous team environment. Qualifications - 1+ years of experience in Content Moderation, Trust and Safety, AI data annotation, LLM evaluation, or a related analytical role, with exposure to quality assurance, red teaming, and/or prompt engineering preferred. - Experience applying detailed guidelines to complex and sensitive content, with strong contextual and sociocultural judgment and the ability to recognize and manage personal bias. - Emotional resilience: Comfort working with content that contains unsafe, explicit, and/or toxic content, including content of a sexual, violent, or psychologically disturbing nature. - Excellent command of written English and the ability to clearly justify content evaluations, including why an output is safe, unsafe, high-quality or low-quality. Bonus points if you are fluent in another language! - Strong attention to detail and commitment to accuracy, with the ability to maintain consistency across high-volume and monotonous tasks. - Strong execution in a remote environment, including good time management, comfort using new tools, and the ability to work independently in a global, asynchronous team. Candidate Journey - Initial Screening - Once you have submitted your application our Talent Team will review your resume and writing samples. - Virtual Annotation Test - This assignment will test your written skill through various language-based tasks, such as a writing sample, interacting with a chat bot, and more. - Video Screen - If selected to move forward, you will have a short video call with a member of our Operations Team! Benefits - An open and inclusive culture and work environment - Work closely with a team on the cutting edge of AI research - Weekly lunch stipend, in-office lunches & snacks - Full health and dental benefits, including a separate budget to take care of your mental health - 100% Parental Leave top-up for up to 6 months - Personal enrichment benefits towards arts and culture, fitness and well-being, quality time, and workspace improvement - Remote-flexible, offices in Toronto, New York, San Francisco, London and Paris, as well as a co-working stipend - 6 weeks of vacation (30 working days!)
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Senior Technical Lead for Data
Peraton CorporationPeraton Corporation, a national security company headquartered in Herndon, Virginia, supplies solutions for mission-critical programs and systems. Founded in 2017, Peraton's missio
Role Description We are looking for a highly experienced Senior Technical Lead for Data to lead engineering and architectural efforts supporting the design, development, and maintenance of large-scale enterprise data systems. The ideal candidate will have deep expertise in data systems architecture, cloud environments, and enterprise database products, with proven leadership managing high-availability mission-critical systems in federal environments. You will oversee engineering teams, manage risk, develop technical performance metrics, and ensure the stable, secure, and efficient operation of data systems integrated with Identity and Access Management (IAM) and cloud platforms. As the Senior Technical Lead for Data, you will serve as the authoritative technical voice for all data-related workstreams, translating complex business and mission requirements into scalable, secure, and compliant data architectures while directly mentoring and guiding a team of data engineers. Sitting at the intersection of data engineering, cloud architecture, data governance, and technical leadership, you will collaborate closely with IAM engineers, platform architects, data scientists, security teams, and program leadership to ensure the data platform is modern, resilient, and aligned with federal data governance mandates and Zero Trust principles. - Lead the design, development, evaluation, testing, and integration of organizational data systems infrastructure, including both hardware and software components. - Ensure all systems meet stability, integrity, and efficient operation requirements in accordance with approved specifications; verify systems are properly maintained and optimized once operational. - Provide technical support for acquisition documentation, milestone reviews, and key operational and test events throughout the program lifecycle. - Oversee the systems engineering team and all engineering activities supporting organizational data and information systems, including networked software and associated operating systems. - Execute and manage risk assessments, develop mitigation strategies, and plan cost, schedule, and execution plans for assigned projects. - Develop and review technical performance metrics to track and communicate system health, delivery progress, and operational readiness. - Plan, manage, and coordinate multiple concurrent tasks, each with strict deadlines across a large-scale, highly available system integrated with business-critical platforms. - Drive continuous improvements and optimizations to database systems, data encryption, storage, and retention strategies. - Collaborate with IAM engineers, platform architects, data scientists, security teams, and program leadership to ensure data solutions align with organizational goals, AWS GovCloud architecture, and IAM frameworks. - Support agile development teams by assessing how enhancements, defects, and other requests impact the architecture and performance of the application. - Demonstrate expert knowledge of recent technological advances in computer science and engineering to inform architectural decisions and modernization efforts. - Provide excellent written and verbal communication to stakeholders at all levels, including executive briefings, technical documentation, and team guidance. Qualifications - Minimum of 12 years with BS/BA or 16 years with a HS diploma/equivalent. - Minimum 10 years of experience in Information Technology supporting complex data systems. - 7+ years of experience working with IBM, Oracle, and cloud-based database products. - Training and certification (mandatory) in target ICAM database products preferred, especially Oracle and DB2. - Strong understanding of software development, enhancement, and maintenance lifecycles. - Demonstrated experience with the planning, management, leadership, and coordination of multiple concurrent tasks with strict deadlines for large-scale, highly available systems integrated with business-critical platforms. - Demonstrated expertise in data encryption, data storage, and retention policies in secure federal environments. - Proven ability to perform complex technical tasks in a highly secure environment. - Strong technical knowledge of IT systems, IAM, and AWS GovCloud services. - Experience with agile/scrum methodologies. - Expert knowledge of recent advances in computer science and engineering. - Must be able to obtain and maintain the required agency clearance. - Must be a US Citizen. Preferred Skills - Experience with federal IT programs, regulatory compliance, and federal acquisition documentation. - Familiarity with enterprise data governance and security frameworks (NIST, FISMA, FedRAMP). - Knowledge of cloud-native database and data warehousing technologies on AWS GovCloud. - Experience supporting ATO processes, System Security Plans (SSPs), and continuous monitoring activities. - Familiarity with Zero Trust Architecture principles and federal mandates (OMB M-22-09, EO 14028). Company Description Peraton is a next-generation national security company that drives missions of consequence spanning the globe and extending to the farthest reaches of the galaxy. As the world’s leading mission capability integrator and transformative enterprise IT provider, we deliver trusted, highly differentiated solutions and technologies to protect our nation and allies. Peraton operates at the critical nexus between traditional and nontraditional threats across all domains: land, sea, space, air, and cyberspace. The company serves as a valued partner to essential government agencies and supports every branch of the U.S. armed forces. Each day, our employees do the can’t be done by solving the most daunting challenges facing our customers. Target Salary Range $135,000 - $216,000. This represents the typical salary range for this position. Salary is determined by various factors, including but not limited to, the scope and responsibilities of the position, the individual’s experience, education, knowledge, skills, and competencies, as well as geographic location and business and contract considerations. Depending on the position, employees may be eligible for overtime, shift differential, and a discretionary bonus in addition to base pay. EEO EEO: Equal opportunity employer, including disability and protected veterans, or other characteristics protected by law.
Role Description We are looking for a motivated and technically solid L1 Data Engineer to join our growing Data & Analytics team. In this role, you will be responsible for designing, building, and maintaining the data architecture and infrastructure that supports our organization's data strategy. You will work hands-on to develop, test, and deploy reliable data solutions — ensuring pipelines are scalable, efficient, and aligned with business requirements. This is an ideal opportunity for a data professional who is eager to deepen their expertise in cloud-native data platforms, particularly within the Microsoft Azure and Databricks ecosystem, and who thrives in a collaborative, fast-paced environment. - Design, develop, and maintain scalable data pipelines and ETL/ELT workflows to support business intelligence and analytics use cases. - Build and optimize data ingestion processes using Azure Data Factory and Databricks, ensuring data quality and consistency across all layers of the data platform. - Transform and process large datasets using PySpark and Python, applying best practices for performance and maintainability. - Write and optimize complex SQL queries to support analytical reporting and data validation requirements. - Collaborate with data architects and senior engineers to implement and maintain data models aligned with organizational standards. - Monitor, troubleshoot, and resolve pipeline failures and data quality issues, applying root-cause analysis to prevent recurrence. - Contribute to documentation of data pipelines, data dictionaries, and engineering standards. - Support the team in exploring and evaluating new tools and approaches to continuously improve the data infrastructure. Qualifications - 2+ years of professional experience in a Data Engineering or closely related role. - Strong proficiency in Python for data processing, transformation, and automation tasks. - Hands-on experience with Pandas for data manipulation and PySpark for distributed data processing. - Practical experience with Databricks, including notebook development, clusters, and job orchestration. - Experience building and managing data pipelines with Azure Data Factory. - Working knowledge of Azure Synapse Analytics, particularly Spark pool integration. - Solid SQL skills, including query writing, optimization, and performance tuning. - Familiarity with data engineering principles including incremental loading, data lake architecture, and Delta Lake. - Understanding of data governance and security concepts within a cloud data platform. Requirements - Experience with SQL Server migration projects, including schema conversion and data movement. - Exposure to Terraform for Azure infrastructure provisioning and management. - Familiarity with CI/CD practices applied to data engineering workflows. - Experience with Delta Sharing or Lakehouse Federation concepts. Certification Requirement Candidates are expected to hold or be actively working toward the Databricks Certified Data Engineer Associate certification. This certification validates foundational knowledge across the following domains: - Databricks Lakehouse Platform architecture and capabilities - ETL and ELT workflows using Spark SQL and PySpark - Incremental data processing and structured streaming - Production pipeline development and orchestration - Data governance and security within the Databricks environment
• Design, develop, and optimize enterprise-scale data pipelines and ETL/ELT workflows using Azure and Databricks technologies. • Architect and implement scalable data ingestion, transformation, and orchestration processes using Azure Data Factory, Databricks, and Azure Synapse Analytics. • Develop high-performance data transformation frameworks using PySpark, Python, and Spark SQL for large-scale distributed data processing. • Optimize SQL queries, Spark jobs, and data workflows to improve performance, scalability, and cost efficiency. • Lead data migration initiatives, including SQL Server migrations and modernization of legacy data platforms. • Implement and maintain Delta Lake architecture, incremental data loading strategies, and enterprise data lake best practices. • Collaborate with architects and cross-functional teams to design robust and scalable data models aligned with business and governance standards. • Monitor and troubleshoot production pipelines, perform root-cause analysis, and implement preventive measures for recurring issues. • Support CI/CD implementation and infrastructure automation for data engineering workflows. • Mentor junior engineers and contribute to engineering standards, reusable frameworks, and technical best practices. • Create and maintain technical documentation including architecture diagrams, pipeline documentation, and operational runbooks. • Evaluate and recommend modern data engineering tools, frameworks, and optimization strategies.
Role Description As a Data Engineer, He/She will be responsible for designing, building, and maintaining the data architecture and infrastructure required for our organization's data needs. They will also be responsible for developing, testing, and deploying data solutions, ensuring that they meet our organization's needs. Qualifications - Experience working with Python for data analysis and processing. - Proficient in using the Pandas library for data manipulation and analysis. - Experience working with Pyspark for distributed data processing. - Familiar with Databricks for managing big data workloads. - Experience with Azure Synapse Spark for building and managing data pipelines. - Experience with Azure Data Factory for creating and managing data integration workflows. - Proficient in SQL and experienced in optimizing SQL queries for performance. - Experience migrating data from SQL Server to other platforms. - Bonus: Experience with Terraform for Azure. - Knowledge assessed in the Databricks certification: Databricks Certified Data Engineering Associate. - Knowledge assessed in the Databricks certification: Databricks Certified Data Engineering Professional. Company Description
