Data Scientist Job at Mercor, Remote

eWlVSTlHVEt5QkR0R21Mc0E2bkhaTkVIMHc9PQ==
  • Mercor
  • Remote

Job Description

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.

Role Description

We're seeking a data-driven analyst to conduct comprehensive failure analysis on AI agent performance across finance-sector tasks. You'll identify patterns, root causes, and systemic issues in our evaluation framework by analyzing task performance across multiple dimensions (task types, file types, criteria, etc.).

  • Statistical Failure Analysis : Identify patterns in AI agent failures across task components (prompts, rubrics, templates, file types, tags)
  • Root Cause Analysis : Determine whether failures stem from task design, rubric clarity, file complexity, or agent limitations
  • Dimension Analysis : Analyze performance variations across finance sub-domains, file types, and task categories
  • Reporting & Visualization : Create dashboards and reports highlighting failure clusters, edge cases, and improvement opportunities
  • Quality Framework : Recommend improvements to task design, rubric structure, and evaluation criteria based on statistical findings
  • Stakeholder Communication : Present insights to data labeling experts and technical teams

Qualifications

  • Statistical Expertise : Strong foundation in statistical analysis, hypothesis testing, and pattern recognition
  • Programming : Proficiency in Python (pandas, scipy, matplotlib/seaborn) or R for data analysis
  • Data Analysis : Experience with exploratory data analysis and creating actionable insights from complex datasets
  • AI/ML Familiarity : Understanding of LLM evaluation methods and quality metrics
  • Tools : Comfortable working with Excel, data visualization tools (Tableau/Looker), and SQL

Requirements

  • Experience with AI/ML model evaluation or quality assurance
  • Background in finance or willingness to learn finance domain concepts
  • Experience with multi-dimensional failure analysis
  • Familiarity with benchmark datasets and evaluation frameworks
  • 2-4 years of relevant experience

Job Tags

Remote job,

Similar Jobs

Unicity Care Management

Clinical Liaison-RN (Sales) Job at Unicity Care Management

 ...As a Clinical Liaison, your primary responsibility is to generate referrals and increase visibility of Unicitys services by building trusted relationships with healthcare providers and professional advisors. You will actively engage with physicians, senior living communities... 

Reno Orthopedic Center

OR Circulator Registered Nurse Job at Reno Orthopedic Center

 ...Reno Orthopedic Center wants you to join the team as an OR Circulator Registered Nurse! We are seeking an enthusiastic and dedicated team member to provide patient care by keeping them informed of the procedures, medications and what to expect in surgery at our main ROC... 

Rubenstein Law, P.A.

Legal Assistant/Secretary Job at Rubenstein Law, P.A.

 ...every single client with the excellent care and attention that has built the Firm's reputation and success. Job Summary The Legal Assistant works alongside their Attorney, handling cases in the Pre-Litigation phase while maintaining an open line of communication with... 

Lollipups Pet Resort

Assistant Zoo Director Job at Lollipups Pet Resort

 ...Unique petting zoo needs an experienced individual to oversee zoo. Job Details: ~Salaried position & based on experience. ~Housing is provided (if needed). ~Hands-on background in animal care. ~Have leadership qualities. ~Be able to lift 50 pounds or more... 

One World Global Services

Spanish:Interpreters Job at One World Global Services

 ...English proficiency~ Steady wired internet connection ~ USB Wired headset ~ Windows 10or MacOS WHAT WE OFFER YOU: Fully remote position with a clear schedule Full-time or part-time position Immediate availability to start working after onboarding...