AI Evaluation / Test Automation Engineer

Job Overview

Griot Solutions is building a candidate pipeline for an AI Evaluation / Test Automation Engineer supporting potential U.S. Department of Veterans Affairs Enterprise AI Support Services work.

This is a contingent contract opportunity. A funded vacancy and start date have not been confirmed. Placement depends on final contract requirements, assigned work, client approval and final engagement arrangements.

The role focuses on repeatable evaluation and regression testing of approved enterprise AI capabilities. You would develop reproducible test evidence, identify meaningful failures and support informed release recommendations. Final acceptance and risk decisions remain with authorized client representatives.

Key Responsibilities

• Develop repeatable evaluation rubrics, representative test cases and automated checks for approved AI capabilities and integrations within assigned scope.
• Translate approved requirements into measurable success criteria, failure definitions, scoring methods and human-review procedures.
• Evaluate relevant output quality, grounding, citations, retrieval quality, permission boundaries, security, privacy, safety, latency and auditability.
• Use approved evaluation data and follow restrictions on access, retention, reuse and disclosure.
• Record tested versions, configurations, datasets and environments so results can be reproduced and compared with available baselines.
• Identify regressions, changed behavior and material risks affecting continued use.
• Document findings, severity, probable causes, proposed mitigations and retest results.
• Coordinate defect triage and retesting with integration, platform and service-provider teams.
• Prepare concise evidence packages with traceability from requirements through findings and verification.
• Escalate issues requiring restriction, rollback or risk acceptance rather than acting beyond delegated authority.

Relevant Experience

We are interested in candidates who can demonstrate hands-on experience with:

• Automated testing of software, APIs, integrations or data-driven workflows.
• Test plans, reproducible test data, measurable criteria and clear defect reporting.
• Scripting, version control and automated checks within development or release processes.
• Failure analysis, defect investigation, retesting and reporting limitations.
• Communicating results to technical and nontechnical stakeholders.
• Handling sensitive test data under appropriate access, retention and reuse controls.

Experience evaluating AI-assisted or retrieval-based workflows, testing permission boundaries, or supporting release assurance in controlled environments is relevant. Describe your personal contributions rather than presenting team accomplishments as your own.

Role-specific minimum years, degree requirements, certifications, technology platforms and personnel-security requirements have not yet been finalized. No unsupported numeric experience or credential requirement is imposed in this posting.

Work Arrangement and Travel

Griot Solutions is recruiting this role on a remote basis. Any eventual performance location remains subject to the approvals and restrictions of the awarded work.

Limited travel within the continental United States may be required for meetings, reviews, training, readiness or transition activities. The draft requirements contemplate trips of up to three on-site business days each; frequency has not been established.

Assignment Details

Weekly hours, start date, employment classification and assignment duration remain subject to final arrangements. The scope includes annual baseline regression verification, while additional testing depends on assigned and authorized work. A dedicated full-time testing workload is not guaranteed.

How to Apply

Submit a current résumé identifying relevant projects, dates, technologies and your testing responsibilities. Include your work location, availability and a brief example of automated testing or AI evaluation work you personally delivered.

Do not submit confidential client information, protected data or restricted test artifacts.

Pay: $55.00 - $75.00 per hour

Application Question(s):

• What automated testing or AI evaluation work have you personally delivered? Include the technologies, project dates and your responsibilities.
• How have you evaluated AI output quality, grounding, retrieval or permission boundaries using repeatable criteria and appropriately protected test data? …