EvalTask

One declarative eval task.

Example Usage

import { EvalTask } from "@meetkai/mka1/models/components";

let value: EvalTask = {
  id: "<id>",
  type: "math",
  dataset: {},
  promptTemplate: "<value>",
  grader: {
    type: "python",
  },
};

Fields

Field	Type	Required	Description
id	string	✔️	N/A
name	string	➖	N/A
type	components.EvalTaskType	✔️	N/A
dataset	components.EvalDataset	✔️	Dataset backing an eval task.
promptTemplate	string	✔️	N/A
targetTemplate	string	➖	N/A
choices	string[]	➖	N/A
outputExtraction	components.EvalOutputExtraction	➖	N/A
metrics	any[]	➖	N/A
grader	components.EvalPythonGrader	✔️	N/A
preprocess	components.EvalPythonPreprocessor	➖	N/A
fewshot	components.EvalFewshotConfig	➖	N/A
numFewshot	number	➖	lm-eval style alias for fewshot.count.
metadata	Record<string, any>	➖	N/A