EvalTask

One declarative eval task.

Example Usage

import { EvalTask } from "@meetkai/mka1/models/components";

let value: EvalTask = {
  id: "<id>",
  type: "custom",
  dataset: {},
};

Fields

Field	Type	Required	Description
id	string	✔️	N/A
name	string	➖	N/A
type	components.EvalTaskType	✔️	N/A
dataset	components.EvalDataset	✔️	Dataset backing an eval task.
promptTemplate	string	➖	N/A
targetTemplate	string	➖	N/A
choices	string[]	➖	N/A
outputExtraction	components.EvalOutputExtraction	➖	N/A
metrics	components.EvalMetric[]	➖	N/A
grader	components.EvalPythonGrader	➖	N/A
embedding	components.EvalEmbeddingConfig	➖	Configuration of an embedding task: which columns hold the texts and labels, the auxiliary datasets, and the scoring protocol.
preprocess	components.EvalPythonPreprocessor	➖	N/A
fewshot	components.EvalFewshotConfig	➖	N/A
numFewshot	number	➖	lm-eval style alias for fewshot.count.
language	string	➖	BCP-47/ISO language hint passed to the transcription model for transcription tasks (e.g. "ur", "uk").
metadata	Record<string, any>	➖	N/A