Llm.Evals

Overview

Available Operations

• createSuite - Create an eval suite
• listSuites - List eval suites
• getSuite - Get an eval suite
• deleteSuite - Delete an eval suite
• createSuiteVersion - Create an eval suite version
• listSuiteVersions - List eval suite versions
• getSuiteVersion - Get an eval suite version
• createRun - Create an eval run
• listRuns - List eval runs
• getRun - Get an eval run
• deleteRun - Delete an eval run
• cancelRun - Cancel an eval run
• rerunFailedSamples - Rerun failed eval samples
• retryFailedRun - Retry an eval run
• listSamples - List eval samples
• getArtifacts - Get eval run artifacts
• importHistoricalResults - Import historical eval results from Hugging Face

createSuite

Creates a team-scoped declarative eval suite. Dataset and Python grader files must be uploaded separately with purpose 'evals'.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.createSuite({
    createEvalSuiteRequest: {
      name: "Multilingual QA smoke suite",
      description: "Declarative eval tasks backed by uploaded files or Hugging Face datasets and Python graders.",
      manifest: {
        tasks: [
          {
            id: "spanish_qa",
            type: "custom",
            dataset: {
              source: "huggingface",
              path: "IIC/AQuAS",
              split: "test",
            },
            promptTemplate: "Responde usando el contexto.\n\nContexto: {{context}}\n\nPregunta: {{question}}\n\nRespuesta:",
            targetTemplate: "{{answer}}",
            grader: {
              type: "python",
              contract: "model_backed",
              modelAccess: "mka1",
              fileId: "file_grader123",
            },
            preprocess: {
              type: "python",
              source: "def transform(row):\n    return row\n",
            },
            numFewshot: 1,
          },
        ],
      },
      metadata: {
        "owner": "eval-team",
      },
    },
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsCreateSuite } from "@meetkai/mka1/funcs/llmEvalsCreateSuite.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsCreateSuite(sdk, {
    createEvalSuiteRequest: {
      name: "Multilingual QA smoke suite",
      description: "Declarative eval tasks backed by uploaded files or Hugging Face datasets and Python graders.",
      manifest: {
        tasks: [
          {
            id: "spanish_qa",
            type: "custom",
            dataset: {
              source: "huggingface",
              path: "IIC/AQuAS",
              split: "test",
            },
            promptTemplate: "Responde usando el contexto.\n\nContexto: {{context}}\n\nPregunta: {{question}}\n\nRespuesta:",
            targetTemplate: "{{answer}}",
            grader: {
              type: "python",
              contract: "model_backed",
              modelAccess: "mka1",
              fileId: "file_grader123",
            },
            preprocess: {
              type: "python",
              source: "def transform(row):\n    return row\n",
            },
            numFewshot: 1,
          },
        ],
      },
      metadata: {
        "owner": "eval-team",
      },
    },
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsCreateSuite failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsCreateSuiteMutation
} from "@meetkai/mka1/react-query/llmEvalsCreateSuite.js";

Parameters

Parameter	Type	Required	Description
request	operations.CreateEvalSuiteRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<components.EvalSuiteObject>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

listSuites

Returns eval suites visible to the authenticated team context.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.listSuites({});

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsListSuites } from "@meetkai/mka1/funcs/llmEvalsListSuites.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsListSuites(sdk, {});
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsListSuites failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Query hooks for fetching data.
  useLlmEvalsListSuites,
  useLlmEvalsListSuitesSuspense,

  // Utility for prefetching data during server-side rendering and in React
  // Server Components that will be immediately available to client components
  // using the hooks.
  prefetchLlmEvalsListSuites,
  
  // Utilities to invalidate the query cache for this query in response to
  // mutations and other user actions.
  invalidateLlmEvalsListSuites,
  invalidateAllLlmEvalsListSuites,
} from "@meetkai/mka1/react-query/llmEvalsListSuites.js";

Parameters

Parameter	Type	Required	Description
request	operations.ListEvalSuitesRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<components.EvalSuiteList>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

getSuite

Retrieves an eval suite by ID.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.getSuite({
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsGetSuite } from "@meetkai/mka1/funcs/llmEvalsGetSuite.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsGetSuite(sdk, {
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsGetSuite failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Query hooks for fetching data.
  useLlmEvalsGetSuite,
  useLlmEvalsGetSuiteSuspense,

  // Utility for prefetching data during server-side rendering and in React
  // Server Components that will be immediately available to client components
  // using the hooks.
  prefetchLlmEvalsGetSuite,
  
  // Utilities to invalidate the query cache for this query in response to
  // mutations and other user actions.
  invalidateLlmEvalsGetSuite,
  invalidateAllLlmEvalsGetSuite,
} from "@meetkai/mka1/react-query/llmEvalsGetSuite.js";

Parameters

Parameter	Type	Required	Description
request	operations.GetEvalSuiteRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<components.EvalSuiteObject>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

deleteSuite

Soft-deletes an eval suite and all of its eval runs so they no longer appear in user-facing reads.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.deleteSuite({
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsDeleteSuite } from "@meetkai/mka1/funcs/llmEvalsDeleteSuite.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsDeleteSuite(sdk, {
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsDeleteSuite failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsDeleteSuiteMutation
} from "@meetkai/mka1/react-query/llmEvalsDeleteSuite.js";

Parameters

Parameter	Type	Required	Description
request	operations.DeleteEvalSuiteRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<components.EvalSuiteDeletedObject>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

createSuiteVersion

Creates an immutable suite version. Use this endpoint for edits to an existing eval suite.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.createSuiteVersion({
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
    createEvalSuiteVersionRequest: {
      manifest: {
        tasks: [
          {
            id: "spanish_qa",
            type: "custom",
            dataset: {
              source: "huggingface",
              path: "IIC/AQuAS",
              split: "test",
            },
            promptTemplate: "Responde usando el contexto.\n\nContexto: {{context}}\n\nPregunta: {{question}}\n\nRespuesta:",
            targetTemplate: "{{answer}}",
            grader: {
              type: "python",
              contract: "model_backed",
              modelAccess: "mka1",
              fileId: "file_grader123",
            },
            preprocess: {
              type: "python",
              source: "def transform(row):\n    return row\n",
            },
            numFewshot: 1,
          },
        ],
      },
    },
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsCreateSuiteVersion } from "@meetkai/mka1/funcs/llmEvalsCreateSuiteVersion.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsCreateSuiteVersion(sdk, {
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
    createEvalSuiteVersionRequest: {
      manifest: {
        tasks: [
          {
            id: "spanish_qa",
            type: "custom",
            dataset: {
              source: "huggingface",
              path: "IIC/AQuAS",
              split: "test",
            },
            promptTemplate: "Responde usando el contexto.\n\nContexto: {{context}}\n\nPregunta: {{question}}\n\nRespuesta:",
            targetTemplate: "{{answer}}",
            grader: {
              type: "python",
              contract: "model_backed",
              modelAccess: "mka1",
              fileId: "file_grader123",
            },
            preprocess: {
              type: "python",
              source: "def transform(row):\n    return row\n",
            },
            numFewshot: 1,
          },
        ],
      },
    },
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsCreateSuiteVersion failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsCreateSuiteVersionMutation
} from "@meetkai/mka1/react-query/llmEvalsCreateSuiteVersion.js";

Parameters

Parameter	Type	Required	Description
request	operations.CreateEvalSuiteVersionRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.CreateEvalSuiteVersionResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

listSuiteVersions

Returns immutable versions for an eval suite, including each version manifest.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.listSuiteVersions({
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsListSuiteVersions } from "@meetkai/mka1/funcs/llmEvalsListSuiteVersions.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsListSuiteVersions(sdk, {
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsListSuiteVersions failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Query hooks for fetching data.
  useLlmEvalsListSuiteVersions,
  useLlmEvalsListSuiteVersionsSuspense,

  // Utility for prefetching data during server-side rendering and in React
  // Server Components that will be immediately available to client components
  // using the hooks.
  prefetchLlmEvalsListSuiteVersions,
  
  // Utilities to invalidate the query cache for this query in response to
  // mutations and other user actions.
  invalidateLlmEvalsListSuiteVersions,
  invalidateAllLlmEvalsListSuiteVersions,
} from "@meetkai/mka1/react-query/llmEvalsListSuiteVersions.js";

Parameters

Parameter	Type	Required	Description
request	operations.ListEvalSuiteVersionsRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.ListEvalSuiteVersionsResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

getSuiteVersion

Retrieves an immutable eval suite version and its manifest.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.getSuiteVersion({
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
    version: 1,
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsGetSuiteVersion } from "@meetkai/mka1/funcs/llmEvalsGetSuiteVersion.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsGetSuiteVersion(sdk, {
    suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
    version: 1,
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsGetSuiteVersion failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Query hooks for fetching data.
  useLlmEvalsGetSuiteVersion,
  useLlmEvalsGetSuiteVersionSuspense,

  // Utility for prefetching data during server-side rendering and in React
  // Server Components that will be immediately available to client components
  // using the hooks.
  prefetchLlmEvalsGetSuiteVersion,
  
  // Utilities to invalidate the query cache for this query in response to
  // mutations and other user actions.
  invalidateLlmEvalsGetSuiteVersion,
  invalidateAllLlmEvalsGetSuiteVersion,
} from "@meetkai/mka1/react-query/llmEvalsGetSuiteVersion.js";

Parameters

Parameter	Type	Required	Description
request	operations.GetEvalSuiteVersionRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.GetEvalSuiteVersionResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

createRun

Starts a durable eval run over the selected suite version, tasks, and models.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.createRun({
    createEvalRunRequest: {
      suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
      models: [
        "auto",
      ],
      judgeModel: "auto",
      embeddingModel: "auto",
      generation: {
        temperature: 0,
        maxGenToks: 512,
        until: [
          "<|endoftext|>",
        ],
        doSample: false,
        chatTemplateKwargs: {
          "enable_thinking": false,
        },
        timeoutSeconds: 120,
        maxRetries: 2,
        maxEmptyRetries: 1,
      },
      generationConcurrency: 4,
      graderConcurrency: 2,
      maxWorkflowSampleActivities: 5000,
      metadata: {
        "purpose": "mvp",
      },
    },
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsCreateRun } from "@meetkai/mka1/funcs/llmEvalsCreateRun.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsCreateRun(sdk, {
    createEvalRunRequest: {
      suiteId: "eval_suite_aa87e2b1112a455b8deabed784372198",
      models: [
        "auto",
      ],
      judgeModel: "auto",
      embeddingModel: "auto",
      generation: {
        temperature: 0,
        maxGenToks: 512,
        until: [
          "<|endoftext|>",
        ],
        doSample: false,
        chatTemplateKwargs: {
          "enable_thinking": false,
        },
        timeoutSeconds: 120,
        maxRetries: 2,
        maxEmptyRetries: 1,
      },
      generationConcurrency: 4,
      graderConcurrency: 2,
      maxWorkflowSampleActivities: 5000,
      metadata: {
        "purpose": "mvp",
      },
    },
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsCreateRun failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsCreateRunMutation
} from "@meetkai/mka1/react-query/llmEvalsCreateRun.js";

Parameters

Parameter	Type	Required	Description
request	operations.CreateEvalRunRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.CreateEvalRunResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

listRuns

Returns eval runs visible to the authenticated team context.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.listRuns({});

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsListRuns } from "@meetkai/mka1/funcs/llmEvalsListRuns.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsListRuns(sdk, {});
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsListRuns failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Query hooks for fetching data.
  useLlmEvalsListRuns,
  useLlmEvalsListRunsSuspense,

  // Utility for prefetching data during server-side rendering and in React
  // Server Components that will be immediately available to client components
  // using the hooks.
  prefetchLlmEvalsListRuns,
  
  // Utilities to invalidate the query cache for this query in response to
  // mutations and other user actions.
  invalidateLlmEvalsListRuns,
  invalidateAllLlmEvalsListRuns,
} from "@meetkai/mka1/react-query/llmEvalsListRuns.js";

Parameters

Parameter	Type	Required	Description
request	operations.ListEvalRunsRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.ListEvalRunsResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

getRun

Retrieves an eval run by ID.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.getRun({
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsGetRun } from "@meetkai/mka1/funcs/llmEvalsGetRun.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsGetRun(sdk, {
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsGetRun failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Query hooks for fetching data.
  useLlmEvalsGetRun,
  useLlmEvalsGetRunSuspense,

  // Utility for prefetching data during server-side rendering and in React
  // Server Components that will be immediately available to client components
  // using the hooks.
  prefetchLlmEvalsGetRun,
  
  // Utilities to invalidate the query cache for this query in response to
  // mutations and other user actions.
  invalidateLlmEvalsGetRun,
  invalidateAllLlmEvalsGetRun,
} from "@meetkai/mka1/react-query/llmEvalsGetRun.js";

Parameters

Parameter	Type	Required	Description
request	operations.GetEvalRunRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.GetEvalRunResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

deleteRun

Soft-deletes an eval run so it no longer appears in run lists, details, or score leaderboards.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.deleteRun({
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsDeleteRun } from "@meetkai/mka1/funcs/llmEvalsDeleteRun.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsDeleteRun(sdk, {
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsDeleteRun failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsDeleteRunMutation
} from "@meetkai/mka1/react-query/llmEvalsDeleteRun.js";

Parameters

Parameter	Type	Required	Description
request	operations.DeleteEvalRunRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<components.EvalRunDeletedObject>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

cancelRun

Requests cancellation for a queued or running eval run.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.cancelRun({
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsCancelRun } from "@meetkai/mka1/funcs/llmEvalsCancelRun.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsCancelRun(sdk, {
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsCancelRun failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsCancelRunMutation
} from "@meetkai/mka1/react-query/llmEvalsCancelRun.js";

Parameters

Parameter	Type	Required	Description
request	operations.CancelEvalRunRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.CancelEvalRunResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

rerunFailedSamples

Queues a completed eval run to retry only samples with sample status 'failed'. Retried samples keep the same run ID and sample IDs, replacing errored sample results in place.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.rerunFailedSamples({
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsRerunFailedSamples } from "@meetkai/mka1/funcs/llmEvalsRerunFailedSamples.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsRerunFailedSamples(sdk, {
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsRerunFailedSamples failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsRerunFailedSamplesMutation
} from "@meetkai/mka1/react-query/llmEvalsRerunFailedSamples.js";

Parameters

Parameter	Type	Required	Description
request	operations.RerunFailedEvalSamplesRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.RerunFailedEvalSamplesResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

retryFailedRun

Queues a failed or cancelled eval run to retry in place. Completed samples keep their results, generated-but-unscored cancelled samples resume at scoring, unfinished samples are requeued, and runs with no persisted samples are prepared from scratch with the same run ID.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.retryFailedRun({
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsRetryFailedRun } from "@meetkai/mka1/funcs/llmEvalsRetryFailedRun.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsRetryFailedRun(sdk, {
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsRetryFailedRun failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsRetryFailedRunMutation
} from "@meetkai/mka1/react-query/llmEvalsRetryFailedRun.js";

Parameters

Parameter	Type	Required	Description
request	operations.RetryFailedEvalRunRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<operations.RetryFailedEvalRunResponseBody>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

listSamples

Returns prompt, target, model output, extraction, score, and judge details for an eval run.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.listSamples({
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsListSamples } from "@meetkai/mka1/funcs/llmEvalsListSamples.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsListSamples(sdk, {
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsListSamples failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Query hooks for fetching data.
  useLlmEvalsListSamples,
  useLlmEvalsListSamplesSuspense,

  // Utility for prefetching data during server-side rendering and in React
  // Server Components that will be immediately available to client components
  // using the hooks.
  prefetchLlmEvalsListSamples,
  
  // Utilities to invalidate the query cache for this query in response to
  // mutations and other user actions.
  invalidateLlmEvalsListSamples,
  invalidateAllLlmEvalsListSamples,
} from "@meetkai/mka1/react-query/llmEvalsListSamples.js";

Parameters

Parameter	Type	Required	Description
request	operations.ListEvalSamplesRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<components.EvalSampleList>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

getArtifacts

Returns generated result and sample artifact file IDs for a completed eval run.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.getArtifacts({
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsGetArtifacts } from "@meetkai/mka1/funcs/llmEvalsGetArtifacts.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsGetArtifacts(sdk, {
    runId: "eval_run_aa87e2b1112a455b8deabed784372198",
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsGetArtifacts failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Query hooks for fetching data.
  useLlmEvalsGetArtifacts,
  useLlmEvalsGetArtifactsSuspense,

  // Utility for prefetching data during server-side rendering and in React
  // Server Components that will be immediately available to client components
  // using the hooks.
  prefetchLlmEvalsGetArtifacts,
  
  // Utilities to invalidate the query cache for this query in response to
  // mutations and other user actions.
  invalidateLlmEvalsGetArtifacts,
  invalidateAllLlmEvalsGetArtifacts,
} from "@meetkai/mka1/react-query/llmEvalsGetArtifacts.js";

Parameters

Parameter	Type	Required	Description
request	operations.GetEvalArtifactsRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<components.EvalArtifactsObject>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*

importHistoricalResults

Imports simplified historical samples.jsonl-style eval results from a Hugging Face dataset repository into a completed eval suite run shape. Aggregates are recomputed from samples and include canonical score metrics for leaderboards.

Example Usage

import { SDK } from "@meetkai/mka1";

const sdk = new SDK({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const result = await sdk.llm.evals.importHistoricalResults({
    xHuggingFaceToken: "hf_your_read_token",
    importHistoricalEvalResultsRequest: {
      source: {
        pathPrefix: "qwen/ur/",
      },
      suiteName: "MKA1 historical eval results",
      metadata: {
        "owner": "eval-team",
      },
    },
  });

  console.log(result);
}

run();

Standalone function

The standalone function version of this method:

import { SDKCore } from "@meetkai/mka1/core.js";
import { llmEvalsImportHistoricalResults } from "@meetkai/mka1/funcs/llmEvalsImportHistoricalResults.js";

// Use `SDKCore` for best tree-shaking performance.
// You can create one instance of it to use across an application.
const sdk = new SDKCore({
  bearerAuth: "<YOUR_BEARER_TOKEN_HERE>",
});

async function run() {
  const res = await llmEvalsImportHistoricalResults(sdk, {
    xHuggingFaceToken: "hf_your_read_token",
    importHistoricalEvalResultsRequest: {
      source: {
        pathPrefix: "qwen/ur/",
      },
      suiteName: "MKA1 historical eval results",
      metadata: {
        "owner": "eval-team",
      },
    },
  });
  if (res.ok) {
    const { value: result } = res;
    console.log(result);
  } else {
    console.log("llmEvalsImportHistoricalResults failed:", res.error);
  }
}

run();

React hooks and utilities

This method can be used in React components through the following hooks and associated utilities.

Check out this guide for information about each of the utilities below and how to get started using React hooks.

import {
  // Mutation hook for triggering the API call.
  useLlmEvalsImportHistoricalResultsMutation
} from "@meetkai/mka1/react-query/llmEvalsImportHistoricalResults.js";

Parameters

Parameter	Type	Required	Description
request	operations.ImportHistoricalEvalResultsRequest	✔️	The request object to use for the request.
options	RequestOptions	➖	Used to set various options for making HTTP requests.
options.fetchOptions	RequestInit	➖	Options that are passed to the underlying HTTP request. This can be used to inject extra headers for examples. All Request options, except method and body, are allowed.
options.retries	RetryConfig	➖	Enables retrying HTTP requests under certain failure conditions.

Response

Promise<components.EvalHistoricalImportObject>

Errors

Error Type	Status Code	Content Type
errors.APIError	4XX, 5XX	*/*