Tool Calling

Previous Next

To give your LLM the ability to interact with the outside world, you will need tool calling.

Note that not every model supports tool calling. If the model does not have such an option, it might not call your tools. For reliable tool calling, we recommend trying the Qwen family of models.

Declaring a tool

A tool can be created from any Dart function that returns a String or Future<String>. To perform the conversion, you simply need to use the describeTool() function. To get a good sense of how such a tool can look like, consider this geometry example:

import 'dart:math' as math;
import 'package:quaynor/quaynor.dart' as quaynor;

final circleAreaTool = quaynor.Tool(
  name: "circle_area",
  description: "Calculates the area of a circle given its radius",
  function: ({ required double radius }) {
    final area = math.pi * radius * radius;
    return "Circle with radius $radius has area ${area.toStringAsFixed(2)}";
  }
);

As you can see, every Tool() call needs a function, a name, and a description of what such tool does. To let your LLM use it, simply add it when creating Chat:

final chat = quaynor.Chat.fromPath(
  modelPath: './model.gguf',
  tools: [circleAreaTool]
);

Quaynor then figures out the right tool calling format, inspects the names and types of the parameters, and configures the sampler.

Naturally, more tools can be defined and the model can chain the calls for them:

import 'dart:io';
import 'package:quaynor/quaynor.dart' as quaynor;

final getCurrentDirTool = quaynor.Tool(
  name: "get_current_dir",
  description: "Gets path of the current directory",
  function: () => Directory.current.path
);

final listFilesTool = quaynor.Tool(
  name: "list_files",
  description: "Lists files in the given directory.",
  function: ({required String path}) {
    final dir = Directory(path);
    final files = dir.listSync()
        .where((entity) => entity is File)
        .map((file) => file.path.split('/').last)
        .toList();
    return "Files: ${files.join(', ')}";
  },
  parameterDescriptions : {"path" : "The path to directory you want list. Must be a valid path." }
);

final getFileSizeTool = quaynor.Tool(
  name: "get_file_size",
  description: "Gets the size of a file in bytes.",
  function: ({required String filepath}) async {
    final file = File(filepath);
    final size = await file.length();
    return "File size: $size bytes";
  },
  parameterDescriptions : {"filepath" : "The path to file you wish to know the size of. Must be a valid path." }
);

final chat = await quaynor.Chat.fromPath(
  modelPath: './model.gguf',
  tools: [getCurrentDirTool, listFilesTool, getFileSizeTool],
  templateVariables: {"enable_thinking": false}
);

final response = await chat.ask('What is the biggest file in my current directory?').completed();
print(response); // The largest file in your current directory is `model.gguf`.

Pre-packaged tools

We ship Quaynor with two packaged-in tools, which are general enough for multiple use-cases - monty Python interpreter and bashkit Bash interpreter. Both of them should serve similar purpose - to give your small LLM a better chance to answer questions requiring precise reasoning or some kind of computation, possibly on a big context.

The usage is straightforward. Use the Tool.python() and Tool.bash() factory constructors:

import 'package:quaynor/quaynor.dart' as quaynor;

final chat = await quaynor.Chat.fromPath(
  modelPath: './model.gguf',
  tools: [quaynor.Tool.python(), quaynor.Tool.bash()],
);

Lastly, keep in mind that for most use-cases it is reasonable to constrain the tools with some limits regarding memory and computation time, so that you don't end up executing infinite loop code. To solve this, Tool.python() provides maxDuration, maxMemoryBytes and maxRecursionDepth and Tool.bash() provides maxCommands.

Tool calling and the context

As with everything made to improve response quality, using tool calls fills up the context faster than simply chatting with an LLM. So be aware that you might need to use a larger context size than expected when using tools.

Chat Vision & Hearing