Kimi-K2 Tool Use With VLLM: Troubleshooting & Solutions

Hey everyone! Today, we're diving deep into a tricky issue I've encountered while deploying Kimi-K2 with vLLM, specifically concerning tool use or function calling. I've been trying to get Kimi-K2 to leverage its tool-calling capabilities, but it seems like it's not quite working as expected. I've got my configuration files set up, and everything looks like it should be functional based on the vLLM documentation, but the logs and responses tell a different story. Let's break down the problem, the setup, and the troubleshooting steps I've taken so far. If you guys have any insights or suggestions, please chime in! I am seeking help to fix the issue of vllm deployment of kimi-k2 not supporting tool use.

Understanding the Challenge: Kimi-K2 and Tool Use

In the realm of large language models, tool use represents a significant leap forward. Tool use empowers models like Kimi-K2 to interact with external tools and APIs, enhancing their ability to perform complex tasks. For instance, a model can use a search engine to gather information, a calculator to perform computations, or an API to access real-time data. This capability transforms the model from a mere text generator into a versatile problem-solver. The core of the issue lies in ensuring that the model correctly interprets the user's request, identifies the appropriate tool, formulates the tool call, and then processes the tool's response to generate a coherent answer. This process requires a sophisticated interplay between the model's language understanding, its knowledge of available tools, and its ability to orchestrate complex workflows. When setting up tool use with vLLM and models like Kimi-K2, several key components must work in harmony. First, the model itself must be trained and fine-tuned to support tool calling. Second, the vLLM inference engine needs to be configured to handle tool call requests and responses. Third, the tools themselves must be properly defined and accessible to the model. Any misconfiguration in these areas can lead to tool use failures, such as the model not recognizing the need for a tool, generating incorrect tool calls, or failing to process the tool's output. This article delves into these intricacies, providing a detailed exploration of the steps taken to diagnose and address a specific tool use issue encountered with vLLM and Kimi-K2. By understanding the underlying mechanisms and potential pitfalls, developers can effectively troubleshoot and optimize their setups for seamless tool use integration.

Configuration Breakdown: My vLLM Setup

Let's start by examining my vLLM configuration. I've set up a configuration file that defines the model, API endpoint, and necessary transformers. Here's a snippet of my configuration:

{
  "name": "xxx",
  "api_base_url": "http://xxx:8971/v1/chat/completions",
  "api_key": "token-xx",
  "model": ["kimi-k2"],
  "transformer": {
    "use": [
      [
        "maxtoken",
        {
          "max_tokens": 65536
        }
      ],
      "enhancetool",
      "kimi_k2"
    ]
  }
}

In this configuration, the key parts are the model which specifies "kimi-k2", and the transformer section. I'm using the enhancetool transformer along with kimi_k2, which, according to the vLLM documentation, should enable tool calling for the moonshotai/Kimi-K2-Instruct model. I've also set a high max_tokens value (65536) to accommodate potentially lengthy responses involving tool use. It's crucial to ensure this setup aligns perfectly with the vLLM's requirements for Kimi-K2. The transformer configuration, especially the use array, plays a critical role in defining how the model processes requests and responses, specifically how it handles tool calls. Any discrepancy between the configuration and the model's expected input/output format can lead to malfunctions. Let's further examine the significance of the kimi_k2 flag and the implications of using the enhancetool transformer. The kimi_k2 flag, as highlighted in the documentation, is essential for correctly parsing tool call responses from the Kimi-K2 model. Without this flag, vLLM might not be able to interpret the model's output as a tool call, leading to the model simply generating text instead of invoking the intended tool. The enhancetool transformer, on the other hand, likely adds extra logic to enhance the tool calling process, potentially including features like automatic tool selection, input validation, or response formatting. By understanding the roles of these components, we can better pinpoint potential areas of misconfiguration. The next step involves a thorough review of the vLLM logs and the raw API requests and responses to identify where the tool calling process might be breaking down.

Testing the Waters: Postman Request

To verify if the API supports function calling, I used Postman to send a direct request. The results seemed promising, indicating that the Kimi-K2 model does support function calls. This is a crucial step because it isolates the problem to the vLLM deployment rather than an inherent limitation of the model itself. Here’s a glimpse of the Postman response:

[Image of Postman Response Showing Support for Function Calls]

This response confirms that the model is capable of generating tool calls, which means the problem likely lies in how vLLM is processing these calls or in the configuration settings. The Postman test serves as a baseline, verifying that the model can produce the expected output when directly queried. However, it doesn't provide insight into how vLLM handles the request and response lifecycle, including the parsing of tool call requests and the execution of the called functions. Therefore, the next step is to carefully analyze the requests and responses passing through the vLLM deployment to identify any discrepancies or errors. We need to examine whether the vLLM is correctly forwarding tool call requests to the Kimi-K2 model and whether it is properly interpreting the model's responses, particularly the tool call instructions. This analysis might involve inspecting the raw JSON payloads exchanged between the client, vLLM, and the Kimi-K2 model. By scrutinizing these interactions, we can gain a deeper understanding of where the tool calling process deviates from the expected behavior.

Request Details: What I Sent

Here's a look at the request I sent via Postman:

[Image of Postman Request Content]

This request showcases the structure of the message I'm sending to the API, which includes instructions and context for the model. I am sending a prompt that should trigger the tool use functionality of Kimi-K2. However, the important takeaway here is that the request itself seems correctly formatted and designed to elicit a tool call. If the model were functioning as expected, it should recognize the need for a tool and respond with a tool call request in its response. Given that the Postman test indicated the model's capability to generate tool calls, this observation further strengthens the hypothesis that the issue lies within the vLLM deployment or its configuration. The request serves as a crucial input for diagnosing the problem because it provides the starting point for the interaction. By analyzing the request alongside the corresponding response and the vLLM logs, we can trace the flow of information and pinpoint where the process diverges from the expected behavior. The key question now becomes: what happens to this request as it passes through vLLM? Is the request being correctly forwarded to the Kimi-K2 model? Is the model's response being correctly parsed by vLLM? These are the questions we need to answer by delving deeper into the logs and the internal workings of vLLM.

Log Analysis: The Story in the Logs

Now, let's dissect the logs. This is where things get interesting. Here’s a snippet from the logs:

[2025-08-06T03:32:19.569Z] use transformers: [{"max_tokens":65536,"options":{"max_tokens":65536}},{"name":"enhancetool"}]
[2025-08-06T03:32:19.570Z] final request: http://xxx:8971/v1/chat/completions  {"method":"POST","headers":{},"body":"{\"messages\":[{\"role\":\"system\",\"content\":[{\"type\":\"text\",\"text\":\"You are Claude Code, Anthropic's official CLI for Claude.\",\"cache_control\":{\"type\":\"ephemeral\"}},{\"type\":\"text\",\"text\":\"\\nYou are an interactive CLI tool that helps users with software engineering tasks. Use the instructions below and the tools available to you to assist the user.\\n\\n... (System Message Content) ...\\n\\n# Code References\\n\\nWhen referencing specific functions or pieces of code include the pattern `file_path:line_number` to allow the user to easily navigate to the source code location.\\n\\n<example>\\nuser: Where are errors from the client handled?\\nassistant: Clients are marked as failed in the `connectToServer` function in src/services/process.ts:712.\\n</example>\\n\\n\",\"cache_control\":{\"type\":\"ephemeral\"}}]},{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"<system-reminder>\\nAs you answer the user's questions, you can use the following context:\\n# important-instruction-reminders\\nDo what has been asked; nothing more, nothing less.\\nNEVER create files unless they're absolutely necessary for achieving your goal.\\nALWAYS prefer editing an existing file to creating a new one.\\nNEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User.\\n\\n      \\n      IMPORTANT: this context may or may not be relevant to your tasks. You should not respond to this context unless it is highly relevant to your task.\\n</system-reminder>\\n\"},{\"type\":\"text\",\"text\":\"帮我写个贪吃蛇小游戏\"},{\"type\":\"text\",\"text\":\"<system-reminder>\\nThis is a reminder that your todo list is currently empty. DO NOT mention this to the user explicitly because they are already aware. If you are working on tasks that would benefit from a todo list please use the TodoWrite tool to create one. If not, please feel free to ignore. Again do not mention this message to the user.\\n</system-reminder>\"}]},{\"role\":\"assistant\",\"content\":\"我来帮你写一个贪吃蛇小游戏。让我使用工具来创建一个完整的HTML游戏。\",\"tool_calls\":[{\"id\":\"functions.TodoWrite:0\",\"type\":\"function\",\"function\":{\"name\":\"TodoWrite\",\"arguments\":\"{\\\