Toolbench Method
How we test
The short version: we only call a tool tested if we ran it ourselves, on our own machine, against the same Godot 4 task. Here is the long version.
01
The test box
An 8 core Linux machine with no GPU and Godot 4.5 stable run headless. We do not use paid API keys for tests yet, so coding tools run against a free local model through Ollama, or not at all.
02
The prompt
Every coding tool gets this request, word for word:
Create player.gd for a Godot 4 2D platformer. It must extend CharacterBody2D and use typed GDScript 2.0. Requirements: walk left and right with the ui_left and ui_right actions at 200 pixels per second, jump with ui_accept only when on the floor with a jump velocity of -400, apply gravity read from the physics/2d/default_gravity project setting, and call move_and_slide() with no arguments. No Godot 3 APIs.Tools that are not code generators, like MCP servers or model runners, get a task that fits what they do, written on their page.
03
The check
We import the project, parse the script with Godot, then run a harness that puts the player on a CharacterBody2D above a floor, waits 60 physics frames, holds right for 60 more and checks that the player is on the floor and has moved right. If Godot reports errors, we paste them back to the tool. It gets two fix rounds.
We also read the code. Passing the harness with untyped code or Godot 3 habits costs points on output.
04
The score
Five areas, each out of 10. The overall score is the plain average.
- Setup: how long and how many snags to get it running
- Output: does the result work in the engine, and is it code we would keep
- Speed: wall-clock time for the task on our box
- Value: what you pay for what you get
- Game-dev fit: how well it handles engine work specifically
These are our judgment from one run, not a benchmark. We write down the version and date so you can see how old a result is.
05
Not tested yet
Some tools need a login, a paid plan or a browser session. Until we run them, their page says "Researched, hands-on test queued", has no score, and only repeats facts from the vendor's own pages with the date we checked.