Puzzle difficulty: why move count is only one measure
A short solution can hide among many choices. Compare route counts, calculate bits of surprise and see why a puzzle difficulty score still needs player feedback.

A shortest solution tells you how many moves a puzzle needs. It does not tell you how easy that solution is to notice. Two boards can have the same goal and very different numbers of choices along the way. They can also differ in how clearly they show what a move will do.
When I make a puzzle, I want to know both whether it can be solved and what sort of thinking it asks for. Those are separate questions. A numerical difficulty score can help describe the second one, provided its assumptions stay visible.
Start with a small model you can check
Imagine a sequence of six decisions. At every decision you have two choices, and exactly one complete sequence succeeds. There are 2 to the power of 6 possible sequences, which is 64. A person choosing uniformly at random has a 1 in 64 chance of selecting the successful sequence.
Now keep the six decisions but offer four choices each time. There are 4 to the power of 6 sequences, which is 4,096. With one successful sequence, a uniform random attempt succeeds 1 time in 4,096. The route length is unchanged, while the second chance is 64 times smaller.
These are calculated examples, rather than results from people playing a game. They show how move count can leave out information about the choices. They do not establish that the second puzzle takes a person 64 times longer.
Keep the route length. Change the choices.
One successful sequence, equally likely choices, and no early stops or merged positions.
4,096possible sequences
1 in 4,096chance of success
12.00 bitssurprise of success
4 choices raised to 6 steps gives 4,096 sequences.
This is a probability model. It does not predict a player's solving time.
The calculator assumes the same number of choices at every step, one successful full sequence, and equally likely choices. Every sequence runs for exactly the chosen number of steps. Changing any of those assumptions changes the probability.
Why express a probability in bits?
For a probability p, the quantity minus log base 2 of p is measured in bits. It is often called the self-information or surprise of an outcome. MIT’s introduction to information explains the same relationship between probability and information.
In the two-choice example, p is 1 divided by 64, so the score is 6 bits. In the four-choice example, p is 1 divided by 4,096, so it is 12 bits. An additional bit means half the probability under the same model. It does not mean one more move or one more second of thinking.
Bits are useful because multiplying probabilities becomes adding scores. If one independent choice succeeds half the time and another succeeds a quarter of the time, their joint probability is one eighth. Their scores add too: 1 bit plus 2 bits is 3 bits.
Real puzzle boards do not have a constant branching tree
In Plumb, a tilt moves every marble at once. Some directions change nothing. Different tilt sequences can reach the same board. Other moves lose a marble down a pit. A constant four-way tree therefore describes possible input sequences, rather than a count of distinct reachable positions.
Plumb’s calculation follows the board’s actual changing positions and the tilts that change them. It tracks the probability of reaching a shortest solution under its random-choice model. The resulting bits score is one design measurement alongside the goal and the positions considered before a shortest clear. The maths of tilting explains that calculation in more detail.
That model is deliberately simpler than a person. A player can read a wall, recognise a familiar pattern, use undo to inspect a consequence, or remember a previous attempt. Uniform random choices capture none of those abilities.
Count what the player can understand too
Suppose two boards have the same route length and the same random-choice score. One clearly shows where a marble will stop. The other makes that consequence hard to see. The numbers alone give no basis for treating their presentation as equally readable.
This is where I would inspect the player’s experience separately:
- Can a new player tell which pieces move together?
- Does the board explain the consequence of a move before it matters?
- Is a mistake easy to understand and recover from?
- Does the puzzle require a new idea, or repeat one the player already knows?
- Is difficulty coming from the decision or from an awkward control?
Those are design questions to investigate, rather than quantities the calculator has measured. I would watch people play before assigning them a numerical weight.
Evaluate a generator on more than solvability
A procedural puzzle generator can produce valid boards while still repeating a narrow kind of experience. The research textbook Procedural Content Generation in Games, chapter 12 describes evaluating generators through their expressive range and through player feedback. That gives a useful distinction: measurements describe what the generator produces, while player observation helps establish how that content is experienced.
For a puzzle maker, a practical first comparison is to group generated boards by two measurements, such as shortest route length and number of meaningful choices. Inspect examples from different groups. Then ask players to try a small selection and describe where they had to stop and think. This is a proposed evaluation process, rather than a claim that Chaosmos has completed a player study.
What to take from a difficulty score
Use a score to ask a more precise question. If route length rises but the choice score falls, inspect whether the extra moves are mostly forced. If two boards have similar measurements but people struggle with only one, inspect the rule teaching and the visual reading.
A score is useful when it exposes a difference you can investigate. It becomes misleading when it is presented as a complete measurement of human thought. For the game behind these examples, see Plumb and its Google Play download.
