Frequency Analysis: How to Break a Cipher by Counting

Frequency analysis is the practice of counting symbols in an enciphered message and matching the counts against what the language normally does. It works because a cipher that replaces one letter with one symbol changes the identities and leaves the proportions untouched. E is still the commonest letter. It is simply wearing a different face.

What is frequency analysis?

Take any English text of reasonable length and count its letters. You will get roughly the same distribution every time: E far ahead, T close behind, then a cluster of A, O, I and N, and a long tail ending in J, Q, X and Z.

Now encipher that text with a substitution cipher. The letters change. The counts do not. Whatever symbol has replaced E will appear as often as E did, which means the commonest symbol in the ciphertext is almost certainly E in disguise.

That is the whole technique. Everything else is refinement.

English letter frequencies

Figures: Robert Lewand, Cryptological Mathematics, via Wikipedia.

The first nine letters carry about seventy percent of everything written in English. The last four carry about half a percent between them. That imbalance is what makes the technique work, and it is also why the middle of the table is nearly useless: C, U, M and W sit so close together that counting cannot separate them. You will place E and T by counting and the middle of the alphabet by reading.

Solvers remember the top of the table as ETAOIN, which is the first six in order. It is worth memorising. The rest is worth knowing exists.

How do you use it?

Four steps, and the first one is mechanical.

1. Count every symbol in the ciphertext. Write the tally somewhere you can see it. Do not estimate.
2. Compare the shape, not the numbers. You are not looking for a symbol that appears 12.7% of the time. You are looking for the one that appears clearly more than the others, then the one behind it. In a short message the percentages will be nowhere near the table.
3. Place your two most confident letters and stop counting. Usually E and T. Two letters placed across a whole message reveal word shapes, and word shapes are faster than arithmetic from here on.
4. Switch to reading. Find the three-letter words containing your E. Find the doubled letters. Let partial words suggest their own endings. Counting opens the door; it does not walk you through the house.

The mistake to avoid is trusting the table too far. A solver who assigns the top six symbols to ETAOIN in order, and treats that as settled, will usually be wrong about most of them and will spend the next twenty minutes defending the error.

Where the technique came from

Al-Kindi wrote the earliest surviving description of it in the ninth century. His treatise set out the method almost exactly as it is used today: take a text in the same language as the message, count its letters, count the symbols in the cipher, and line the two up.

That date matters more than it sounds. It means substitution ciphers were breakable roughly seven hundred years before Europe worked out how to stop them, and that every cipher of that type used in the intervening centuries was, in principle, already open. The people relying on them did not know that. Several of them found out the hard way.

When does counting fail?

Four situations, and the first is the common one.

The message is too short. Under about forty letters, the counts are noise. A single sentence can easily contain more T than E, and no amount of care will fix that. Short messages are solved by word shape, not statistics.

The text is unusual. A message full of names, numbers or technical terms does not follow ordinary English distribution. Nor does one written to avoid a letter deliberately, which is rarer but has happened.

The cipher uses more than one alphabet. This is the real defence. If E is sometimes one symbol and sometimes another, the counts flatten and the peak disappears. That idea produced the Vigenere cipher, and it is why Vigenere held for three centuries after substitution had fallen.

The cipher does not substitute at all. A transposition cipher keeps every letter and scrambles the order. Count it and you will find perfectly ordinary English frequencies attached to nonsense words. That result is itself the diagnosis: normal counts plus unreadable text means the letters were moved, not replaced, which is the one case where a substitution cipher is not what you are holding.