The Hopfield Model — An Interactive Explainer

The Big Idea

What is this model for?

Imagine you see a blurry, half-obscured photo of a face. Somehow your brain fills in the gaps and recognizes the person. The Hopfield model is a mathematical toy that does the same thing — it's a network of binary neurons that can store memories as attractors and recall them from partial or noisy inputs.

Each neuron is either firing (+1) or silent (−1). A "memory" is just a specific pattern of +1s and −1s across all neurons. The network is wired so that once you start near a stored memory, it snaps you all the way there.

Key insight: The weights w_ij encode the memories. The update rule drives the network toward those memories. You never need to search — the physics does it for you.

Equation 2.4

The Update Rule

At each step, each neuron looks at all its inputs and decides whether to fire:

$$S_i := \operatorname{sgn}\!\left(\sum_j w_{ij}\, S_j\right)$$

$S_j$ — the current state of every other neuron $j$ (either $+1$ or $-1$).

$w_{ij}$ — the connection weight from neuron $j$ to neuron $i$. Positive means "fire together", negative means "suppress each other".

$\sum_j w_{ij} S_j$ — the weighted vote. If positive, neuron $i$ fires ($+1$); if negative, it stays silent ($-1$). That's what $\operatorname{sgn}(\cdot)$ does.

Neurons update asynchronously — one at a time, randomly — until nothing changes. That stable state is a stored memory.

Equations 2.7 & 2.9

How to Wire the Weights

Given a set of patterns you want to memorize, how do you set the weights? The answer is the Hebb rule (inspired by neuroscience: "neurons that fire together, wire together").

For one pattern $\xi_i$:

$$w_{ij} = \frac{1}{N}\,\xi_i\,\xi_j$$

For p patterns $\xi^{\mu}$:

$$w_{ij} = \frac{1}{N}\sum_{\mu=1}^{p}\xi_i^{\mu}\,\xi_j^{\mu}$$

Why does this work? If $\xi_i$ and $\xi_j$ agree ($+1,+1$ or $-1,-1$), their product is $+1$ → they excite each other. If they disagree ($+1,-1$), the product is $-1$ → they inhibit each other. The weight encodes whether each pair of neurons is "friends" or "enemies" in a given memory.

Interactive · Weight Matrix for a 4-Neuron Network

Click neurons to toggle a pattern, then see the resulting weight matrix w_ij = (1/N) ξ_iξ_j.

PATTERN ξ (click to toggle)

WEIGHT MATRIX w_ij

Equations 2.10 – 2.12

Recall: Fixing a Noisy Pattern

Once the weights are set, the network can recover a stored pattern from a corrupted version. The math behind why this works: substituting the Hebb rule into the net input gives

$$h_i^{\nu} = \xi_i^{\nu} \;+\; \underbrace{\frac{1}{N}\sum_j\sum_{\mu\neq\nu}\xi_i^{\mu}\,\xi_j^{\mu}\,\xi_j^{\nu}}_{\text{crosstalk from other patterns}}$$

The first term is exactly the stored pattern value $\xi_i^\nu$. If the crosstalk term is small (few patterns stored), then $\operatorname{sgn}(h_i^\nu) = \xi_i^\nu$, so the network faithfully retrieves the memory.

Interactive · Store & Recall on a 5×5 Grid

① Store a pattern

② Corrupt & Recall

Draw a pattern by clicking the cells (black = +1, white = −1), then click Store Pattern.

Start with a stored pattern, add noise, then watch the network recover it.

CURRENT STATE (click to flip)

Equation 2.13

Crosstalk & Storage Capacity

When many patterns are stored, the weight matrix mixes them all together. For each neuron $i$ in pattern $\nu$, there's a crosstalk term:

$$C_i^{\nu} \;\equiv\; -\xi_i^{\nu}\,\frac{1}{N}\sum_j\sum_{\mu\neq\nu}\xi_i^{\mu}\,\xi_j^{\mu}\,\xi_j^{\nu}$$

If $C_i^\nu$ is negative: the crosstalk helps (or is harmless). If it's positive and large: it flips the sign of $h_i^\nu$ and corrupts the memory.

The 0.138N rule: With random patterns, the crosstalk stays manageable only up to about $p \approx 0.138 \times N$ patterns. Beyond that, memories interfere catastrophically. A 1000-neuron network can store ~138 patterns reliably.

Interactive · Crosstalk as a function of stored patterns

Drag the slider to add more patterns and see how crosstalk noise grows.

Patterns stored (p): 2

RECALL RELIABILITY PER PATTERN (N=100 neurons)

Summary

Putting it all together

Neurons take values $+1$ or $-1$. A "memory" is a specific vector of these values.

Hebb rule (Eq. 2.9): set $w_{ij} = \frac{1}{N}\sum_\mu \xi_i^\mu \xi_j^\mu$. Neurons that co-activate in memories get positive weights.

Update rule (Eq. 2.4): each neuron takes the sign of its weighted input $\sum_j w_{ij}S_j$. Repeated until stable — the network has found an attractor.

Capacity: the crosstalk term $C_i^\nu$ (Eq. 2.13) from other memories stays small only up to $p \approx 0.138N$ patterns. Beyond this threshold, retrieval fails.