🗺️ An image is a dataset of coordinate–colour pairs

A stored image defines colour only at integer pixel locations. The implicit view instead treats the picture as an unknown continuous function and fits a network to it. The training set is the image itself — one sample per pixel, with the pixel-centre coordinate (normalized to [−1, 1]²) as input and the RGB value as target:

Move the cursor over the image (kodim19, the classic Kodak “lighthouse” test image) to read off training pairs. The picket fence and the railing are dense high-frequency detail — they are exactly what will separate the models below.

Two conventions matter later. Coordinates use pixel centres (a half-pixel offset), so grids of different resolutions sample the same continuous domain consistently — this detail is what makes “render the same network at a new resolution” geometrically meaningful. And reconstruction quality is measured as PSNR = −10·log₁₀ MSE (in dB, for colours in [0, 1]): 20 dB is visibly blurry, 30 dB is a faithful reconstruction at this size.

🌊 Spectral bias: gradient descent learns low frequencies first

Neural networks trained by gradient descent fit the low-frequency content of a target first, and for low-dimensional inputs like a coordinate the high frequencies arrive so slowly that they effectively never come (Rahaman et al., 2019). In neural-tangent-kernel language: the kernel eigenvalues associated with high-frequency components are tiny, so those components converge at a correspondingly tiny rate. The cleanest place to watch it is one dimension. The target below is a sum of three sines — frequencies 0.5, 3 and 8 cycles — and two identical ReLU MLPs fit it, one on the raw coordinate x, one on a Fourier-feature lift of x.

The bars track how much of each frequency component is still missing from each fit (amplitude of the residual at that frequency):

The raw-coordinate MLP kills the 0.5-cycle component almost immediately, grinds down the 3-cycle one, and barely touches the 8-cycle one — more steps do not help, because that component converges exponentially slower than the others. The Fourier-feature model treats all three frequencies as equally easy. This is not underfitting in the usual sense: both networks have the same capacity, and only the input parametrization differs.

🏁 Three networks, one lighthouse

The same failure and the same two classic fixes, now on the actual image. All three models are MLPs of identical size (3 hidden layers, width 36) trained with Adam on minibatches of pixels; the only difference is how the coordinate enters the network.

• ReLU MLP — raw (x, y) in, colours out. The baseline that spectral bias condemns to blur.
• Random Fourier features (Tancik et al., 2020) — lift the coordinate through a fixed random projection before the very same ReLU MLP:
  γ(v) = [ sin(2πBv), cos(2πBv) ],   B ∈ ℝ^m×2,   B_ij ∼ 𝒩(0, σ²)
  In NTK terms the mapping makes the network's kernel stationary with a bandwidth set by σ — the network becomes equally willing to learn every frequency up to that bandwidth. σ is the one knob: too small and the result is still blurry, too large and the network wastes capacity fitting frequencies finer than the pixels, which shows up as slow-to-clear speckle.
• SIREN (Sitzmann et al., 2020) — attack the problem from inside the network instead: replace every ReLU with a sine,
  h_l+1 = sin( ω₀ · (W_l h_l + b_l) )
  with two load-bearing details from the paper: a frequency scale ω₀ = 30, and weights initialized uniformly in ±√(6 / fan_in) / ω₀ so that activations stay well-distributed through depth and deep sine stacks train stably. High frequencies are available from step one.

Press train. The networks fit a 48 × 72 version of the image, live. Watch the fence: it is the first thing the ReLU model gives up on and the last thing the others perfect.

The curves are the quantitative version of the story: the ReLU MLP plateaus almost immediately — its remaining error lives in frequencies it learns exponentially slowly — while SIREN jumps ahead within a couple hundred steps and the Fourier-feature model keeps climbing past both. Try σ = 1 (blurry, the bandwidth is too narrow for the fence) and σ = 12 (speckle that takes a long time to clean up). One honesty footnote: each model uses the learning rate it likes best (3·10⁻³ / 5·10⁻³ / 10⁻³ respectively) so all three converge as fast as they can in a browser; the ranking matches the full-batch PyTorch version of this experiment.

🔍 The payoff: one network, sampled at any resolution

Nothing about a trained f_θ knows or cares about the grid it was trained on — it is a continuous function, and a pixel grid is just one way to sample it. So: train a SIREN on a tiny 24 × 36 version of the lighthouse, then evaluate the same weights on any grid, including non-integer scale factors that a pixel-grid pipeline would need special-case resampling for. This “train once, decode at any resolution” mechanism is exactly what learned super-resolution methods like LIIF build on.

PSNR against the true high-resolution image appears where a ground truth exists (×2 and ×4); at other scales there is nothing exact to compare against.

🖼️ The same race, on any image

Nothing above is specific to the lighthouse: any picture is just another dataset of (coordinate, colour) pairs, and the three parametrizations can be compared on it directly. Drop an image below (or click to choose a file) to rerun the experiment on it — the picture is downscaled to roughly 3,500 pixels, exactly the size the race above trains on, and three freshly initialized networks start fitting it immediately. The file is read locally with the browser's File API and used only as training targets for the JavaScript on this page; it is never uploaded anywhere.

🧭 Where this goes

Everything above generalizes far beyond one photo of a lighthouse. The recipe — coordinates in, signal out, a spectral-bias fix in between — is the backbone of:

• 3-D shapes and scenes — DeepSDF and Occupancy Networks represent geometry as f(x, y, z) → distance/occupancy; NeRF represents a whole scene as f(x, y, z, θ, φ) → (colour, density) and renders it along camera rays. NeRF's positional encoding is the deterministic cousin of the Fourier features above.
• Learned super-resolution — LIIF: the continuous-decoding mechanism from the demo plus a learned prior, which is what finally beats bicubic.
• Compression — the SR network above has ≈3.5k parameters and reproduces the image it was trained on; weights-as-image is an actively studied compression scheme.
• Physics — a SIREN is infinitely differentiable, so it can be supervised through its derivatives: fitted to image gradients, or used as the solution ansatz for PDEs and wave equations.