pixel-bench

An open benchmark for pixel-art reconstruction. Point it at a folder of native (1x) pixel art. It distorts every image the way the real world does - fractional and non-square upscales, blur, JPEG, noise, grid drift, AI-upscaler mush, painterly fake-pixel texture - then runs any number of reconstruction methods and scores them on the signals that actually matter: correct native size, correct colours, correct pixel placement, and correct grid alignment.

Tools that claim to turn fake pixel art back into real pixel art have had no common way to be measured against each other. pixel-bench is that yardstick.

Example results

The five built-in methods on suite v1 (43 distortion types, 100 images, 4,300 distorted inputs). Reproduce with pixelbench run and render these with python tools/benchmark_charts.py.

Why it exists

"Pixel art" from image generators, upscalers, and lossy pipelines usually isn't: the pixels drift off the grid, sit at non-integer sizes, blur into their neighbours, and the file is stored at many times its true resolution. A growing set of tools try to recover the real art. Until now there was no standard, reproducible way to say which one is better, or where each one fails. This benchmark fixes that with a versioned distortion suite, ground-truth-based metrics, and a results format that is comparable across tools, machines, and time.

You bring the data

pixel-bench deliberately ships no image corpus. Pixel art is yours to choose, and the benchmark is only meaningful on true native 1x pixel art - images where one stored pixel is one art pixel, with no upscaling or resampling. Point --data at a folder of such images. pixelbench validate will warn you if some look pre-scaled.

Install

pip install -e .                 # core: numpy, scipy, opencv, Pillow
pip install -e ".[all]"          # + optional method backends (unfake, pixelfixer)

Quickstart

# run the whole suite on your 1x art and render a report
pixelbench run --data ./my_pixel_art --out results/run.json
pixelbench report results/run.json                 # -> results/run.md

# a quick pass over a sample of images
pixelbench run --data ./my_pixel_art --limit 200 --methods fixer,snapper

# see what's registered
pixelbench list-methods
pixelbench list-metrics
pixelbench list-suite
pixelbench validate --data ./my_pixel_art          # warn on non-1x inputs
pixelbench preview --image art.png --out preview.png   # see the distortions

What it measures

Each reconstruction is scored on independent signals (higher is better except rel_err and ΔE) - see docs/METRICS.md:

• resolution - exact, within1, within5pct, rel_err: did it recover the true native cols x rows?
• color - delta_e (CIELAB), psnr: are the pixels the right colour?
• placement - pixel_match (share of pixels exactly right) and grid_align: do the detected grid lines actually sit on the real grid? grid_align uses the engine's exact per-pixel source-coordinate ground truth, a signal no prior tool offered.
• palette - ncolor_err, palette_de: did it recover the right colours?

Signals are reported separately, per distortion category and overall, so you see exactly where a method wins and where it breaks. The report also conditions colour on the exact-resolution subset, because raw colour error otherwise flatters methods that over-segment.

Distortion suite

Suite v1 is a fixed, versioned set of named categories (clean_nn, fractional, nonsquare, soft_bilinear, soft_bicubic, jpeg, noise, drift, mush, painterly, kitchen_sink). The distortion applied to a given (image, category) is deterministic, so everyone's numbers are comparable and citable. Details in docs/DISTORTIONS.md.

Built-in methods

name	what it is
fixer	Pixel Art Fixer by Retro Diffusion (algorithm, non-ML). Optional dep: pixelfixer.
snapper	PixelSnapper (bundled faithful port)
pixeldetector	PixelDetector (bundled port)
unfake	unfake.py. Optional dep: unfake.
naive	modal-run-length floor baseline

Missing optional backends are skipped automatically.

Add your own

A new method or metric is a single file with a @register decorator - the runner and CLI pick it up with no other changes. See CONTRIBUTING.md. Adapters for other reconstruction tools are especially welcome; that is how the benchmark becomes the standard.

Results format

One JSON file per run, with per-sample scores and a precomputed aggregate. Documented in docs/RESULTS_FORMAT.md; schema in results/schema.json.

License

MIT. Built and maintained by Retro Diffusion.