CountryGuesser

A fine-tuned image classifier that predicts a photo's country from a single street-level photo. On a shared 1,000-image benchmark, it beat a human comparison group by about 6×.

▶ Live demo: tysonjohnson.dev/countryguesser (play the human-vs-model guessing game yourself).

At a Glance

Domain: Computer vision (image geolocation, country-level classification, 180 classes)
Role: Technical lead of a 5-person course project. I built the data pipeline, model, web app, and statistical analysis; teammates also served as the case-study participants
Stack: Python, PyTorch, EfficientNet-B0, OpenStreetView-5M (OSV-5M), Flask (gunicorn) web app
Headline metric: 64.70% top-1 accuracy on the 1,000-image webapp benchmark vs. ~10.85% human average (McNemar χ²=444.7, p<0.001)
Status: Complete; analysis notebook finalized April 2026; live demo deployed

Problem / Context

A street-level photo carries real geographic signal: vegetation, road markings, signage style, terrain. Turning that into a country prediction is still a hard 180-way classification problem. The random baseline is about 0.6%, and even engaged human players rarely break 12%. I built a deep classifier on a 500K-image subset of OpenStreetView-5M (OSV-5M), then ran a structured human-vs-model comparison to find where and why it fails.

Approach

I fine-tuned an EfficientNet-B0 backbone on 500K images drawn from OSV-5M, treating country-level prediction as a 180-class classification problem. EfficientNet-B0 was chosen for its accuracy-to-parameter-count ratio: large enough to learn country-level visual cues (vegetation, road markings, signage style, terrain), compact enough to train within the available compute budget (Colab, one A100, 6 epochs). Training ran at batch size 32 and took roughly 4.7 hours of wall-clock time.

To measure progress rigorously, I kept a held-out test split separate from the validation set. An early checkpoint (the "old model") was already deployed to a webapp when I recruited case-study participants; the final fine-tuned checkpoint (the "new model") was later re-scored on the exact same 1,000 webapp images, giving a clean paired comparison with no data leakage between the two model evaluations.

For the human-vs-model analysis I built a Python notebook (April 2026) that ingested 2,579 human guesses across the 1,000 images alongside both model predictions. I used McNemar's test throughout (the right choice for paired binary outcomes) to quantify improvement from old to new checkpoint, and then to compare the new model against human performance.

Results

Benchmark	Accuracy
Random baseline (1/180)	~0.6%
Old model checkpoint (1,000-image webapp set)	32.10%
New model, held-out test split (general)	57.6%
New model, 1,000-image webapp benchmark	64.70%
Validation top-1	75.0%
Human average (5-person case-study cohort)	~10.85%

Model vs. old checkpoint: +32.6 pp improvement on the same 1,000 images (McNemar χ²=272.5, p<0.001).

Model vs. humans: McNemar χ²=444.7, p<0.001 (any-human-correct vs. new model). On the 1,000-image set the model is roughly 6× the human average.

Bar chart of accuracy on the 1,000-image webapp set: old model, new model, five human participants, and random baseline — Accuracy on the 1,000-image webapp set. The new model (64.7%) roughly doubles the old checkpoint (32.1%) and clears every individual human (8.7–15.3%) by a wide margin; random is 0.6%.

Stacked bar chart of pairwise outcomes per participant: both correct, model only, human only, both wrong — Pairwise outcomes, each participant vs. the model. The blue "model-only" band dominates; the orange "human-only" sliver is what drives the few human wins.

Row-normalised confusion matrix heatmap for the top 15 countries — Confusion matrix (row-normalised %, top 15 countries). The off-diagonal CA→US cell is the clearest failure mode: 16% of Canadian images are predicted as the US.

Reliability diagram: mean predicted confidence vs. empirical accuracy, binned, marker size by image count — Reliability diagram for the new model. The largest bin (n=542, confidence 0.9–1.0) sits well below the diagonal at ~89% accuracy vs. ~98% confidence, so the model is systematically overconfident.

Per-country accuracy grouped bar chart for the top 20 countries: old model, new model, human average, group any-correct — Per-country accuracy (top 20 by image count). The new model leads humans in nearly every country; performance is highest for data-rich countries (US ~90%, AU ~98%) and lowest for sparse ones.

Key Decisions / What I Learned

The clearest failure mode is the US/Canada confusion: the model predicts US or Canada for 80.9% of Canada's images in the benchmark. Seven of the 22 images where a human beat the model are US↔Canada flips, cases where a person caught a Canada-specific cue (a sign font, a license plate style) that the model collapsed into a shared North American visual space. The model learned region-level priors and missed the fine-grained cues that separate visually similar neighbors.

The model is also overconfident, and I measured by how much. In its top confidence bin (0.9 to 1.0, n=542), empirical accuracy is about 89% against a mean predicted confidence of about 98%. That's a known failure mode of softmax classifiers, and it matters if the score is ever used as a decision threshold, which is why I charted the reliability diagram explicitly instead of only reporting top-1.

Humans and the model fail on the same images more than chance would predict: observed human-only wins are well below what independence would expect (Jimin: 19 observed vs. 41.7 expected; Juheon: 8 observed vs. 30.7 expected). Humans mostly win on images with a strong cultural giveaway, like a US stop sign or a specific license plate.

Limitations

The gap between val (75.0%) and test (57.6%) is real. Roughly 17 percentage points of generalization loss is meaningful and likely reflects distribution shift within OSV-5M: the val and test splits may not be equally representative across countries and capture conditions. I'd want to audit per-country image counts before claiming production readiness.
The human cohort is small. Five case-study participants (n=144 to n=1,000 guesses each) is enough for statistical significance given the effect size, but the human average could shift with a larger or more geographically diverse group of players.
Overconfidence is uncorrected. The softmax output is not calibrated. A temperature scaling step post-training would bring the reliability diagram closer to the diagonal and make the confidence score more trustworthy.
Class imbalance in OSV-5M. Countries with fewer street-level images in the dataset (e.g., CD, ET) have substantially lower model accuracy. Performance is heavily skewed toward data-rich countries (US: ~90%, AU: ~98%).
Country-level is a coarse target. Predicting country from an image is an interesting benchmark but the practical geolocation use case usually requires finer resolution. This project does not address city or coordinate-level prediction.
The webapp's answer dropdown lists only 161 countries. That cap is a legacy constraint from the older deployed model and is a UI artifact, not the model's class count. The committed model classifies all 180 trained countries. In practice only 120 of those 180 appear as ground-truth answers in the 1,000-image benchmark, and 19 of the trained countries are never selectable as a human's guess.

Tech Stack

Modeling: Python, PyTorch, EfficientNet-B0 (fine-tuned), torchvision
Data: OpenStreetView-5M (OSV-5M), 500K-image training subset
Analysis: pandas, NumPy, matplotlib, seaborn, SciPy (McNemar's test)
Webapp / data collection: Flask (gunicorn), Leaflet map frontend, SQLite guess logging
Environment: Google Colab (NVIDIA A100) for training; Linux VPS (nginx + Cloudflare) for the live demo