netviz_tools

A Python package that takes a raw edge table to an interactive network chart in three steps: clean it, with a report of every change; build a NetworkX graph; plot it with Plotly. Every step uses the same argument names. Version 1.0 is a ground-up rewrite of the 0.x package I published in 2025, with FAOSTAT agricultural trade data as the worked example of messy input.

At a Glance

Domain: Open-source tooling / network data cleaning and visualization
Role: Solo project
Stack: Python 3.11+, NetworkX, pandas, Plotly, DuckDB, Parquet (pyarrow), pooch, SciPy
Verification: 260 tests at 99.11% coverage, passing on Linux (Python 3.11, 3.12, 3.13) and Windows; strict mypy
Status: v1.0.0 complete, MIT-licensed; PyPI release pending (PyPI still serves 0.2.0)

Problem / Context

Network data rarely arrives in a shape NetworkX can use. Column names differ between sources, the same country is spelled several ways, trade statistics list each flow twice (once by each country) with different numbers, and units get mixed. Once the data is clean, you still have to pick among plotting packages that each want their own input format and have their own quirks.

My own 0.x release from 2025 did not solve that. It bundled about 736 MB of FAOSTAT CSV files inside the wheel, required Python 3.13, and was built around stateful manager classes that printed, logged and plotted as side effects. I rewrote it from the ground up for 1.0; the 0.x API is not compatible.

Approach

Version 1.0 is one pipeline. You can enter at any step: a NetworkX graph you already have goes straight to the plots.

raw table ──clean()──▶ flow table ──build_graph()──▶ NetworkX graph ──plot.<kind>()──▶ Plotly figure

Clean. clean works out which raw column holds each role and says which ones it guessed. It turns reporter/partner rows into source-to-target flows, resolves flows reported by both sides, parses numbers and years written as text, converts units, merges spelling variants, and drops aggregates, zeros and self-loops. It returns a CleaningReport with per-step counts and every dropped row with its reason.
Build. build_graph turns one slice (one period, one category) into a weighted NetworkX graph. graphs_by builds one graph per period.
Plot. Nine chart functions (network, flow map, ego, Sankey, adjacency matrix, ranking, compare, time series, degree distribution) take any NetworkX graph, a flow table, or a mapping from period to graph, and share one set of keyword arguments: color_by, size_by, edge_width_by, top_n, focus, time, labels and a few more.
Name the chart or let it pick. plot.auto(g, kind="network") is the same as plot.network(g). With no kind, it follows documented rules in order: a focus node gives an ego, compare or time-series chart; nodes that all have coordinates give a flow map; a directed graph where no node both sends and receives gives a Sankey; anything else gives a network. The choice and the reason appear under the title and in fig.layout.meta.
Time. Pass several periods and the network, ego and flow-map charts become animations with a play button and a timeline slider. Node positions and scales stay fixed across frames, so movement means change.
Full dataset on request. The wheel ships a 0.6 MB FAOSTAT sample, so every example runs offline. build_store() downloads the full Detailed Trade Matrix (558 items since 1986, about 420 MB) with pooch, checks it against a SHA-256 pinned in the release, and converts it with DuckDB into a Parquet dataset partitioned by item. A manifest.json records the source URL, the hash actually used, the retrieval time, row counts and the library version. load() then reads only the items you ask for.

Results

Cleaning, measured. On the bundled wheat sample, clean turns 44,738 raw FAOSTAT rows into 29,627 flows. The report shows 15,095 flows reported by both countries, 6,994 by the exporter only and 7,538 by the importer only, with a median disagreement of 23.9% between the two reports.
Tested. 260 tests (including Hypothesis property tests and doctests) at 99.11% branch coverage, against a 90% gate. CI runs them on Linux with Python 3.11, 3.12 and 3.13, on Windows, and against the lowest supported dependency versions, plus ruff and strict mypy.
Smaller and wider. The wheel is about 0.75 MB, down from about 736 MB in 0.x, and the minimum Python dropped from 3.13 to 3.11.
Documented. A docs site with a Concepts page, an API reference and five gallery notebooks committed with their executed outputs.

Two examples of what the charts make visible, both from the bundled sample (FAOSTAT quantities in tonnes). The charts show the numbers; they do not say why the numbers changed.

Which wheat exporters fell the most, 2022 to 2023? One call, plot.ranking(flows, category="Wheat", time=[2022, 2023], size_by="out_strength", change=True), ranks exporters by change. Argentina has the largest drop, from 14.5 to 3.1 million tonnes (11.4 million less), ahead of India (7.1 million less) and France (6.4 million less). The largest increase is the Russian Federation, 11.5 million tonnes more.
What does one country send and receive? plot.compare(flows, "Brazil", time=2024) shows that in 2024 Brazil exported 99.8 million tonnes of soya beans and 43.1 million tonnes of maize, and imported 6.6 million tonnes of wheat while exporting 2.9 million. On the animated flow map, Brazil's soya bean exports to mainland China grow from 18.6 million tonnes in 2010 to 74.6 million in 2024.

Key Decisions / What I Learned

I used one generic column vocabulary instead of trade words: source, target, time, category, weight, unit. For trade data the source is the exporter; for migration it is the origin. The same words are the arguments of clean, build_graph and every plot. Display labels travel with the data in flows.attrs["labels"] and G.graph["labels"], so FAOSTAT charts still say Exporter and Importer without the plotting code knowing anything about trade.

Mirror flows are handled in clean, not left to the user. When both countries report the same flow, summing the reports double-counts it and picking one silently hides how far apart they are. clean keeps one report per flow (prefer= chooses which) and records the disagreement in the report.

A graph holds exactly one slice. Building a graph from several periods or categories raises MixedSliceError unless you pass aggregate="sum" or "mean", and mixed units raise MixedUnitError. Live-animal items in FAOSTAT, for example, have quantities in both head and tonnes, and adding those together would give a number that means nothing.

I kept a single rendering backend. Every chart is a Plotly figure with the same colours and hover style, returned to the caller. No function calls show() or writes a file, which is the opposite of 0.x.

A chart that cannot use an argument does not accept it, so passing one raises TypeError instead of being ignored. A silently ignored size_by produces a chart that looks fine and is not the one you asked for. plot.auto adds the chart it chose and why to the error message.

The full-dataset loader defaults to importer-reported flows. The Russian Federation reports no wheat exports after 2021, so exporter-reported data drop it from the 2022 wheat network entirely, while importers report 22.4 million tonnes of Russian wheat in 2022 and 34.0 million in 2023. More countries report imports, too: for wheat in 2021, 155 importing countries report against 99 exporting countries. reporter="exporter" and "combined" are still available.

Limitations

PyPI release pending. v1.0.0 is finished and passes CI, but until the upload, pip install netviz-tools still installs 0.2.0.
Dense graphs. Text labels can overlap in the dense centre of a large network. The defaults label only the 12 largest nodes once a graph has more than 30, and graphs over 2,000 nodes are cut to the 2,000 largest with a warning. For dense graphs, plot.adjacency (a matrix ordered by community) reads better than a node-link diagram.
Layout choice matters. The spectral layout places nodes by Laplacian eigenvectors, so nodes in structurally identical positions, such as many leaves on one hub in a scale-free graph, can land on the same spot. Kamada-Kawai spreads a dense core more evenly but is slow above about 500 nodes.
Power-law fits are not a verdict. The fits report the KS distance but do not run the Clauset et al. bootstrap goodness-of-fit test. In the gallery, a lognormal tail fits as well as a power law in every case, so the data do not support calling these trade networks scale-free.
Centrality depends on who reports. In importer-reported data, Russia's reverse PageRank for wheat falls from 0.105 in 2021 to 0.024 in 2022 while its exports stay at 22.4 million tonnes, because Russia stopped reporting its own imports. The gallery documents this as a reporting artifact; volume questions should use out-strength or shares.
Pinned source file. FAO replaces its bulk file in place, so a build against a newer file raises SourceHashMismatchError until the user opts in with known_hash=None or a new release pins the new hash. The one-time full build needs about 9 GB of temporary disk.

Tech Stack

Graphs and analysis: NetworkX (Louvain communities, PageRank, betweenness), SciPy (power-law fits and likelihood-ratio tests)
Data: pandas, NumPy, DuckDB, Parquet via pyarrow, pooch (download with SHA-256 check)
Visualization: Plotly
Quality: pytest, Hypothesis, pytest-cov (90% gate), mypy (strict), ruff, pre-commit
Docs and release: MkDocs Material, mkdocstrings, mkdocs-jupyter; hatchling; GitHub Actions CI, PyPI Trusted Publishing, Dependabot; MIT license
Data sources: FAOSTAT Detailed Trade Matrix (CC BY 4.0), UNSD M49 regions, Natural Earth label points