Docs
Sources and provenance

Sources and provenance

The source catalog, static and live sources, freshness, publication lags, vintages and known gaps.

The platform computes only from data that names its source, and it reads only the sources of its catalog. The catalog is code, reviewed like any other change: no deployment, administrator or client can add, alter or remove a source at run time.

Two kinds of source

KindWhat it isHow it arrives
StaticA versioned research dataset held in the platform's own database, loaded from the publisher's checksummed releasePublished by a database connector at its release
LiveA public, keyless APIPulled on its own interval, from six hours to thirty days

Both kinds feed the same pipeline, so no reader needs to know which kind a record came from. The static datasets include UCDP's armed conflict and organized violence data, the Correlates of War datasets and the reference registry of states, organizations and maritime corridors that every other source is placed on. The data page lists the publishers by domain.

Reading the catalog

GET /sources lists every source with:

  • Provenance: title, publisher, license, citation and documentation.
  • Synchronization: the last attempt, the last success, the reason for the last failure and when the next pull is due.
  • Knowledge: what the current data revision holds from the source, and its currency: current, stale, static or unknown.

GET /meta also reports each source's currency and when it last confirmed its data, so a client can show how fresh every input is.

Freshness and publication lags

A live source is pulled only when its interval has elapsed since its last success, and the intervals follow how often each source changes. Values are dated by their publishers' release calendars: an annual statistic for last year typically appears months into this year. The as-of date respects these lags, so an assessment as of a date never uses a value released after it.

Withdrawn sources

A source the catalog no longer lists stops synchronizing. Its records stay readable at the revisions during which they were current.

Known gaps

  • Armed conflict. Conflict objects and the conflict forecasts read cover state-based violence; non-state conflict and one-sided violence are published as annual country series.
  • Sanctions. The catalog covers the EU's regimes and the UN Security Council measures the EU transposes. Measures adopted only by other states, such as the United States or the United Kingdom, are not covered.
  • Forecasting questions. Questions are sampled from Polymarket and Manifold Markets above their liquidity or participation floors; other venues are not read.
  • Displacement. UNHCR statistics cover its mandate and exclude UNRWA's Palestine refugees.
  • Trade. IMF trade statistics publish no reporter data for Taiwan, which appears only as a partner in other states' statistics.

A public, keyless endpoint does not imply unrestricted reuse. Check each source's license and citation in GET /sources before republishing its data.