Warning
The connectors are experimental and subject to change without prior notice. They are not covered by semantic versioning guarantees, and backward incompatible modifications to these classes will not result in a major version increment. Use them with caution in production environments or critical processes.
.Stat Suite connector
Many statistical organisations disseminate SDMX data through the .Stat Suite platform (OECD, ILO, ABS, the Pacific Data Hub and others). These deployments expose the SDMX-REST v2 API, serving structural metadata as SDMX-ML 2.1 and data as SDMX-CSV.
The pysdmx.api.stat.StatConnector inherits the Fusion
Metadata Registry client (RegistryClient),
so its get_* methods return the same model objects –
get_dataflows returns Dataflow objects, get_codes a
Codelist, and so on. It reads .Stat’s SDMX-ML and parses it with
read_sdmx(). Endpoints .Stat does not serve (e.g.
get_schema – there is no /schema endpoint – or reference
metadata) raise Invalid. On top, it implements
the SDMX data discovery and retrieval profile
(BasicConnector) – dataflows, dataflow
and data – with the same signatures as the other connectors.
Important
To use the StatConnector, install the pysdmx[data,xml]
extras.
Check the installation guide for more information.
Setup
StatConnector defaults to the OECD public service. Known .Stat
Suite deployments are available via the StatEndpoints enum:
from pysdmx.api.stat import StatConnector, StatEndpoints
conn = StatConnector() # OECD (default)
# conn = StatConnector(StatEndpoints.ILO) # or another deployment
Reads are anonymous by default. For an access-controlled deployment,
pass a bearer token (see Uploading for the flows that obtain
one):
conn = StatConnector(
api_endpoint="https://nsi.example.org/rest/v2",
token=auth.get_token(),
)
You can find the agency, id and version of a dataflow in
the OECD Data Explorer via its
“Developer API” button.
Retrieving structures
agency = "OECD.SDD.TPS"
flow_id = "DSD_G20_PRICES@DF_G20_PRICES"
version = "1.0"
# Structures as model objects (the inherited FMR get_* methods)
flows = conn.get_dataflows(agency, flow_id, version) # [Dataflow]
dsds = conn.get_data_structures(agency, "DSD_G20_PRICES", version)
freq = conn.get_codes("SDMX", "CL_FREQ") # a Codelist
The get_* methods return the same model objects as
RegistryClient. To retrieve data, use the
data-discovery profile below.
Data discovery and retrieval profile
StatConnector implements the SDMX data discovery and retrieval
profile (pysdmx.api.dc.BasicConnector) with the same method
names and signatures as the other connectors, so it works wherever that
profile is expected (see the data discovery guide):
ref = f"{agency}:{flow_id}({version})" # or a full/short SDMX URN
conn.dataflows("prices") # list / search dataflows
df = conn.dataflow(ref) # a dataflow + its structure
rows = conn.data(ref, "FREQ='A' AND REF_AREA='CHN'")
dataflows searches id/name/description; data accepts the same
SQL-like / Python-expression / pysdmx.api.dc.query filters as the
other connectors and yields one dict per observation.
Availability-scoped dataflow(ref, filters=...) is not supported on
.Stat (it raises Invalid); use data to query
filtered subsets.
Asynchronous access
pysdmx.api.stat.AsyncStatConnector is the async counterpart
(inheriting AsyncRegistryClient); its get_*
and profile (dataflows/dataflow/data) methods are coroutines:
from pysdmx.api.stat import AsyncStatConnector
conn = AsyncStatConnector()
flows = await conn.dataflows("prices")
df = await conn.dataflow(f"{agency}:{flow_id}({version})")
Uploading
Submitting structures and data requires a writable .Stat Suite instance and an OAuth2 / Keycloak bearer token. .Stat splits submission across two services: structures go to the NSI web service, while data is uploaded to the Transfer service, scoped to a data space and processed asynchronously.
from pysdmx.api.stat import StatUploader
from pysdmx.api.stat.authentication import (
ClientCredentialsAuthentication,
KeycloakAuthentication,
KeycloakDeviceAuthentication,
)
authority = "https://keycloak.example/realms/<realm>"
# Pick ONE of the three flows; each exposes get_token():
# 1. Interactive device flow (browser sign-in) -- the only option for
# federated logins such as GitHub or ADFS. Prints a URL + short
# code and waits for the sign-in to finish.
auth = KeycloakDeviceAuthentication(authority, client_id="app")
# 2. Machine-to-machine (confidential client + service account):
# auth = ClientCredentialsAuthentication(
# authority, client_id="my-client", client_secret="...",
# )
# 3. Resource-owner password grant (local Keycloak accounts only):
# auth = KeycloakAuthentication(
# authority, user="alice", password="...", client_id="app",
# )
uploader = StatUploader(
nsi_endpoint="https://my.stat/nsi",
# The Transfer endpoint includes the API-version segment:
transfer_endpoint="https://my.stat/transfer/3",
dataspace="<data-space>", # the target .Stat data space
token=auth.get_token(), # or any bearer token you already have
)
# Structure first, then data (data submission is asynchronous)
result = uploader.submit(structures=dataflow, dataset=dataset)
# Poll to completion, then check the outcome
final = uploader.submission_status(result.request_id, wait=True)
print(final.execution_status, final.outcome)
submit/submit_data return a SubmissionResult whose
request_id identifies the asynchronous transaction; feed it to
submission_status to obtain the final execution_status and
outcome. Pass wait=True to poll until the transaction reaches a
terminal state (tune the loop with interval and attempts). A
completed-with-warnings import (execution_status="Completed",
outcome="Warning") counts as success=True — inspect outcome
and logs if you need to treat warnings more strictly.
The dataset passed to submit/submit_data must be
Schema-backed — for example one returned by
pysdmx.io.get_datasets(). Its schema must reference a dataflow
(context="dataflow"), not a data structure — .Stat rejects a
datastructure-bound dataset and the failure surfaces only at status-poll
time (submit_data uploads it without complaint). The SDMX action
(Append/Replace/Merge/Delete) is carried inside the file (the SDMX-CSV
2.0 ACTION column, or the SDMX-ML dataset action), not as a request
parameter.
Note
pysdmx.api.stat.authentication offers one class per OAuth2
flow, each taking the realm (authority) URL and exposing
get_token() (which re-runs the flow when the token has expired):
KeycloakDeviceAuthentication
(interactive device flow — the only option for federated logins such
as GitHub or ADFS),
ClientCredentialsAuthentication
(machine-to-machine, with a client id, secret and service account),
and KeycloakAuthentication
(resource-owner password grant, for local Keycloak accounts only).
Alternatively, pass any bearer token obtained elsewhere (for example
the Transfer service’s Swagger “Authorize” button) directly as
token=.
Note
submit_structure reports failures two ways, so check both. A
per-artefact rejection carried in the SubmitStructureResponse
body (even on an HTTP 2xx response) surfaces as
StructureSubmissionResult.success = False (with .messages),
not as an exception. But the NSI can also reject a submission with an
HTTP error – notably 422 Unprocessable Entity with an SDMX error
document – which is raised as Invalid.
submit is the convenience wrapper that submits the structures and
then the data; it raises Invalid if the
structure step does not succeed.
Note
submit_structure serializes to SDMX-JSON 2.0 by default –
the only format whose writer covers every SDMX artefact type (the
SDMX-ML writers do not support the metadata artefacts and have
narrower type coverage). Override with structure_format= for a
deployment that requires SDMX-ML; a type that cannot be written in
the chosen format raises a clear Invalid.
Caveat: some .Stat deployments drop a measure’s local
representation when ingesting SDMX-JSON (the stored DSD loses the
measure’s data type / codelist). If that matters, submit those
structures as SDMX-ML 2.1 (structure_format= with an SDMX-ML
format) or read the artefacts back after upload to verify.
Note
Submitted (and deleted) content must use an agency you are
authorized to maintain on the target instance (the agency must
exist in the SDMX:AGENCIES scheme). The agency is taken from the
artefacts and datasets you pass, so build them under your writable
agency.
Note
A few .Stat submission behaviours worth planning for:
Structure submission is per-artefact, not atomic — a
StructureSubmissionResult.success = Falsecan still mean some artefacts were inserted (inspect.messages).A dangling structure reference (e.g. a dataflow whose DSD is absent) or an unchanged re-submission can surface as HTTP 500 →
InternalErroreven though the cause is client-side; there is no idempotent “ensure exists” upsert.A single invalid observation rejects the whole data batch, valid rows included.
Deleting
Deletion mirrors submission and needs the same bearer token. Delete
observations by uploading them with the SDMX Delete action, and delete
structures with delete_structure().
# Delete observations (SDMX-CSV ACTION=D under the hood)
uploader.delete_data(dataset)
# Delete structures. A data import auto-creates an actual content
# constraint (CR_A_<dataflow>); tear down in dependency order:
uploader.delete_structure([
"DataConstraint=MD:CR_A_DF_EXAMPLE(1.0)",
"Dataflow=MD:DF_EXAMPLE(1.0)",
"DataStructure=MD:DSD_EXAMPLE(1.0)",
"ConceptScheme=MD:CS_EXAMPLE(1.0)",
])
# If the DSD is re-created with changed component types, also clear
# the Transfer data-side objects (else the next import fails):
uploader.cleanup_dsd("DataStructure=MD:DSD_EXAMPLE(1.0)")
delete_structure accepts maintainable artefacts or short-URN strings,
and deletes them in the order given — pass dependents first. It is
fail-fast: the first artefact the NSI rejects (e.g. a still-referenced
one → HTTP 409) raises Invalid and stops the
sequence, so an indeterminate prefix may already be deleted (re-run with
the remainder). The returned list holds one StructureSubmissionResult
per artefact deleted before the failure — informational only; a
failure is signalled by the raised exception, not by .success.
Note
Re-creating a DSD with changed component types (e.g. a measure’s
data type) needs an extra step: the Transfer service keeps per-DSD
data-side objects that NSI deletion does not remove, so the next
import fails (non-backward-compatible change …) until they are
cleared. After deleting the DSD through the NSI, call
cleanup_dsd() (the Transfer
DELETE {transfer}/cleanup/dsd method) as the final teardown step:
delete_data → constraint → dataflow → DSD (→ concept scheme) →
cleanup_dsd.
Endpoints
The connectors take different URL forms:
StatConnector(api_endpoint=…)— the SDMX-REST v2 base, e.g.https://my.stat/rest/v2.StatUploader(nsi_endpoint=…, transfer_endpoint=…)— the NSI host (structures are posted to{nsi}/rest/structure) and the Transfer service including its API-version segment, e.g.https://my.stat/transfer/3.
The StatEndpoints enum lists ready-made read bases for known
public deployments; it does not apply to StatUploader, which takes
raw URLs (writable hosts are deployment-specific). When round-tripping,
the StatConnector read base is usually the NSI host with /rest/v2
appended, e.g. StatConnector(f"{nsi}/rest/v2") (reads are anonymous).