Warning

The connectors are experimental and subject to change without prior notice. They are not covered by semantic versioning guarantees, and backward incompatible modifications to these classes will not result in a major version increment. Use them with caution in production environments or critical processes.

.Stat Suite connector

Many statistical organisations disseminate SDMX data through the .Stat Suite platform (OECD, ILO, ABS, the Pacific Data Hub and others). These deployments expose the SDMX-REST v2 API, serving structural metadata as SDMX-ML 2.1 and data as SDMX-CSV.

The pysdmx.api.stat.StatConnector inherits the Fusion Metadata Registry client (RegistryClient), so its get_* methods return the same model objects – get_dataflows returns Dataflow objects, get_codes a Codelist, and so on. It reads .Stat’s SDMX-ML and parses it with read_sdmx(). Endpoints .Stat does not serve (e.g. get_schema – there is no /schema endpoint – or reference metadata) raise Invalid. On top, it implements the SDMX data discovery and retrieval profile (BasicConnector) – dataflows, dataflow and data – with the same signatures as the other connectors.

Important

To use the StatConnector, install the pysdmx[data,xml] extras.

Check the installation guide for more information.

Setup

StatConnector defaults to the OECD public service. Known .Stat Suite deployments are available via the StatEndpoints enum:

from pysdmx.api.stat import StatConnector, StatEndpoints

conn = StatConnector()                    # OECD (default)
# conn = StatConnector(StatEndpoints.ILO) # or another deployment

Reads are anonymous by default. For an access-controlled deployment, pass a bearer token (see Uploading for the flows that obtain one):

conn = StatConnector(
    api_endpoint="https://nsi.example.org/rest/v2",
    token=auth.get_token(),
)

You can find the agency, id and version of a dataflow in the OECD Data Explorer via its “Developer API” button.

Retrieving structures

agency = "OECD.SDD.TPS"
flow_id = "DSD_G20_PRICES@DF_G20_PRICES"
version = "1.0"

# Structures as model objects (the inherited FMR get_* methods)
flows = conn.get_dataflows(agency, flow_id, version)  # [Dataflow]
dsds = conn.get_data_structures(agency, "DSD_G20_PRICES", version)
freq = conn.get_codes("SDMX", "CL_FREQ")              # a Codelist

The get_* methods return the same model objects as RegistryClient. To retrieve data, use the data-discovery profile below.

Data discovery and retrieval profile

StatConnector implements the SDMX data discovery and retrieval profile (pysdmx.api.dc.BasicConnector) with the same method names and signatures as the other connectors, so it works wherever that profile is expected (see the data discovery guide):

ref = f"{agency}:{flow_id}({version})"   # or a full/short SDMX URN

conn.dataflows("prices")                 # list / search dataflows
df = conn.dataflow(ref)                  # a dataflow + its structure
rows = conn.data(ref, "FREQ='A' AND REF_AREA='CHN'")

dataflows searches id/name/description; data accepts the same SQL-like / Python-expression / pysdmx.api.dc.query filters as the other connectors and yields one dict per observation. Availability-scoped dataflow(ref, filters=...) is not supported on .Stat (it raises Invalid); use data to query filtered subsets.

Asynchronous access

pysdmx.api.stat.AsyncStatConnector is the async counterpart (inheriting AsyncRegistryClient); its get_* and profile (dataflows/dataflow/data) methods are coroutines:

from pysdmx.api.stat import AsyncStatConnector

conn = AsyncStatConnector()
flows = await conn.dataflows("prices")
df = await conn.dataflow(f"{agency}:{flow_id}({version})")

Uploading

Submitting structures and data requires a writable .Stat Suite instance and an OAuth2 / Keycloak bearer token. .Stat splits submission across two services: structures go to the NSI web service, while data is uploaded to the Transfer service, scoped to a data space and processed asynchronously.

from pysdmx.api.stat import StatUploader
from pysdmx.api.stat.authentication import (
    ClientCredentialsAuthentication,
    KeycloakAuthentication,
    KeycloakDeviceAuthentication,
)

authority = "https://keycloak.example/realms/<realm>"

# Pick ONE of the three flows; each exposes get_token():

# 1. Interactive device flow (browser sign-in) -- the only option for
#    federated logins such as GitHub or ADFS. Prints a URL + short
#    code and waits for the sign-in to finish.
auth = KeycloakDeviceAuthentication(authority, client_id="app")

# 2. Machine-to-machine (confidential client + service account):
# auth = ClientCredentialsAuthentication(
#     authority, client_id="my-client", client_secret="...",
# )

# 3. Resource-owner password grant (local Keycloak accounts only):
# auth = KeycloakAuthentication(
#     authority, user="alice", password="...", client_id="app",
# )

uploader = StatUploader(
    nsi_endpoint="https://my.stat/nsi",
    # The Transfer endpoint includes the API-version segment:
    transfer_endpoint="https://my.stat/transfer/3",
    dataspace="<data-space>",   # the target .Stat data space
    token=auth.get_token(),     # or any bearer token you already have
)

# Structure first, then data (data submission is asynchronous)
result = uploader.submit(structures=dataflow, dataset=dataset)

# Poll to completion, then check the outcome
final = uploader.submission_status(result.request_id, wait=True)
print(final.execution_status, final.outcome)

submit/submit_data return a SubmissionResult whose request_id identifies the asynchronous transaction; feed it to submission_status to obtain the final execution_status and outcome. Pass wait=True to poll until the transaction reaches a terminal state (tune the loop with interval and attempts). A completed-with-warnings import (execution_status="Completed", outcome="Warning") counts as success=True — inspect outcome and logs if you need to treat warnings more strictly.

The dataset passed to submit/submit_data must be Schema-backed — for example one returned by pysdmx.io.get_datasets(). Its schema must reference a dataflow (context="dataflow"), not a data structure — .Stat rejects a datastructure-bound dataset and the failure surfaces only at status-poll time (submit_data uploads it without complaint). The SDMX action (Append/Replace/Merge/Delete) is carried inside the file (the SDMX-CSV 2.0 ACTION column, or the SDMX-ML dataset action), not as a request parameter.

Note

pysdmx.api.stat.authentication offers one class per OAuth2 flow, each taking the realm (authority) URL and exposing get_token() (which re-runs the flow when the token has expired): KeycloakDeviceAuthentication (interactive device flow — the only option for federated logins such as GitHub or ADFS), ClientCredentialsAuthentication (machine-to-machine, with a client id, secret and service account), and KeycloakAuthentication (resource-owner password grant, for local Keycloak accounts only). Alternatively, pass any bearer token obtained elsewhere (for example the Transfer service’s Swagger “Authorize” button) directly as token=.

Note

submit_structure reports failures two ways, so check both. A per-artefact rejection carried in the SubmitStructureResponse body (even on an HTTP 2xx response) surfaces as StructureSubmissionResult.success = False (with .messages), not as an exception. But the NSI can also reject a submission with an HTTP error – notably 422 Unprocessable Entity with an SDMX error document – which is raised as Invalid. submit is the convenience wrapper that submits the structures and then the data; it raises Invalid if the structure step does not succeed.

Note

submit_structure serializes to SDMX-JSON 2.0 by default – the only format whose writer covers every SDMX artefact type (the SDMX-ML writers do not support the metadata artefacts and have narrower type coverage). Override with structure_format= for a deployment that requires SDMX-ML; a type that cannot be written in the chosen format raises a clear Invalid.

Caveat: some .Stat deployments drop a measure’s local representation when ingesting SDMX-JSON (the stored DSD loses the measure’s data type / codelist). If that matters, submit those structures as SDMX-ML 2.1 (structure_format= with an SDMX-ML format) or read the artefacts back after upload to verify.

Note

Submitted (and deleted) content must use an agency you are authorized to maintain on the target instance (the agency must exist in the SDMX:AGENCIES scheme). The agency is taken from the artefacts and datasets you pass, so build them under your writable agency.

Note

A few .Stat submission behaviours worth planning for:

  • Structure submission is per-artefact, not atomic — a StructureSubmissionResult.success = False can still mean some artefacts were inserted (inspect .messages).

  • A dangling structure reference (e.g. a dataflow whose DSD is absent) or an unchanged re-submission can surface as HTTP 500 → InternalError even though the cause is client-side; there is no idempotent “ensure exists” upsert.

  • A single invalid observation rejects the whole data batch, valid rows included.

Deleting

Deletion mirrors submission and needs the same bearer token. Delete observations by uploading them with the SDMX Delete action, and delete structures with delete_structure().

# Delete observations (SDMX-CSV ACTION=D under the hood)
uploader.delete_data(dataset)

# Delete structures. A data import auto-creates an actual content
# constraint (CR_A_<dataflow>); tear down in dependency order:
uploader.delete_structure([
    "DataConstraint=MD:CR_A_DF_EXAMPLE(1.0)",
    "Dataflow=MD:DF_EXAMPLE(1.0)",
    "DataStructure=MD:DSD_EXAMPLE(1.0)",
    "ConceptScheme=MD:CS_EXAMPLE(1.0)",
])

# If the DSD is re-created with changed component types, also clear
# the Transfer data-side objects (else the next import fails):
uploader.cleanup_dsd("DataStructure=MD:DSD_EXAMPLE(1.0)")

delete_structure accepts maintainable artefacts or short-URN strings, and deletes them in the order given — pass dependents first. It is fail-fast: the first artefact the NSI rejects (e.g. a still-referenced one → HTTP 409) raises Invalid and stops the sequence, so an indeterminate prefix may already be deleted (re-run with the remainder). The returned list holds one StructureSubmissionResult per artefact deleted before the failure — informational only; a failure is signalled by the raised exception, not by .success.

Note

Re-creating a DSD with changed component types (e.g. a measure’s data type) needs an extra step: the Transfer service keeps per-DSD data-side objects that NSI deletion does not remove, so the next import fails (non-backward-compatible change …) until they are cleared. After deleting the DSD through the NSI, call cleanup_dsd() (the Transfer DELETE {transfer}/cleanup/dsd method) as the final teardown step: delete_data → constraint → dataflow → DSD (→ concept scheme) → cleanup_dsd.

Endpoints

The connectors take different URL forms:

  • StatConnector(api_endpoint=…) — the SDMX-REST v2 base, e.g. https://my.stat/rest/v2.

  • StatUploader(nsi_endpoint=…, transfer_endpoint=…) — the NSI host (structures are posted to {nsi}/rest/structure) and the Transfer service including its API-version segment, e.g. https://my.stat/transfer/3.

The StatEndpoints enum lists ready-made read bases for known public deployments; it does not apply to StatUploader, which takes raw URLs (writable hosts are deployment-specific). When round-tripping, the StatConnector read base is usually the NSI host with /rest/v2 appended, e.g. StatConnector(f"{nsi}/rest/v2") (reads are anonymous).