fw.missingness

Use fw.missingness to distinguish isolated gaps from the shape of a collection process: fields arriving as a family, one field implying another, or different forms serving different populations. Repeated records can be counted as rows or collapsed into equally weighted entities.

Availability describes whether information is present. Use value_patterns to investigate the contents of populated cells.

Jump to the worked example or the argument reference.

Function reference

fw.missingness(
    df,
    *,
    features=None,
    by=None,
    entity=None,
    unit='rows',
    entity_presence='any',
    min_implication=0.9,
    min_similarity=0.8,
    limits=None,
    scope=None,
    missing=None,
    table_id='table',
    progress=None,
    cancel=None,
    timeout=None,
    safe_errors=None,
)

This page describes Fieldwork 0.3.2. Import the package with import fieldwork as fw. View the documented source.

Arguments

Argument	Type / default	Effect
`df`	pandas.DataFrame; required	The source table. Analysis preserves its values and row order; column labels are identified by their string form, which must be unique.
`features`	iterable of str or None; `None`	Columns to analyze, in candidate-search order; `None` selects all columns, skipping (and listing in `skipped_features`) columns with unsupported cells such as lists. Use a short, deliberate list when identifiers or free text would distract from the question.
`by`	iterable of str or None; `None`	Joint context columns. Each observed combination defines a context; this does not search all subsets of the context columns.
`entity`	str, iterable of str, or None; `None`	One or more columns identifying an entity. With row units, adds entity summaries without changing row availability. With entity units, required; incomplete entity keys are excluded and counted.
`unit`	“rows” or “entities”; `'rows'`	Counting unit for availability, signatures, families, and pair evidence. Entity units give each distinct complete key one vote.
`entity_presence`	“any” or “all”; `'any'`	For entity units, a field is present if any row or every row for that entity is populated. Row units require the default `"any"`; supplying entity keys alone does not aggregate availability.
`min_implication`	float in [0, 1]; `0.9`	Minimum conditional co-presence fraction for A-present ⇒ B-present. Denominator: units where A is present, not all units.
`min_similarity`	float in [0, 1]; `0.8`	Minimum Jaccard similarity for co-presence: both present divided by either present. Shared absences do not inflate the score.
`limits`	mapping or None; `None`	Work and output budgets (`fieldwork.typing.MissingnessLimits`); omitted keys keep their defaults. `max_pairs` (`200`): unordered feature pairs tested, in selected feature order. `max_signatures` (`50`): availability signatures retained, by support. `max_contexts` (`32`): observed joint contexts analyzed. `example_limit` (`5`): source positions saved per example or exception list; `select` and `inspect(..., all_matches=True)` still recover every matching row. Budgets bound search and output, never the population; omissions are counted in `coverage`.
`scope`	Scope or None; `None`	A saved row selection tied to this dataframe; `None` uses every row. Construct with `fw.Scope.from_positions(df, positions)`. Pass the original dataframe with its scope, rather than a reordered or changed copy.
`missing`	mapping or None; `None`	Column names mapped to lists of sentinel values, for example `{"reading": [-999]}`. Native nulls are always missing. Sentinels affect this analysis without replacing source values.
`table_id`	str; `'table'`	Nonempty table identity recorded with the evidence. It does not bypass source-fingerprint checks.
`progress`	bool or callable; `None`	Silent by default; `True` displays progress, or pass a synchronous `ProgressEvent` callback. See runtime controls.
`cancel`	CancellationToken or None; `None`	Cooperative cancellation token; cancellation raises `AnalysisCancelled` without returning a partial result.
`timeout`	float or None; `None`	Finite nonnegative seconds from call start; `None` disables the cooperative deadline. A running pandas/NumPy work item must finish before cancellation.
`safe_errors`	bool or None; `None`	`True` raises any other failure as `AnalysisError`, naming the operation, phase and column without echoing the original message, which can quote cell values. See runtime controls.

Returns and output choices

Returns a fieldwork.Result with kind="missingness" (schema 2.0). Useful sections are availability, families, signatures, contexts, entities, findings, analysis_unit, and coverage; parameters records the settings, including the effective budgets under parameters["limits"]. Use to_frame("availability") for rates and to_frame() for findings you can inspect or select. Context values are plain JSON, such as context["values"]["season"] == "spring".

Use plaintext for a text report, SVG for a static figure, or HTML for a shareable report. to_dict() retains analytical evidence as plain JSON; visualization_data provides the labels and values used to draw a figure.

Interpretation and boundaries

The main denominator is in analysis_unit; context summaries have their own denominators. Entity summaries describe raw-row any/all/one/some/none patterns; these categories overlap (a populated singleton is both “one” and “all”). The summary counts every pattern, but a pattern that matches no entity is not a finding. Each context availability finding states whether all, some or none of the context’s units are populated (structure["presence"]), so topology exports keep that state. Column availability findings and availability families record the same state, "some" or "none": a complete column is not a finding, and its count stays in availability. Under entity counting, the same entity can contribute to more than one context when its rows span contexts.

Identical availability is not evidence that two fields mean the same thing, and vacuous evidence (always-present targets, identical columns) is not reported as a finding. Implication is descriptive and directional. Implication and similarity findings record structure["strength"]: an implication is "exact" when the target is populated on every unit where the source is, and otherwise "approximate" and stated as “approximately implies”. Similar presence is always approximate, because identical availability is reported as a family. Zero-denominator rates are None, not zero. Invalid units, thresholds, or budget values raise ValueError; unknown budget names raise TypeError; unknown columns raise KeyError.

Walkthrough: can four marsh stations support a paired analysis?

A monitoring team revisits four marsh stations, measuring water stage, salinity, and environmental DNA (eDNA). Visits are uneven: Alder has four, Birch and Dogwood three each, and Cedar two. Before comparing eDNA with water conditions, the team needs to distinguish three questions: how many visits contain each measurement, how many stations have ever supplied it, and whether the measurements occur together.

The runnable example contains the dataset and follows the same steps.

Define what a missing measurement looks like

The field logger uses -999 for an unavailable sensor reading. The laboratory uses "NOT_RUN" when an assay was not performed. These codes convey absence in this dataset; an ordinary zero eDNA count would still be a populated measurement. Declare the codes per column rather than silently counting them as observations.

import pandas as pd
import fieldwork as fw

visits = pd.DataFrame(
    [
        ("Alder", "A01", "spring", 11.2, 12.1, 140),
        ("Alder", "A02", "spring", 11.8, 12.4, None),
        ("Alder", "A03", "fall", 9.8, 12.7, "NOT_RUN"),
        ("Alder", "A04", "fall", 10.3, -999, 170),
        ("Birch", "B01", "spring", 6.0, 8.1, 96),
        ("Birch", "B02", "spring", 5.7, 7.9, "NOT_RUN"),
        ("Birch", "B03", "fall", 5.1, 8.5, "NOT_RUN"),
        ("Cedar", "C01", "spring", 4.2, 5.6, None),
        ("Cedar", "C02", "fall", None, 6.0, 78),
        ("Dogwood", "D01", "spring", 2.5, 3.2, 35),
        ("Dogwood", "D02", "spring", 2.9, 3.4, "NOT_RUN"),
        ("Dogwood", "D03", "fall", -999, 3.6, 41),
    ],
    columns=[
        "station",
        "visit",
        "season",
        "stage_cm",
        "salinity_psu",
        "edna_copies",
    ],
)
features = ["stage_cm", "salinity_psu", "edna_copies"]
sentinels = {
    "stage_cm": [-999],
    "salinity_psu": [-999],
    "edna_copies": ["NOT_RUN"],
}

Count usable observations before choosing a population

raw = fw.missingness(visits, features=features)
rows = fw.missingness(
    visits,
    features=features,
    by=["season"],
    missing=sentinels,
    table_id="marsh_visits",
)
coverage = rows.to_frame("availability")
print(
    coverage[
        ["feature", "populated", "missing", "denominator"]
    ].to_string(index=False)
)
print(
    "Without sentinel rules:",
    [item["populated"] for item in raw["availability"]],
)

     feature  populated  missing  denominator
    stage_cm         10        2           12
salinity_psu         11        1           12
 edna_copies          6        6           12
Without sentinel rules: [11, 12, 10]

The default row unit answers a visit-level question. eDNA is available on half of the 12 visits. Without the declared sentinel, its four "NOT_RUN" entries would count as populated, raising the apparent coverage to 10/12. Native nulls remain missing in both calls. These rules classify the data for analysis; they do not replace values in visits.

The by result helps locate the gap without changing the overall denominator:

for context in rows["contexts"]:
    season = context["values"]["season"]
    availability = pd.DataFrame(context["availability"])
    edna = availability[availability["feature"] == "edna_copies"]
    print(season)
    print(
        edna[["feature", "populated", "denominator"]].to_string(
            index=False
        )
    )

spring
    feature  populated  denominator
edna_copies          3            7
fall
    feature  populated  denominator
edna_copies          3            5

Each season supplied three assays. Fall’s larger coverage fraction comes from fewer visit rows, not more assays. The result describes collection coverage; it does not establish why an assay was skipped.

Give each station one vote

For a station inventory, four Alder visits should not carry twice the weight of two Cedar visits. Set unit="entities" and identify stations explicitly. Compare "any" (at least one populated visit) with "all" (every visit populated), separately for each feature.

any_visit = fw.missingness(
    visits,
    features=features,
    entity="station",
    unit="entities",
    entity_presence="any",
    missing=sentinels,
)
every_visit = fw.missingness(
    visits,
    features=features,
    entity="station",
    unit="entities",
    entity_presence="all",
    missing=sentinels,
)
columns = ["feature", "populated", "denominator"]
print("At least one visit:")
print(
    any_visit.to_frame("availability")[columns].to_string(index=False)
)
print("Every visit:")
print(
    every_visit.to_frame("availability")[columns].to_string(index=False)
)

At least one visit:
     feature  populated  denominator
    stage_cm          4            4
salinity_psu          4            4
 edna_copies          4            4
Every visit:
     feature  populated  denominator
    stage_cm          2            4
salinity_psu          3            4
 edna_copies          0            4

All four stations have contributed every measurement at least once. None has eDNA on every visit. Stage is present throughout Alder and Birch’s visits; salinity is present throughout Birch, Cedar, and Dogwood’s. The denominator is now four stations, with no missing station keys to exclude. Equal station weighting does not equalize the opportunity to fail an "all" requirement: a station with more visits has more occasions to miss a measurement.

Check whether the measurements actually coincide

An entity’s "any" presence is computed feature by feature. Cedar has stage on C01 and eDNA on C02; it never has both on the same visit. The apparently complete station inventory therefore cannot justify a paired visit-level analysis.

Display the signatures to find the rows where every measurement is present:

signatures = rows.to_frame("signatures")
print(
    signatures[["finding_id", "present", "absent", "count"]].to_string(
        index=False
    )
)

finding_id                               present         absent  count
        f3              [stage_cm, salinity_psu]  [edna_copies]      6
        f4 [stage_cm, salinity_psu, edna_copies]             []      3
        f5           [salinity_psu, edna_copies]     [stage_cm]      2
        f6               [stage_cm, edna_copies] [salinity_psu]      1

The signature with no absent fields is f4 in this result. Inspect its matching rows:

complete_visits = rows.inspect(visits, "f4", all_matches=True)
print(complete_visits[["station", "visit"]].to_string(index=False))

station visit
  Alder   A01
  Birch   B01
Dogwood   D01

Only three visits, representing three stations, contain the full measurement set. All three are spring visits. Restricting a later analysis to these rows would also remove fall and Cedar; completeness alone does not make that subset representative.

Turn a co-availability finding into a concrete check

The default implication threshold is 0.9. Stage and salinity meet it in one direction: nine of the ten visits with stage also have salinity. This is a conditional presence count, not a physical relationship between water level and salt concentration. Salinity is already present on 11/12 visits overall, so the directional finding is not evidence that stage adds predictive information.

findings = rows.to_frame("findings")
implications = findings[findings["pattern"] == "presence_implication"]
print(implications[["id", "statement"]].to_string(index=False))

id                                                       statement
f7 stage_cm populated approximately implies salinity_psu populated

The stage-to-salinity finding is f7. Use its ID to inspect the exceptions and save them as a selection:

exceptions = rows.inspect(
    visits, "f7", exceptions=True, all_matches=True
)
print(
    exceptions[
        ["station", "visit", "stage_cm", "salinity_psu"]
    ].to_string(index=False)
)

repair_scope = rows.select(
    visits, "f7", exceptions=True, name="Check salinity log"
)

station visit  stage_cm  salinity_psu
  Alder   A04      10.3        -999.0

The exception is A04, still carrying its original sentinel. inspect returns the source row for review; select creates a reusable scope at source position 3. Neither repairs the measurement. The next useful action is to check A04’s field log and the laboratory’s skipped-assay records before deciding which missingness can be resolved.

print(fw.render_plaintext(rows, width=120, max_nodes=1))

Fieldwork · missingness
Population: 12 rows
Search coverage (untested work is unknown)
  missingness: 3/3 pairs evaluated
  missingness: 2/2 contexts retained
  missingness: 4/4 signatures retained
Analysis: 12 rows; presence=per_row
[f0] stage_cm: populated in some rows
  Analysis: rows; presence=per_row
  Unit: rows; populated=10, missing=2, denominator=12, populated_fraction=0.8333333333333334
  Examples: 5/10 saved source positions [0, 1, 2, 3, 4]
  Exceptions: 2/2 saved source positions [8, 11]
... more findings not rendered (max_nodes)

This preview shows one finding; the other findings remain available in the result.

max_nodes=1 shortens the rendered report; it does not shrink the result used above. The five example positions are also a preview, which is why the inspection calls explicitly request all_matches=True. With only three features, this example is below the default pair and signature search limits; a larger audit should check search coverage before interpreting an absent finding.

Continue exploring

fw.value_patterns, fw.compare, fw.explore. See results and configuration for serialization and result-method contracts.