Investigate families and exceptions

Use the dataframe df and the fieldwork as fw import from Getting started. We will ask whether the missing second image is concentrated at one site, then save those rows for further investigation.

Compare sites

by=["site"] adds a separate availability summary for each site:

availability = fw.missingness(
    df,
    features=["image_1", "image_2"],
    by=["site"],
)
counts = availability.to_frame("availability")
print(
    counts[["feature", "populated", "missing"]].to_string(index=False)
)

feature  populated  missing
image_1          6        0
image_2          5        1

The overall table still counts all six rows. To read the site summaries, follow the contexts records:

for context in availability["contexts"]:
    site = context["values"]["site"]
    print(site)
    site_counts = pd.DataFrame(context["availability"])
    print(
        site_counts[["feature", "populated", "denominator"]].to_string(
            index=False
        )
    )

North
feature  populated  denominator
image_1          4            4
image_2          4            4
South
feature  populated  denominator
image_1          2            2
image_2          1            2

Both images are present in every North row. At South, the second image is present in one of two rows. This locates the gap without explaining why it occurred.

Follow a finding to its rows

findings = availability.to_frame("findings")
print(findings[["id", "statement"]].to_string(index=False))

id                                           statement
f0                     image_2: populated in some rows
f1             Present: image_1, image_2; absent: none
f2                   Present: image_1; absent: image_2
f3  image_1: populated in all rows within site = North
f4  image_2: populated in all rows within site = North
f5                    Availability within site = North
f6  image_1: populated in all rows within site = South
f7 image_2: populated in some rows within site = South
f8                    Availability within site = South

The overall image_2 availability finding is f0 in this example. image_1 is populated in every row, so it has no availability finding of its own. Inspect all its missing rows:

missing_images = availability.inspect(
    df, "f0", exceptions=True, all_matches=True
)
print(missing_images.to_string(index=False))

 site  exam  image_1  image_2
South     3       31      NaN

Without all_matches=True, inspect returns the representative rows saved with the result (five at most by default). The finding’s exceptions record gives the total and the number not saved. Requesting all matches retrieves every matching row from the original dataframe.

To reuse those rows in another analysis, make a scope: a saved row selection tied to that dataframe.

missing_image_scope = availability.select(
    df, "f0", exceptions=True, name="Missing second image"
)
selected_tree = fw.census(
    df, ["site", "exam"], scope=missing_image_scope
)
print(fw.render_plaintext(selected_tree))

Fieldwork · census
Census (computed, conditional)
  Conditional cohort · 1 evaluated / 6 input rows · 0 missing, 5 restricted exclusions
  path: site > exam
  total: 1 row
    site=South: 1 row
      exam=3: 1 row

Keep using the original df with this scope. If you edit or reorder the dataframe, rerun the analysis before inspecting or selecting rows.

Recognize fields that arrive together

In a wider table, findings can also describe shared availability patterns:

Pattern	What it tells you
availability_family	Columns are filled in on exactly the same rows.
similar_availability	Columns are often filled in together.
presence_implication	When one column is filled in, another usually is too.
mutually_exclusive	Both columns occur in the data, but never on the same row.

Always-missing columns can form an exact family; always-present columns do not, because complete availability is not a finding. Similarity counts shared presence, so two mostly empty columns do not look similar just because both are blank. These patterns describe collection behavior; they do not mean the columns contain equivalent values.

Recognize special missing-value codes

If your export uses -999 for a missing second image, declare that convention:

sentinel_availability = fw.missingness(df, missing={"image_2": [-999]})

Native missing values such as None are recognized automatically. A declared sentinel changes how Fieldwork counts values without replacing them in df. See availability and counting units for the difference between counting rows and counting exams.