fw.grain

Use fw.grain when you have candidate identifiers and want to learn which attributes remain constant within them. It tests exact observed functional dependencies and organizes the results into a grain structure.

A key can identify a group without identifying a row. This function helps separate an order’s attributes from its line items, an observation’s attributes from its repeated measurements, or a work’s attributes from its editions. For automatic and approximate search, use discover_dependencies.

Jump to the worked example or the argument reference.

Function reference

fw.grain(
    df,
    candidate_keys,
    *,
    dropna=False,
    scope=None,
    missing=None,
    table_id="table",
    progress=None,
    cancel=None,
    timeout=None,
    safe_errors=None,
)

This page describes Fieldwork 0.3.2. Import the package with import fieldwork as fw. View the documented source.

Arguments

Argument	Type / default	Effect
`df`	pandas.DataFrame; required	The source table. Analysis preserves its values and row order; duplicate column labels are rejected.
`candidate_keys`	iterable of str, KeySpec or mapping; required	Explicit determinants to evaluate. A column name is a single-column candidate; use `fw.KeySpec("name", ("column_a", "column_b"))`, or its saved form `{"name": "name", "columns": ["column_a", "column_b"]}`, for a composite candidate. Separate names mean separate candidates.
`dropna`	bool; `False`	`False` treats native missing as a value. `True` excludes rows missing any candidate component or the target separately for each dependency test; comparisons may therefore use different populations.
`scope`	Scope or None; `None`	A saved row selection tied to this dataframe; `None` uses every row. Construct with `fw.Scope.from_positions(df, positions)`. Pass the original dataframe with its scope, rather than a reordered or changed copy.
`missing`	mapping or None; `None`	Extra missing-value sentinels per column, such as `{"binding": ["unknown"]}`. Matching values count as missing, so `dropna=True` excludes them; the dataframe is not changed.
`table_id`	str; `"table"`	Name recorded in the result’s `source` so saved evidence says which table it describes.
`progress`	bool or callable; `None`	Silent by default; `True` displays progress, or pass a synchronous `ProgressEvent` callback. See runtime controls.
`cancel`	CancellationToken or None; `None`	Cooperative cancellation token; cancellation raises `AnalysisCancelled` without returning a partial result.
`timeout`	float or None; `None`	Finite nonnegative seconds from call start; `None` disables the cooperative deadline. A running pandas/NumPy work item must finish before cancellation.
`safe_errors`	bool or None; `None`	`True` raises any other failure as `AnalysisError`, naming the operation, phase and column without echoing the original message, which can quote cell values. See runtime controls.

Returns and output choices

Returns a fieldwork.Result with kind="grain". dependencies holds one exact test per candidate and target column: key_name, key_columns, target, holds, group counts (evaluated_groups, violating_groups, repeated_groups, singleton_groups), and its own population (evaluated_rows, missing_excluded_rows, affected_rows). targets summarizes where each column is placed (determining_keys, coarsest_candidates), and graph relates the keys on their common rows. Column names and values are plain JSON. The result also records source, scope, missing_convention, and the parameters that produced it, so recompute(df, dropna=True) reruns it on the same verified dataframe.

Graphical output supports a grain map or a matrix. Matrices put features in rows and candidate keys in columns. The HTML view adds sticky headings, feature search, candidate filtering, and links to placement evidence. Use dictionary access or to_frame("dependencies") for calculations, plaintext for terminal output, SVG for a figure, or HTML for a report.

Interpretation and boundaries

Each supplied candidate is tested against non-component columns. A dependency holds when every eligible determinant group has one target value; it does not require one row per determinant. Empty evidence must not be read as a verified rule. A dependency that holds without a repeated key group (every group is one row) holds trivially; projections record this as repeated_support, and text output says so, including in topology. Each graph node also has a structural role, such as unique identifier or repeated grouping, which topology keeps.

Use KeySpec to make composite candidates explicit. A list such as ["order", "line"] supplies two separate keys, not one two-column key. With dropna=True, each dependency record counts its own complete cases, while the graph compares keys on the rows every key covers; read placements from the graph and per-pair support from dependencies. Invalid or duplicate candidate definitions raise ValueError; unknown columns raise KeyError.

Walkthrough: which facts belong to a book, an edition, or a copy?

A small library exports one row per physical copy. Titles repeat because several copies belong to the same work. Binding and publication year repeat for a different reason: they describe an edition. Before splitting the export into linked tables, we want to check those proposed boundaries against the observed data.

This example uses 12 invented copies: three works, two editions of each work, and two copies of each edition. Download the runnable script. The script follows the same steps and saves the two views shown here.

Make local identifiers explicit

An edition number is only meaningful within a work. Likewise, copy 1 is the first copy of a particular edition, not a globally unique accession number.

import pandas as pd
import fieldwork as fw

# Each edition has two physical copies. Edition and copy numbers are local.
editions = [
    ("W1", "Night Ferries", 1, "hardback", 2018),
    ("W1", "Night Ferries", 2, "paperback", 2021),
    ("W2", "Salt Atlas", 1, "hardback", 2019),
    ("W2", "Salt Atlas", 2, "paperback", 2022),
    ("W3", "Small Weather", 1, "paperback", 2020),
    ("W3", "Small Weather", 2, "hardback", 2023),
]
rows = []
for work, title, edition, binding, year in editions:
    for copy, condition in [(1, "good"), (2, "worn")]:
        rows.append(
            (work, title, edition, copy, binding, year, condition)
        )

catalog = pd.DataFrame(
    rows,
    columns=[
        "work",
        "title",
        "edition",
        "copy",
        "binding",
        "year",
        "condition",
    ],
)
print(f"Rows: {len(catalog)}; works: {catalog['work'].nunique()}")
print("Edition numbers:", sorted(catalog["edition"].unique().tolist()))

Rows: 12; works: 3
Edition numbers: [1, 2]

There are six actual editions despite only two distinct edition numbers. Give grain named composite candidates so it tests the intended groupings. Passing ["work", "edition"] would test two separate single-column candidates.

keys = [
    fw.KeySpec("work", ("work",)),
    fw.KeySpec("edition", ("work", "edition")),
    fw.KeySpec("copy", ("work", "edition", "copy")),
]
result = fw.grain(catalog, candidate_keys=keys)

Separate consistency within groups from unique rows

A dependency holds when every group has one observed target value. Convert the dependency records to a table with to_frame; each record names its target column:

dependencies = result.to_frame("dependencies")
columns = [
    "key_name",
    "target",
    "holds",
    "evaluated_groups",
    "repeated_groups",
    "singleton_groups",
]
print(dependencies[columns].to_string(index=False))

key_name    target  holds  evaluated_groups  repeated_groups  singleton_groups
    work     title   True                 3                3                 0
    work   edition  False                 3                3                 0
    work      copy  False                 3                3                 0
    work   binding  False                 3                3                 0
    work      year  False                 3                3                 0
    work condition  False                 3                3                 0
 edition     title   True                 6                6                 0
 edition      copy  False                 6                6                 0
 edition   binding   True                 6                6                 0
 edition      year   True                 6                6                 0
 edition condition  False                 6                6                 0
    copy     title   True                12                0                12
    copy   binding   True                12                0                12
    copy      year   True                12                0                12
    copy condition   True                12                0                12

Read the work → title and edition → binding rows. Each work has four copies that agree on its title, and each edition has two copies that agree on binding. A work does not determine binding because its editions differ.

The copy candidate determines condition, but each copy group has only one row. No agreement across repeated records was tested. A unique row identifier determines every other column in this table, even an erroneous value.

Read the proposed placement, then see the hierarchy

The target summary names the coarsest supported candidates among those supplied. It does not search for other possible determinants or establish a database schema.

targets = result.to_frame("targets")
print(
    targets[["target", "coarsest_candidates"]].to_string(index=False)
)
svg = fw.render_svg(result, view="map")

   target coarsest_candidates
    title              [work]
  edition                  []
     copy                  []
  binding           [edition]
     year           [edition]
condition              [copy]

Observed grain map: work groups contain titles; edition groups contain binding and year; unique copy groups contain condition.

Read the arrows downward as finer groupings: three works become six editions, then 12 copies. Each box reports both group count and row count, making the change from repeated support to singleton support visible. Attributes appear at their coarsest supported grouping. render_svg returns a string; the image above was saved with the export step below.

Introduce one disagreement without changing the copy identities

Suppose the second copy of Salt Atlas, edition 1, is labeled paperback. We keep both records: perhaps a binding label is wrong, perhaps the volume was rebound, or perhaps the edition identifier has merged two different publications.

# A paperback label appears on the second copy of W2's first edition.
questionable = catalog.copy()
changed = (
    (questionable["work"] == "W2")
    & (questionable["edition"] == 1)
    & (questionable["copy"] == 2)
)
questionable.loc[changed, "binding"] = "paperback"
review = fw.grain(questionable, candidate_keys=keys)
review_table = review.to_frame("dependencies")
binding_check = review_table.loc[
    (review_table["key_name"] == "edition")
    & (review_table["target"] == "binding")
].iloc[0]
print(
    f"Edition -> binding: holds={binding_check['holds']}; "
    f"violating groups={binding_check['violating_groups']}/{binding_check['evaluated_groups']}; "
    f"affected rows={binding_check['affected_rows']}/{binding_check['evaluated_rows']}"
)
print(
    questionable.loc[
        (questionable["work"] == "W2") & (questionable["edition"] == 1),
        ["work", "edition", "copy", "binding"],
    ].to_string(index=False)
)
review_targets = review.to_frame("targets")
binding_target = review_targets[review_targets["target"] == "binding"]
print(
    binding_target[["target", "coarsest_candidates"]].to_string(
        index=False
    )
)

Edition -> binding: holds=False; violating groups=1/6; affected rows=2/12
work  edition  copy   binding
  W2        1     1  hardback
  W2        1     2 paperback
 target coarsest_candidates
binding              [copy]

One of six edition groups violates the dependency. Both rows in that group count as affected, even though we edited only one cell. The row rate is therefore 2/12, not 1/12; it measures membership in conflicting groups, not confirmed errors. All rows are evaluated because this example has no missing values.

Open the native evidence view for the changed catalog to compare binding across candidates in the map, matrix, and placement evidence. The export step below saves this changed-catalog HTML alongside the original figure.

Binding now fits only at the unique-copy level among our candidates. That movement locates a discrepancy; it does not decide whether binding belongs permanently on a copy table. Inspect the two physical volumes and their edition records, resolve the cataloging question, then rerun the same candidates before migrating any fields.

Save the figure or report in your working directory:

from pathlib import Path

Path("grain.svg").write_text(svg, encoding="utf-8")
Path("grain.html").write_text(fw.render_html(review), encoding="utf-8")

Continue exploring

fw.discover_dependencies, fw.infer_schema, fw.pairs. See results and configuration for serialization and result-method contracts.