fw.discover_dependencies

Use fw.discover_dependencies when you do not yet know which fields determine other fields. It searches candidate determinants, measures uniqueness separately from mapping accuracy, and exposes the exceptions to approximate mappings.

Use grain instead when you already have a short, explicit set of candidate keys and want exact functional-dependency evidence.

Jump to the worked example or the argument reference.

Function reference

fw.discover_dependencies(
    df,
    *,
    features=None,
    max_key_size=2,
    min_accuracy=0.95,
    by=None,
    dropna=True,
    include_grain=True,
    limits=None,
    scope=None,
    missing=None,
    table_id='table',
    progress=None,
    cancel=None,
    timeout=None,
    safe_errors=None,
)

This page describes Fieldwork 0.3.2. Import the package with import fieldwork as fw. View the documented source.

Arguments

Argument	Type / default	Effect
df	pandas.DataFrame; required	The source table. Analysis preserves its values and row order; column labels are identified by their string form, which must be unique.
features	iterable of str or None; None	Columns to analyze, in candidate-search order; None selects all columns, skipping (and listing in skipped_features) columns with unsupported cells. Use a short, deliberate list when identifiers or free text would distract from the question.
max_key_size	positive int; 2	Largest determinant width to search. Candidates are enumerated by size, then selected feature order; larger combinations increase the search space rapidly.
min_accuracy	float in [0, 1]; 0.95	Minimum modal-repair accuracy for emitting a dependency finding (tested records remain in dependencies): sum of each determinant group’s most frequent target count, divided by eligible rows. Exceptions are rows outside the chosen modal target.
by	iterable of str or None; None	Joint context columns. Each observed combination defines a context; this does not search all subsets of the context columns.
dropna	bool; True	True excludes rows missing any determinant component or the target for each dependency test. Candidate uniqueness excludes missing keys only. False treats native missing and declared sentinels as one category.
include_grain	bool; True	Build exact foundation grain views. False retains dependency tests and candidate summaries while skipping graphs.
limits	mapping or None; None	Work and output budgets (fieldwork.typing.DependencyLimits); omitted keys keep their defaults. max_candidates (100): determinant combinations evaluated, each tested against selected non-key targets globally and in retained contexts; it does not limit reported dependencies or input rows. max_contexts (32): observed joint by contexts evaluated in addition to global tests; zero keeps global tests only. max_dependency_tests (None, unbounded): candidate/target/context tests, including empty tests; it does not cap graph work. max_grain_views (None, unbounded): supported-population graph views. example_limit (5): source positions (and exception groups) saved per test; select and inspect(..., all_matches=True) still recover every matching row. Omitted work is reported in coverage and graph_selection.
scope	Scope or None; None	A saved row selection tied to this dataframe; None uses every row. Construct with fw.Scope.from_positions(df, positions). Pass the original dataframe with its scope, rather than a reordered or changed copy.
missing	mapping or None; None	Column names mapped to lists of sentinel values, for example {"reading": [-999]}. Native nulls are always missing. Sentinels affect this analysis without replacing source values.
table_id	str; 'table'	Nonempty table identity recorded with the evidence. It does not bypass source-fingerprint checks.
progress	bool or callable; None	Silent by default; True displays progress, or pass a synchronous ProgressEvent callback. See runtime controls.
cancel	CancellationToken or None; None	Cooperative cancellation token; cancellation raises AnalysisCancelled without returning a partial result.
timeout	float or None; None	Finite nonnegative seconds from call start; None disables the cooperative deadline. A running pandas/NumPy work item must finish before cancellation.
safe_errors	bool or None; None	True raises any other failure as AnalysisError, naming the operation, phase and column without echoing the original message, which can quote cell values. See runtime controls.

Returns and output choices

Returns a fieldwork.Result with kind="dependencies" (schema 2.0). candidates describes uniqueness and group sizes; dependencies describes all tested global and contextual mappings, including those below the finding threshold; findings provides selectable evidence; coverage records the search performed. grain_views contains exact-grain evidence for evaluated candidates, each with its recorded population, and graph_selection explains which candidates were drawn. parameters records the settings, with the effective budgets under parameters["limits"]. Context values are plain JSON, such as row["context"]["warehouse"] == "West". A finding’s measurements omit the test’s exception groups, which stay in dependencies. Its structure records strength ("exact" or "approximate", matching its pattern) and repeated_support: whether any determinant group has two or more evaluated rows.

Use plaintext for a text report, SVG for a static figure, or HTML for a shareable report. to_dict() retains analytical evidence as plain JSON; visualization_data provides the labels and values used to draw a figure. List sections are available through to_frame(section).

Interpretation and boundaries

A determinant may repeat many times and still map exactly to an attribute; a unique identifier trivially determines every populated target in the observed sample. An exact finding with repeated_support false holds only because every determinant group is one row; renderers note this, including in topology. Approximate findings are stated as “approximately determines”. Topology exports list each candidate’s structural role (unique identifier, repeated grouping, constant, or no evaluated support) in search order, without its counts. Neither observation establishes a business rule beyond these data. Approximate accuracy weights rows, so large groups can dominate.

A missing finding may reflect the threshold, a search limit, or an ineligible population. Check coverage before drawing a negative conclusion. Invalid budget values or accuracy raise ValueError; unknown budget names raise TypeError; unknown columns raise KeyError.

Walkthrough: is a case size a product rule or a warehouse rule?

A warehouse team wants to validate inbound receipts against a packaging lookup. Does an SKU have one case size, or does the warehouse belong in the lookup key?

These 18 synthetic receipts contain three deliveries for each SKU at each of two warehouses. SKU A intentionally has different case sizes at East and West. One West receipt for SKU C records 20 units where the other two record 24. That last value is an investigation lead; a frequency calculation cannot tell us whether it is a typo, a special order, or a legitimate packaging change.

Download the runnable example to follow the same steps.

Build a population with repeated evidence

import pandas as pd
import fieldwork as fw

pack_sizes = [
    ("East", "A", "Tea", 6),
    ("East", "B", "Coffee", 12),
    ("East", "C", "Cocoa", 24),
    ("West", "A", "Tea", 8),
    ("West", "B", "Coffee", 12),
    ("West", "C", "Cocoa", 24),
]
rows = []
for warehouse, sku, product, units in pack_sizes:
    for delivery in range(1, 4):
        receipt = f"{warehouse[0]}-{sku}-{delivery}"
        rows.append((receipt, warehouse, sku, product, units))

receipts = pd.DataFrame(
    rows,
    columns=[
        "receipt",
        "warehouse",
        "sku",
        "product",
        "units_per_case",
    ],
)
# Introduce one receipt that disagrees with its SKU's usual case size.
receipts.loc[receipts["receipt"] == "W-C-3", "units_per_case"] = 20
print(
    receipts.groupby(["warehouse", "sku", "units_per_case"])
    .size()
    .to_string()
)

warehouse  sku  units_per_case
East       A    6                 3
           B    12                3
           C    24                3
West       A    8                 3
           B    12                3
           C    20                1
                24                2

Keep receipt for tracing records, but leave it out of the discovery features. A unique receipt identifier would determine every recorded attribute simply because no receipt repeats. Repeated SKUs let us ask the stronger operational question: do different receipts agree about the same product?

Separate a mapping from a unique key

Start with all four single-column determinants, and also evaluate each warehouse as a context. by adds contextual tests alongside the overall population.

features = ["sku", "warehouse", "units_per_case", "product"]
singles = fw.discover_dependencies(
    receipts,
    features=features,
    max_key_size=1,
    by=["warehouse"],
    min_accuracy=0.9,
    limits={"max_candidates": 4, "example_limit": 1},
)
candidates = singles.to_frame("candidates")
print(
    candidates[
        ["columns", "groups", "unique", "repeated_groups", "determines"]
    ].to_string(index=False)
)

         columns  groups  unique  repeated_groups     determines
           [sku]       3   False                3      [product]
     [warehouse]       2   False                2             []
[units_per_case]       5   False                4 [sku, product]
       [product]       3   False                3          [sku]

Each SKU appears six times, so none is a unique row key. Nevertheless, every receipt with that SKU has the same product name. determines reports that exact overall mapping. It does not mean the candidate uniquely identifies receipts, or list the approximate or contextual relationships.

Find where the global rule breaks

Read the tested dependency records directly, including tests that did not meet the finding threshold:

records = [
    row
    for row in singles["dependencies"]
    if row["determinant"] == ["sku"]
    and row["target"] == "units_per_case"
]
for row in records:
    context = row["context"]
    label = "All" if context is None else context["warehouse"]
    print(
        f"{label}: {row['evaluated_rows']} rows; {row['repair_rows']} repairs; "
        f"accuracy={row['modal_accuracy']:.3f}; exact={row['exact']}"
    )

All: 18 rows; 4 repairs; accuracy=0.778; exact=False
East: 9 rows; 0 repairs; accuracy=1.000; exact=True
West: 9 rows; 1 repairs; accuracy=0.889; exact=False

Modal accuracy asks how many eligible rows already agree with the most frequent target value within their determinant group. Globally, SKU A splits evenly between 6 and 8 units: any single modal choice leaves three disagreements. SKU C adds one more, giving (18 - 4) / 18 = 0.778. This is a row-weighted measure, not the fraction of SKUs without a conflict.

East agrees perfectly once warehouse is held fixed. West retains one discrepancy among nine receipts. At min_accuracy=0.9, the West relationship does not become a finding, but its tested evidence remains in dependencies. Absence from the findings list alone would miss this distinction.

Put the warehouse in the lookup key

Now search pairs as well as single columns. With four features there are four singletons and six pairs; a budget of ten evaluates the complete requested candidate space. No context partition is needed for this run.

composite = fw.discover_dependencies(
    receipts,
    features=features,
    max_key_size=2,
    min_accuracy=0.9,
    limits={"max_candidates": 10, "example_limit": 1},
)
dependency_table = composite.to_frame("dependencies")
case_sizes = dependency_table[
    dependency_table["target"] == "units_per_case"
]
print(
    case_sizes[
        [
            "determinant",
            "evaluated_rows",
            "evaluated_groups",
            "violating_groups",
            "affected_rows",
            "repair_rows",
            "modal_accuracy",
        ]
    ].to_string(index=False)
)

         determinant  evaluated_rows  evaluated_groups  violating_groups  affected_rows  repair_rows  modal_accuracy
               [sku]              18                 3                 2             12            4        0.777778
         [warehouse]              18                 2                 2             18           12        0.333333
           [product]              18                 3                 2             12            4        0.777778
    [sku, warehouse]              18                 6                 1              3            1        0.944444
      [sku, product]              18                 3                 2             12            4        0.777778
[warehouse, product]              18                 6                 1              3            1        0.944444

The proposed lookup now respects the intentional East/West difference for SKU A. Five of the six SKU–warehouse groups agree internally. The sixth contains three affected rows, of which one is a repair row: changing that one target to the group’s modal value would make the observed dependency exact. Hence the accuracy is 17 / 18 = 0.944, not 5 / 6. There are no missing values here, so all 18 receipts enter the global denominator.

Recover the exception before deciding what to change

findings = composite.to_frame("findings")
print(findings[["id", "statement"]].to_string(index=False))

 id                                                  statement
 f0                                     sku determines product
 f1                              units_per_case determines sku
 f2                          units_per_case determines product
 f3                                     product determines sku
 f4     sku, warehouse approximately determines units_per_case
 f5                          sku, warehouse determines product
 f6                     sku, units_per_case determines product
 f7                   warehouse, units_per_case determines sku
 f8               warehouse, units_per_case determines product
 f9                          warehouse, product determines sku
f10 warehouse, product approximately determines units_per_case
f11                     units_per_case, product determines sku

In this result, f4 describes the mapping from SKU and warehouse to units per case. Retrieve all rows that disagree with their group’s most common case size:

exceptions = composite.inspect(
    receipts, "f4", exceptions=True, all_matches=True
)
print(
    exceptions[
        ["receipt", "sku", "warehouse", "units_per_case"]
    ].to_string(index=False)
)

receipt sku warehouse  units_per_case
  W-C-3   C      West              20

all_matches=True recovers the complete exception population from the original source; the example_limit budget of 1 bounds saved examples, not the count of exceptions. Keep the original ordered frame for inspection. Check receipt W-C-3 against its purchase order and packaging specification before editing anything. If 20 is legitimate, a packaging revision or supplier may be the missing determinant.

This final call renders the exact product mapping. Text output displays Unicode by default; pass unicode_mode="safe" instead when you need ASCII-only output.

product_rule = fw.discover_dependencies(
    receipts,
    features=["sku", "product"],
    max_key_size=1,
    limits={"max_candidates": 1},
)
print(
    fw.render_plaintext(
        product_rule, detail="topology", unicode_mode="display"
    )
)

Fieldwork · dependencies
Topology only · quantitative evidence suppressed
Candidate grains
  sku: repeated grouping
sku determines product
  Analysis: rows; presence=per_row

This deliberately narrow rendering evaluates only the first of two possible single-column candidates. It says nothing about the untested reverse direction. Use the complete search’s coverage and numerical records to assess the packaging rule; a topology view intentionally omits those quantities.

Check what remains after missing values are excluded

Suppose four receipts contain two product codes, but only one receipt per code has a recorded product name. An exact mapping can then rest entirely on single observations:

sparse_receipts = pd.DataFrame(
    {"sku": ["A", "A", "B", "B"], "product": ["Tea", None, "Coffee", None]}
)
sparse = fw.discover_dependencies(
    sparse_receipts, max_key_size=1, limits={"max_candidates": 1}
)
sparse_evidence = sparse.to_frame("dependencies")
print(
    sparse_evidence[
        ["evaluated_rows", "target_coverage", "repeated_rows", "repeat_modal_accuracy"]
    ].to_string(index=False)
)

 evaluated_rows  target_coverage  repeated_rows repeat_modal_accuracy
              2              0.5              0                  None

SKU still determines the observed product names on the two evaluated rows. Yet only half of the four determinant-eligible rows have a product name, and neither remaining SKU group repeats. Repeat-only consistency is therefore unavailable. It would be misleading to infer agreement across repeated receipts from this exactness alone.

Candidate determines retains this exact target, while determines_with_repeated_support excludes it. Full summaries show both counts and global_targets_tested / global_targets_possible. These lists describe global tests; conditional exactness stays attached to its context. Candidate repeated_rows describes the determinant before target exclusions and must not replace the dependency’s count.

Singletons can also hide a conflict in a high overall accuracy:

mostly_unique = pd.DataFrame(
    {"sku": list(range(99)) + [98], "product": ["Tea"] * 99 + ["Coffee"]}
)
consistency = fw.discover_dependencies(
    mostly_unique, max_key_size=1, limits={"max_candidates": 1}
)
print(
    consistency.to_frame("dependencies")[
        ["modal_accuracy", "repeat_coverage", "repeat_modal_accuracy"]
    ].to_string(index=False)
)

 modal_accuracy  repeat_coverage  repeat_modal_accuracy
           0.99             0.02                    0.5

There are 98 singleton groups and one pair. Only one of the 100 rows needs repair, but that is one of the pair’s two rows. Overall accuracy is 99%; consistency among repeated observations is 50%. Both describe the observed rows; neither establishes that an SKU is the right business entity.

When dropna=False, missing values participate as a category. Observed target coverage can still be below one even though all determinant-eligible rows were evaluated. Undefined fractions stay None (JSON null). See the population formulas for scoped, composite and conditional cases.

View the full sparse-target report.

Continue exploring

fw.grain, fw.missingness, fw.suggest_paths. See results and configuration for serialization and result-method contracts.