Measurement details

This page records the calculation rules for Fieldwork 0.3.2. Start with Concepts for examples explaining what these measurements mean.

Budgets written name=default below are keys of each analysis’s limits mapping, such as fw.missingness(df, limits={"max_signatures": 20}). The exceptions are max_key_size, max_dimensions and n_paths, which are ordinary arguments. Results record the effective budgets in parameters["limits"].

Availability

Repeated patterns of present and missing values (availability signatures) are ranked by descending analysis-unit count with lexical signature ties. max_signatures=50 limits stored signatures, with omitted row mass reported. Identical masks form families even for always-missing or always-present columns.

Pairs are enumerated in input-column combination order, bounded by max_pairs=200. For A and B, presence Jaccard is both-present / either-present. If neither is ever present, it is undefined, never perfect similarity. Agreement additionally includes co-absence and is reported separately. A implies B has conditional presence both-present / A-present, an exception rate, and B’s baseline presence. No antecedent support means no implication finding. An implication is exact when every A-present unit has B present and approximate otherwise; similar presence is always approximate, since identical masks form families. Similarity and implication thresholds default to 0.8 and 0.9. Mutually exclusive pairs require each field to have observed support and no co-presence; exact families let users interpret exclusive field groups.

Context groups are joint combinations in first-observed order, bounded by max_contexts=32. Per-feature entity counts operate over all eligible distinct keys, without a display truncation affecting their denominator.

Dependencies and candidate grain

Enumerate determinants by size then input-column order, up to max_key_size=2 and max_candidates=100. Report the combinatorial candidate space and tested count. Every selected non-key feature is a possible target; completed tests use the corresponding eligible rows without sampling. max_dependency_tests may omit tests. User-specified contexts add up to max_contexts=32 conditional populations plus the global population.

Modal accuracy = 1 − minimum rows needing target-value repair / evaluated rows. Within each determinant group, choose its most frequent target; canonical code order breaks ties. Nonmodal rows are representative exceptions. Also retain all violating groups, affected rows, group violation rate, and repeated-group support. This separates approximate mapping quality from coverage and singleton effects. min_accuracy=0.95 controls finding emission, not which tests are computed. A finding’s repeated_support records whether its evaluated rows include a repeated determinant group; without one, exactness holds only because every group is one row.

For each determinant and global or conditional context, P is the scoped context population. Q contains complete determinant cases with dropna=True, otherwise all P. O contains rows of Q with an observed target, using native and declared sentinel missingness. E is the evaluated population: O when dropping missing, otherwise Q. Group E by the determinant; R contains rows in groups of at least two after all target exclusions.

Measurement	Meaning
determinant_evaluated_rows	Q: determinant-eligible rows
target_observed_rows	O: rows of Q with observed target
target_coverage	O/Q: observed target coverage
target_missing_excluded_rows	Q−E: exclusions for this target
evaluated_rows	E: rows used in the consistency test
missing_excluded_rows	P−E: all missing exclusions in the context
repeated_rows	R: rows in repeated determinant groups within E
repeat_coverage	R/E
repeat_modal_accuracy	1−repair_rows/R; every repair row belongs to a repeated group

A zero denominator gives None, serialized as JSON null. With dropna=False, missing categories participate in consistency, E=Q, and target exclusions are zero. Observed target coverage can still be below one; determinant eligibility does not then imply observed determinant values. Dependencies remain row-counted and row-weighted, without entity aggregation or resampling.

Candidates report determinant groups, uniqueness, repeated groups/rows and complete-case exclusions. determines lists ordered global exact targets, including singleton-only tests. determines_with_repeated_support includes only global exact targets with a repeated group in that target’s E. Conditional exact targets do not enter either list. global_targets_tested counts completed global tests, even empty ones; global_targets_possible counts selected non-key targets. An empty list under incomplete coverage means nothing was established by the completed tests. Candidate repeated rows describe Q, not a target’s E.

Full overview ordering keeps repeated groupings first, then unique identifiers, constants, and candidates without support. Within each role, ties use descending repeated-supported exact-target count, descending determinant repeated rows, key size, then lexical columns. Raw exact-target counts supply no additional tie-breaker; global coverage is disclosed rather than used as a score multiplier. One repeated group qualifies descriptively, not as a reliability certificate. Analytical candidate order and finding IDs remain unchanged. Topology orders canonically.

include_grain=False skips graph work; max_grain_views caps supported-population views. These controls are independent of max_dependency_tests; graph work does not inflate candidate test counters. Graphs retain their own compatible populations. Pair-specific exactness never transfers to a broader graph assignment; separate views are not composed. Graph and test support counts need not match.

A grain result holds two populations. Each key→target record in dependencies counts its own complete cases for that key and target, as a standalone test would, so it can count more rows than its view. The graph’s tests table, nodes and placements compare keys on the graph’s common rows (graph.evaluated_rows). Read placements from the graph and per-pair support from dependencies. A unique row ID remains a valid row-grain candidate, not proof of entity meaning.

Census paths

max_features=20 bounds the eligible input features (steering columns survive the budget), and max_pairs=200 bounds pair nesting inference. Constants are omitted unless required. Equivalent pairs are retained as aliases. If fine determines coarse but coarse does not determine fine, the soft nesting edge is coarse → fine.

A deterministic beam search extends ordered prefixes; it defaults to four dimensions, beam width 12, 200 evaluated extensions, three returned paths, and display budget 40. Input order defines feature coverage; lexical path order breaks score ties. If the search budget ends, shorter valid paths may be returned. Coverage records returned and requested depth; unsatisfied required columns can yield no recommendation.

For each path, count observed distinct prefixes at every depth. Base score is:

sum(prefix_counts) / display_budget
+ sum(max(0, prefix_count - display_budget)) / display_budget
+ 2 * redundant_steps
+ 3 * equivalent_pair_steps

A redundant step leaves the prefix count unchanged. Lower scores rank first. Structure and context objectives add eight per reversed supported nesting edge. Compact uses the base score. Target adds twelve times summed within-prefix modal impurity of the target values. Availability adds twelve times the same impurity for full boolean signatures. Impurity is rows outside each group’s mode / input rows. The target itself is excluded from browsing candidates unless explicitly required.

start_with fixes the leading dimensions. before enforces acyclic precedence and includes its referenced features; exclude removes candidates. Conflicting or unfittable steering raises ValueError. Context requires start_with; target requires target. These are observed-prefix heuristics, not an exhaustive optimizer or an order-invariant joint-information score.

Value patterns

String formats replace digit runs with 9 and ASCII letter runs with A; report three-character prefixes and lengths. max_patterns=10 bounds displayed counts; min_count omits formats, lengths and prefixes seen in fewer rows. Topology keeps the reported formats, alphabetically and without counts, and drops lengths and prefixes. Indexed-name families are explicitly name evidence, augmented by identical presence when observed. A column is numeric when every populated value is a non-boolean number, whatever its dtype, so an object column of numbers is summarized like its numeric equivalent. Numeric summaries use finite values, observed minimum spacing, and an allclose grid check. Offset and ratio checks require at least two finite paired rows; ratios exclude zero denominators. Tolerances are rtol 1e-5 and atol 1e-8. Pairs are tested in column-combination order up to max_pairs=100 (20 in the overview). They are simple measured relationships, not fitted latent models. Context constancy reports how many populated context groups have a single populated target value; the context columns themselves are not tested.

Reading evidence

A measurement is always tied to a scope, missing convention and counting unit. Support, exception rate, repeated-group support and baseline prevalence are different quantities. Fieldwork does not collapse them into a universal confidence score.

Source inspection uses ordered positions and a dataset fingerprint. A result can be rendered after saving without retaining the source dataframe; retrieving its original rows requires the same ordered source values, labels and column dtypes. Exact grain graphs express observed dependencies among evaluated candidates. Cross-cutting groupings and equivalent candidates can coexist.

Equal entity weights

unit="entities" gives every distinct populated entity key one vote. Presence uses any or all scoped rows as selected by entity_presence. Incomplete entity keys are excluded and counted. Context analysis aggregates an entity within each context, so an entity spanning contexts contributes once to each relevant context. Signatures and availability relations use this unit; selected examples and exception totals still count source rows. For 100 matching rows from one entity and one exception row from another, conditional presence is 100/101 with row weights and 1/2 with entity weights.

Sparse grains and distinct paths

Keys with different missing rows are evaluated in separate exact-grain views. Sparse keys can retain valid evidence in separate views without combining dependencies measured on different populations. Unsupported, constant, unique, and repeated-group candidates remain distinguishable.

At each search depth, paths retain the best order per selected feature set. Returned alternatives collapse alias substitutions; fewer than n_paths may be returned when there are fewer distinct evaluated feature sets. Explanations retain measured prefix counts, nesting, redundancy and per-depth nonmodal fractions.