Measurements and populations

Before interpreting a result, ask two questions: which records are included, and what counts as one observation? Six image records may describe only three exams. A result about records and a result about exams answer different questions.

Availability

Availability describes whether a cell contains a value. A row’s availability signature lists which fields are filled in and which are missing. Across the table, columns filled in on exactly the same rows form an availability family.

For example, a form might collect street name and house number together. Shared availability suggests a collection pattern; it does not mean the two columns contain equivalent information.

Native missing values are recognized automatically. If a logger writes -999 when a reading is unavailable, declare that sentinel with missing={"reading": [-999]}. Otherwise it counts as a value. Declaring a sentinel does not change the dataframe.

Similarity and directional presence

Two common questions use different denominators:

Question	Measurement
How often are A and B filled in together?	Presence Jaccard: rows with both, divided by rows with either.
When A is filled in, how often is B also filled in?	Conditional presence: rows with both, divided by rows with A.

Suppose A is present in ten rows, B in eleven, and both in nine. Conditional presence from A to B is 9/10. The reverse is 9/11. Read B’s overall availability too: a frequently populated column may have high conditional presence simply because it is usually present.

Shared absences do not increase presence Jaccard. If its denominator is zero, the rate is undefined (None). A directional finding also needs at least one observation with its starting field present.

Rows and entities

By default, each row contributes one observation. Use unit="entities" with an entity key when each exam, person, or station should have equal weight.

Imagine one station has 100 visits with both measurements and another has one visit with A but no B. Row-weighted conditional presence is 100/101. Giving each station one vote makes it 1/2.

For entity availability, choose what “present” means:

Setting	An entity has the field when…
entity_presence="any"	At least one of its selected rows has a value.
entity_presence="all"	Every selected row has a value.

Presence is assessed separately for each field. A station may have A on one visit and B on another; having both at the station level does not imply a visit with both. The marsh monitoring example shows this distinction.

Missing entity keys are excluded and counted. If an entity has rows in several contexts, it contributes once within each relevant context. Inspection still returns source rows, even when a finding counts entities.

Supplying entity without changing the default unit="rows" adds entity summaries while the main availability calculation continues to count rows. The summary categories any/all/one/some/none can overlap: a single populated row is both “one” and “all.”

Groups, keys, and grain

The grain of a table describes what its records represent. A library export may contain one row per physical copy, while repeating facts about books and editions on those rows.

A candidate key is a column or combination of columns you want to evaluate as a grouping. A book identifier might group several copies. If every copy of a book has the same title, the book identifier determines title in this dataset. This is an exact dependency.

A unique row ID determines every other column automatically: each group has only one row. Agreement across repeated groups is more useful evidence when deciding which facts belong together. Fieldwork reports unique and repeated groups separately.

Some identifiers are local. If edition 1 occurs for many books, identify an edition using both book and edition number. KeySpec names such a composite key; see the book catalog example.

When records disagree

An approximate dependency measures agreement with the most common target value within each group. If 19 of 20 rows agree with their group’s most common value, modal accuracy is 95%.

Distinguish two counts. If one record disagrees in a three-row group, one row would need changing to make that group consistent, but all three rows belong to a conflicting group. Neither count establishes which record is wrong. Review the source before making corrections.

Missing keys can mean two candidate groupings use different rows. Fieldwork keeps separate grain views when their populations differ, rather than combining incompatible evidence. Exact mappings describe the observed data; they are not guarantees about future deliveries.

Exactness, repeated consistency, and entity meaning

Two repeated product codes can each have only one observed product name. Dropping the other two missing names leaves an exact mapping on two singleton groups, even though no repeated observations remain to compare. Observed target coverage is 2/4; repeat-only consistency cannot be assessed.

With 98 singleton groups and one conflicting two-row group, overall modal accuracy is 99%, while repeat-only accuracy is 50%. Read both the available target fraction and the support within repeated groups before interpreting a mapping. The dependency walkthrough shows these examples.

Full candidate summaries show exact-target counts with and without repeated support, plus completed/possible global tests. Within the same role, ordering uses the repeated-supported count. It remains sensitive to search budgets and feature order. Neither exactness nor repeated consistency establishes a meaningful entity or guarantees future data; a unique identifier remains a useful row-grain lead.

Census paths

A census groups rows in an order you choose. With site → exam, the first level counts sites and the second counts exams within those sites. Each step is a prefix of the full path.

Changing the order changes the intermediate groups. Starting with a unique record ID creates one branch per row immediately, which is useful for lookup but cumbersome for browsing. Suggested paths favor useful intermediate groups for the chosen objective.

The final counts for the same set of columns do not depend on their order. Recommendations use a limited search, so “best” means the best returned by that search. It is not a probability or proof of the best possible arrangement. See path scoring details for formulas and tie-breaking rules.

Selection, search, and display limits

These limits answer different questions:

Kind of limit	What it changes	Example
Row selection	Which records contribute to the analysis.	A scope containing one site.
Search limit	Which possible relationships are examined.	limits={"max_pairs": 50} limits the column pairs tested.
Saved-result limit	Which details are retained in the result.	limits={"example_limit": 3} saves a few representative rows.
Rendering limit	How much of the saved result is shown.	max_nodes shortens a report.

Analyses take their search and saved-result limits in one limits mapping; limits you leave out keep their defaults. An untested relationship has not failed a test. An omitted branch may still contain rows. A renderer can show more saved detail when you increase its limits, but it cannot recover details the analysis never retained.

For census top_n_mode="post", hidden branches still contribute to the original counts and denominators. With top_n_mode="pre", selecting the leading branches changes the population before counting. The library loans example compares both choices.

Values and formats

value_patterns describes populated cells. It can find common string formats, indexed column names, numerical spacing, and constant offsets or ratios. A column is summarized as numeric when every populated value is a number, whatever the column’s dtype, so numbers stored in an object column are treated like a numeric column. An observed five-degree spacing describes this dataset; it does not establish a sensor’s resolution. Similar column names alone do not establish equivalent values.

The sensor example shows how to check such findings against their source rows. Detailed thresholds and tolerances are in Measurement details.

Save results and return to records

A result records its row selection, missing-value rules, and counting unit. Save result.to_dict() to preserve that analysis. Renderers can read saved results without the original dataframe.

Retrieving original records requires the same ordered source values and labels. A scope is a saved selection tied to that source; it cannot be applied to an unrelated delivery. If you change the dataframe, rerun the analysis. Use a recipe to repeat the same operation on new data.

See Save and compare for the workflow and Results and configuration for the supported methods.