fw.value_patterns

Use fw.value_patterns after you know where information is present. It summarizes string shapes and prefixes, finite numeric ranges and spacing, indexed column families, constant offsets or ratios, and constancy inside declared contexts.

This is especially useful for spotting export conventions: identifiers with a new shape, sensor channels that duplicate a scaled reading, or repeated slots whose names suggest a family.

Jump to the worked example or the argument reference.

Function reference

fw.value_patterns(
    df,
    *,
    features=None,
    by=None,
    limits=None,
    min_count=1,
    scope=None,
    missing=None,
    table_id='table',
    progress=None,
    cancel=None,
    timeout=None,
    safe_errors=None,
)

This page describes Fieldwork 0.3.2. Import the package with import fieldwork as fw. View the documented source.

Arguments

Argument	Type / default	Effect
df	pandas.DataFrame; required	The source table. Analysis preserves its values and row order; column labels are identified by their string form, which must be unique.
features	iterable of str or None; None	Columns to analyze, in candidate-search order; None selects all columns, skipping (and listing in skipped_features) columns with unsupported cells. Use a short, deliberate list when identifiers or free text would distract from the question.
by	iterable of str or None; None	Joint context columns. Each other feature is tested for being constant within each observed combination; the context columns themselves are not tested. This does not search all subsets of the context columns.
limits	mapping or None; None	Work and output budgets (fieldwork.typing.PatternLimits); omitted keys keep their defaults. max_pairs (100): selected feature pairs considered, in input order; only numeric pairs can yield offset/ratio findings, but nonnumeric pairs still consume this budget. max_patterns (10): string formats, lengths, and three-character prefixes retained per feature; format_count and omitted_format_rows disclose suppressed format evidence. example_limit (5): source positions saved per example or exception list; select and inspect(..., all_matches=True) still recover every matching row.
min_count	nonnegative int; 1	Small-cell threshold for string summaries. Formats, lengths and prefixes seen in fewer rows are omitted; omitted_format_rows counts the rows whose format is not reported. The default keeps every value up to max_patterns.
scope	Scope or None; None	A saved row selection tied to this dataframe; None uses every row. Construct with fw.Scope.from_positions(df, positions). Pass the original dataframe with its scope, rather than a reordered or changed copy.
missing	mapping or None; None	Column names mapped to lists of sentinel values, for example {"reading": [-999]}. Native nulls are always missing. Sentinels affect this analysis without replacing source values.
table_id	str; 'table'	Nonempty table identity recorded with the evidence. It does not bypass source-fingerprint checks.
progress	bool or callable; None	Silent by default; True displays progress, or pass a synchronous ProgressEvent callback. See runtime controls.
cancel	CancellationToken or None; None	Cooperative cancellation token; cancellation raises AnalysisCancelled without returning a partial result.
timeout	float or None; None	Finite nonnegative seconds from call start; None disables the cooperative deadline. A running pandas/NumPy work item must finish before cancellation.
safe_errors	bool or None; None	True raises any other failure as AnalysisError, naming the operation, phase and column without echoing the original message, which can quote cell values. See runtime controls.

Returns and output choices

Returns a fieldwork.Result with kind="value_patterns" (schema 2.0). summaries describes each feature; families records indexed names and supporting availability evidence; findings contains string, numeric, family, and context findings (a string finding’s structure lists its reported formats alphabetically, without counts, and formats_omitted); coverage records pair coverage; parameters records the settings, with the effective budgets under parameters["limits"]. Use to_frame("summaries") or inspect selected findings. recompute(df, limits={...}) reruns on the verified source; a budget override keeps the other saved budgets.

Use plaintext for a text report, SVG for a static figure, or HTML for a shareable report. to_dict() retains analytical evidence as plain JSON; visualization_data provides the labels and values used to draw a figure. List sections are available through to_frame(section).

Interpretation and boundaries

String formats collapse letter runs to A and digit runs to 9; these are observed shapes, not validation regexes. Topology exports keep the reported formats but drop lengths and three-character prefixes, which can disclose identifier fragments. A column is numeric when every populated value is a (non-boolean) number, whatever its dtype: an object or categorical column of numbers gets the same numeric findings as its numeric-dtype equivalent. Numeric spacing is the smallest gap between distinct finite observed values, not a guaranteed measurement resolution. Constant offsets and ratios use finite co-populated rows (rtol=1e-5, atol=1e-8); ratio tests also exclude zero denominators and require at least two eligible values.

Context constancy counts populated groups equally, rather than weighting by their row counts; rows missing a context key are omitted. Indexed names alone do not prove interchangeable semantics. Invalid budget values raise ValueError; unknown budget names raise TypeError; unknown columns raise KeyError.

Walkthrough: reading the conventions in a weather-station export

Two weather stations delivered eight records. Before combining them, check the identifier conventions, possible measurement conversions, and station unit labels. A familiar column name alone cannot answer these questions. Download the complete example.

Build a small export you can audit

The synthetic measurements include conversions, zeros, and a missing reading. Two indexed channels share one missing row. One identifier and one unit label use alternate spellings.

import pandas as pd
import fieldwork as fw

# Synthetic records: two stations, four exports each; one mixed unit label.
df = pd.DataFrame(
    {
        "sample_id": [
            "WX-001",
            "WX-002",
            "WX-003",
            "WX-004",
            "WX-005",
            "WX-006",
            "WX-007",
            "WX/008",
        ],
        "station": ["north"] * 4 + ["south"] * 4,
        "unit": ["C", "C", "C", "C", "C", "C", "degC", "C"],
        "temperature_c": [0, 5, 10, 15, 20, 25, None, 30],
        "rain_mm": [0, 10, 20, 30, 0, 10, None, 20],
        "channel_1": [1, 2, 4, None, 3, 5, 6, 8],
        "channel_2": [8, 7, 6, None, 4, 3, 2, 1],
    }
)
df["temperature_k"] = df["temperature_c"] + 273.15
df["rain_cm"] = df["rain_mm"] / 10

features = [
    "sample_id",
    "unit",
    "temperature_c",
    "temperature_k",
    "rain_mm",
    "rain_cm",
    "channel_1",
    "channel_2",
]
patterns = fw.value_patterns(
    df,
    features=features,
    by=["station"],
    limits={"max_patterns": 1, "example_limit": 2},
)
print(fw.render_plaintext(patterns, max_lines=10))

Fieldwork · value_patterns
Population: 8 rows
Search coverage (untested work is unknown)
  value_patterns: 28/28 pairs evaluated
[f0] sample_id: string formats, lengths and prefixes
  Analysis: rows; presence=per_row
  Formats: A-9, ...
  Unit: rows; feature=sample_id, populated=8, missing=0, format_count=2, omitted_format_rows=1
  Examples: 2/8 saved source positions [0, 1]
... more output not rendered (max_lines)

This short preview leaves the other findings available in the result.

Separate a common format from a universal rule

Use the summaries table for column-level measurements and dictionary access for format counts. The max_patterns budget of 1 retains the most common format; it does not filter out the record with the other format.

summaries = patterns.to_frame("summaries").set_index("feature")
for feature in ["sample_id", "temperature_c"]:
    row = summaries.loc[feature]
    print(
        feature,
        "populated=",
        row["populated"],
        "missing=",
        row["missing"],
    )
ids = patterns["summaries"][0]
print("identifier formats:", ids["formats"])
print(
    "distinct formats:",
    ids["format_count"],
    "omitted format rows:",
    ids["omitted_format_rows"],
)
print(
    "temperature grid:",
    summaries.loc["temperature_c", "observed_step"],
    summaries.loc["temperature_c", "on_observed_step_grid"],
)

sample_id populated= 8 missing= 0
temperature_c populated= 7 missing= 1
identifier formats: [['A-9', 7]]
distinct formats: 2 omitted format rows: 1
temperature grid: 5.0 True

Seven identifiers have the letter-run, hyphen, digit-run shape A-9. That is 7 of 8 populated identifiers, not evidence that every identifier follows it. The seven populated temperatures lie on a five-degree grid in this tiny export; this describes observed spacing, not the sensor’s measurement resolution.

Check conversion candidates against their actual rows

Select findings by kind and ordered columns. Offsets mean second column minus first; ratios mean second divided by first.

findings = patterns.to_frame("findings").set_index("id")
print(findings[["pattern", "statement"]].to_string())

               pattern                                                      statement
id
f0     string_patterns                sample_id: string formats, lengths and prefixes
f1     string_patterns                     unit: string formats, lengths and prefixes
f2       numeric_range              temperature_c: numeric range and observed spacing
f3       numeric_range              temperature_k: numeric range and observed spacing
f4       numeric_range                    rain_mm: numeric range and observed spacing
f5       numeric_range                    rain_cm: numeric range and observed spacing
f6       numeric_range                  channel_1: numeric range and observed spacing
f7       numeric_range                  channel_2: numeric range and observed spacing
f8      indexed_family                           Indexed family: channel_1, channel_2
f9      numeric_offset  temperature_k has a constant offset relative to temperature_c
f10      numeric_ratio               rain_cm has a constant ratio relative to rain_mm
f11  context_constancy                            sample_id: constancy within station
f12  context_constancy                                 unit: constancy within station
f13  context_constancy                        temperature_c: constancy within station
f14  context_constancy                        temperature_k: constancy within station
f15  context_constancy                              rain_mm: constancy within station
f16  context_constancy                              rain_cm: constancy within station
f17  context_constancy                            channel_1: constancy within station
f18  context_constancy                            channel_2: constancy within station

The temperature offset and rainfall ratio are f9 and f10 in this result. Read their measurements, then inspect the rows supporting the rainfall ratio:

conversions = findings.loc[["f9", "f10"]]
columns = [
    "pattern",
    "measurements.value",
    "measurements.evaluated_rows",
    "measurements.excluded_rows",
]
print(conversions[columns].to_string())

rain_rows = patterns.inspect(df, "f10", all_matches=True)
print(
    rain_rows[["sample_id", "rain_mm", "rain_cm"]].to_string(
        index=False
    )
)

            pattern  measurements.value  measurements.evaluated_rows  measurements.excluded_rows
id
f9   numeric_offset              273.15                          7.0                         1.0
f10   numeric_ratio                0.10                          5.0                         3.0
sample_id  rain_mm  rain_cm
   WX-002     10.0      1.0
   WX-003     20.0      2.0
   WX-004     30.0      3.0
   WX-006     10.0      1.0
   WX/008     20.0      2.0

The temperature offset uses seven paired finite readings. The rainfall ratio uses only five: the missing pair and the two zero denominators are excluded from its eight-row population. Zero rainfall remains a valid observation; it just cannot support this division. all_matches=True retrieves all five eligible rows despite the two-example display limit. Here the default index labels equal source positions.

These conversions are known because we constructed them. In an unfamiliar export, check such leads against unit definitions before concluding that columns measure the same thing. Numeric constancy uses a floating-point tolerance.

Distinguish channel naming from station-level consistency

print(patterns.to_frame("families").to_string(index=False))

# f12 is the station-to-unit constancy finding in the table above.
unit_rows = patterns.inspect(
    df, "f12", exceptions=True, all_matches=True
)
print(
    unit_rows[["sample_id", "station", "unit"]].to_string(index=False)
)

              features  prefix                               evidence
[channel_1, channel_2] channel [indexed_name, identical_availability]
sample_id station unit
   WX-005   south    C
   WX-006   south    C
   WX-007   south degC
   WX/008   south    C

The channel family combines two pieces of evidence: indexed names and identical availability. Their values differ, so combining them would need a separate justification. For units, one of two station groups is constant. The denominator is stations, not rows. Exception inspection returns the whole nonconstant south group, not just WX-007: deciding whether C and degC are equivalent requires an explicit label convention outside this calculation.

Widen the summary, then deliberately constrain the search

expanded = patterns.recompute(df, limits={"max_patterns": 2})
print("Identifier formats:", expanded["summaries"][0]["formats"])

usual_format = df["sample_id"].str.fullmatch(r"WX-\d{3}")
print(df.loc[~usual_format, ["sample_id"]].to_string(index=False))

bounded = patterns.recompute(df, limits={"max_pairs": 1})
print("Full pair coverage:", patterns["coverage"])
print("Limited pair coverage:", bounded["coverage"])

Identifier formats: [['A-9', 7], ['A/9', 1]]
sample_id
   WX/008
Full pair coverage: {'pair_candidates': 28, 'pairs_evaluated': 28}
Limited pair coverage: {'pair_candidates': 28, 'pairs_evaluated': 1}

The identifier check locates the slash-form record. The bounded pair run visits only sample_id and unit, the first combination in our feature order. Its zero numeric findings say nothing about the unvisited conversions. The next investigation is concrete: confirm the slash identifier and unit alias with the exporter, and inspect the channel definitions before proposing any normalization. Preserve the original columns while those meanings are unresolved.

Continue exploring

fw.missingness, fw.discover_dependencies, fw.levels. See results and configuration for serialization and result-method contracts.