Progress, cancellation, and work budgets

For a large table, first decide which questions you need answered. Restricting analytical features and choosing overview sections can avoid unnecessary work. Use progress=True to see which phase is running and how much work it has completed. The larger-table guide shows a runnable example.

Common runtime arguments

All public dataframe analyses accept keyword-only progress=None, cancel=None, timeout=None and safe_errors=None. These also apply to scope creation/refinement, source inspection/selection/recomputation, Recipe.run, and a recommended path’s census. Rendering reads saved evidence and does not perform dataframe analysis.

Argument	Meaning
progress=None or False	Silent default.
progress=True	One updating terminal or IPython notebook display; redirected output uses stderr lines.
progress=callback	Receive immutable ProgressEvent objects synchronously. Callback errors propagate; keep callbacks quick and do not mutate the source.
cancel=fw.CancellationToken()	Call the token’s cancel() method from another thread or a callback to request cancellation.
timeout=seconds	Finite, nonnegative elapsed seconds from call start. None disables the deadline.
safe_errors=True	Raise any other failure as fw.AnalysisError, without the original message. See below.

Cancellation raises fw.AnalysisCancelled without returning an apparently complete partial result. Checks occur between work items, so a running pandas or NumPy operation must return before cancellation takes effect. This is not a hard process deadline. Runtime objects are not serialized into results or recipes.

ProgressEvent exposes operation, phase, phase_id, parent_id, completed, nullable total, unit, elapsed_seconds, phase_elapsed_seconds, nullable detail, and status. Status is started, running, completed, cancelled, or failed. IDs distinguish nested phases within a call. Running updates are throttled; phase boundaries are always reported. fw.ProgressDisplay is the built-in callable display and can also be used directly.

The built-in display shows the current phase, a completion bar and percentage when the phase’s total is known, elapsed time and the current detail. There is no time estimate. Nested phases overlap, so completing fingerprinting or one section does not mean the whole overview is finished.

Keep source values out of error messages

Fieldwork’s own error messages name columns, arguments and value types, never cell values. Messages raised by pandas, NumPy or Python can still quote the cell that failed. When only shared output may leave your environment, pass safe_errors=True: any other failure is raised as fw.AnalysisError, whose message and attributes give only the operation, the innermost phase, the column when known and the original exception type.

try:
    overview = fw.explore(df, safe_errors=True)
except fw.AnalysisError as error:
    print(error.operation, error.phase, error.column, error.error_type)

This applies to invalid arguments too. The original exception stays available as error.__context__ for local debugging, but printed tracebacks omit it. Cancellation, keyboard interrupts and progress-callback errors propagate unchanged, and nested analyses follow the outermost call’s setting. Like the other runtime controls, safe_errors is never saved in results or recipes.

Bound the analytical work

explore accepts sections and per-section options. Section names are missingness, dependencies, paths, and value_patterns; omitted sections are explicitly not_requested. Per-section options accept the corresponding analytical function’s parameters. Keep source settings (scope, missing, table_id) and runtime controls on explore.

Search and output budgets go in each analysis’s limits mapping, typed by fieldwork.typing.PathLimits, MissingnessLimits, DependencyLimits, PatternLimits and PairLimits. Omitted budgets keep their defaults, unknown names raise TypeError, and results record the effective budgets in parameters["limits"]. In explore’s options, a section’s limits merge with the overview’s defaults key by key:

overview = fw.explore(
    df,
    sections=["missingness", "dependencies"],
    options={
        "dependencies": {
            "include_grain": False,
            "limits": {"max_candidates": 5, "max_dependency_tests": 100},
        },
    },
    progress=True,
)

For dependency discovery, the max_candidates limit bounds determinants and max_dependency_tests bounds candidate/target/context tests. include_grain=False skips exact graphs; the max_grain_views limit caps graph views. Test and graph budgets are independent. Zero tests still computes candidate summaries; graph computations never increment global_targets_tested. Inspect coverage for work not performed.

min_accuracy filters findings, while the dependency table retains every completed test. Display limits shorten output; they do not cap discovery work. Automatic explore defaults to 20 single-column dependency candidates; standalone dependency discovery defaults to 100 candidates of up to two columns.

What still costs time

Feature restrictions reduce analysis, but source identity validation still scans the entire ordered dataframe. Scopes also preserve that source-validation boundary. Physically projecting a dataframe reduces the source being validated and creates a different identity; keep the same projected frame for later inspection.

Caches are private to one call and are discarded on completion, cancellation, or failure; a later call revalidates the source. Within a call, sections share the source fingerprint and column codes, grain views reuse dependency counts on the same population, and path search reuses prefix groupings up to 32 MiB. These are not whole-operation memory limits; unique values, wide frames, many contexts and graph views can still be expensive. No sampling is introduced by these controls.

For saved result size, see saved exports.