Data Management

Overview

Framework manages data through a catalog. Each data file gets a dot-notation name such as inputs.raw.survey, and the catalog records the file's path, type, and integrity settings. Code reads data by name, so the path appears in one place. Framework picks the reader from the file type and checks the file against the hash it recorded earlier.

import framework as fw

fw.scaffold()
survey = fw.data_read("inputs.raw.survey")

Data comes back as a pandas DataFrame.

The Catalog

The catalog is the data section of the settings, stored in settings/data.yml in standard and privacy-sensitive projects. Each part of a name is one level of nesting.

data:
  inputs:
    raw:
      survey:
        path: inputs/raw/survey.csv
        type: csv
        delimiter: comma
        locked: true
        description: Baseline survey, wave 1
      sites: inputs/raw/sites.csv
    intermediate:
      survey_clean:
        path: inputs/intermediate/survey_clean.parquet
        type: parquet
        locked: true

An entry is either a path or a mapping with these fields.

Field	Meaning
path	The file, relative to the project root. Required.
type	The file type. Without it, the type comes from the extension.
delimiter	For delimited text: comma, tab, semicolon, or space.
locked	When true, a changed file raises an error on read.
description	Printed when the data is read.

Other fields are kept and returned by data_info() in its extra attribute.

A key may contain dots. inputs.raw.survey is found as nested YAML, as a raw.survey: key under inputs:, or as an inputs.raw.survey: key directly under data:. Nested YAML is the form Framework writes.

You can write entries by hand, but data_add() and data_save() write them for you.

Adding Existing Files

data_add() registers a file that is already in the project. It writes the catalog entry, records the file's hash in framework.db, and returns the entry.

fw.data_add("inputs/raw/survey.csv")

Without a name, a file in one of the inputs_raw, inputs_intermediate, inputs_final, or inputs_reference folders is named after that folder, so this call registers inputs.raw.survey. Elsewhere the name is the file name, and Framework suggests choosing a more descriptive one.

fw.data_add("inputs/raw/sites_2024.csv", name="inputs.raw.sites")

Entries added this way are locked by default. Pass locked=False for files that are expected to change. The type and delimiter arguments override what the extension suggests. With update_config=False, only framework.db is updated.

Reading Data

data_read() takes a catalog name or a file path.

survey = fw.data_read("inputs.raw.survey")
lookup = fw.data_read("inputs/reference/codes.csv")

A catalog name is resolved from the project root. A path, which is anything containing a slash, is resolved from the current folder. Only files read by catalog name are checked against framework.db.

The delim argument overrides the delimiter. Other keyword arguments go to the pandas or pyreadstat reader.

fw.data_read("inputs.raw.survey", usecols=["id", "score"])

For Stata and SPSS files, keep_attributes=True returns value labels as categories. By default the stored codes are returned, as in R.

Saving Data

data_save() writes a DataFrame, records its hash, and returns the DataFrame, so a call can end a pipeline.

clean = survey.dropna()
fw.data_save(clean, "inputs.intermediate.survey_clean")

When the target is a catalog name, its leading parts select a folder from directories and the remaining parts become the file name. The longest run of leading parts that names a directory key wins. inputs.intermediate.survey_clean uses the inputs_intermediate folder and writes inputs/intermediate/survey_clean.parquet, and inputs.private.intermediate.patients_clean uses inputs_private_intermediate in a privacy-sensitive project. A name is also added to the catalog, so data_read() finds it under the same name. A name needs at least three parts.

Names are saved as Parquet by default. Pass type="csv" for CSV, with delimiter to choose the separator.

fw.data_save(clean, "inputs.final.analysis", type="csv")

When the target is a path ending in .csv or .parquet, the file is written there and its hash is recorded under that path. A path does not create a catalog entry. Register it afterward with data_add() to read it by name.

fw.data_save(clean, "exports/patients_clean.parquet")
fw.data_add("exports/patients_clean.parquet", name="exports.patients_clean")

The R package's data_save() selects the folder by the first two parts only, so in R a name cannot reach a three-part key such as inputs_private_intermediate.

Saved files are locked by default. The target folder must exist unless you pass force=True.

Supported Formats

framework-py reads these formats. It writes CSV and Parquet, which both languages read.

Type	Extensions	Reader	Extra
csv	.csv, .tsv	pandas.read_csv()
tsv	.tsv, .txt, .dat	pandas.read_csv()
parquet	.parquet	pandas.read_parquet()
excel	.xlsx, .xls	pandas.read_excel()	excel for .xlsx
stata	.dta	pandas.read_stata()
spss	.sav, .zsav	pyreadstat.read_sav()	spss
spss_por	.por	pyreadstat.read_por()	spss
sas	.sas7bdat	pandas.read_sas()
sas_xpt	.xpt	pandas.read_sas(), or pyreadstat for version 8 files	sas for version 8

A catalog entry with an extension Framework does not recognize is read as comma-separated text, with a warning, unless the entry or the call names a delimiter.

.rds files are R-only. Reading one raises UnsupportedFormatError, with a message that suggests saving the data from R as Parquet or CSV. The catalog records each file's type, so either language reads the CSV and Parquet entries the other wrote.

Data Integrity

Framework hashes each catalog file with SHA-256 and stores the hash in framework.db.

1. The first read of a name records the file's hash. data_add() and data_save() record it when they register the file.
2. Each later read compares the file with the recorded hash.
3. When an unlocked file has changed, Framework issues a DataChangedWarning, records the new hash, and reads the file.
4. When a locked file has changed, Framework raises IntegrityError and does not read it.

Locking protects source data that should never change, such as the raw files a study received.

IntegrityError: Hash mismatch for locked data 'inputs.raw.survey'
Expected hash: 71da0ff755e5
Current hash:  ddaa40dcf34b

When a locked file changes on purpose, such as a corrected delivery, register it again. data_add() records the new hash.

fw.data_add("inputs/raw/survey.csv", name="inputs.raw.survey")

data_verify() checks a file against framework.db and, for projects connected to Framework Cloud, against the integrity ledger. It returns True when every record matches.

Inspecting the Catalog

data_info() returns how Framework will read one file: its absolute path, type, delimiter, locked flag, and description.

info = fw.data_info("inputs.raw.survey")
info.path
info.locked

data_list() returns every catalog entry as a DataFrame with the columns name, path, type, locked, and description.

fw.data_list()

An unknown name raises DataNotFoundError, and the message lists up to twenty names the catalog does define.

Organizing Data

Keep data in the stage folders, and name catalog entries after the folder.

inputs/
├── raw/           # inputs.raw.*: files as received, locked
├── intermediate/  # inputs.intermediate.*: cleaned and joined data
└── final/         # inputs.final.*: analysis-ready datasets

Raw data stays as it arrived. Each cleaning step reads from one stage and saves to the next, so the pipeline can be rerun from the raw files.