#Stata .dta

readDta reads Stata datasets of every released format from 104 to 119 (Stata 15 and later, more than 32,767 variables), in either byte order. read and readWith detect the format and call it for you.

import "@deckyard/statfiles/" as statfiles;

def ds as statfiles.Dataset init statfiles.read(fs.readBytes("auto.dta"));
io.printf("format %s, %d rows\n", $ds.version, $ds.rowCount);
io.printf("mean price %v\n", stats.mean(statfiles.present($ds, "price")));

statfiles.readDta(data, statfiles.options()) does the same without the detection. A file that is not a .dta throws there.

#Formats

Formats 106, 107, 109, 112 and 116 were never released and are refused.

#What maps to what

A float keeps its single-precision value (1.1 reads as 1.100000023841858). Display formats are returned as written; dates stay numbers (days or milliseconds since 1960-01-01, as the format says). String values keep their trailing blanks ("a " reads as "a ", as in pandas and ReadStat): a str<n> value ends at its first NUL or the field's end, and a strL is read as stored (an ASCII strL without its closing NUL). Variable and value labels lose trailing blanks. missingValues and missingRange stay empty: Stata has no declared missing values. Characteristics, notes, the sort order and the timestamp are skipped.

#Missing values

Stata stores a missing number as a value above the type's largest valid one. missing[i][row] holds its code and numbers[i][row] is 0.0.

A float or double between two codes takes the lower one, as Stata does; one above .z, a NaN or an infinity is an error. Before format 113 there is only .: the type's largest value (127, 32,767, 2,147,483,647, 2^127, 2^1023), and in formats 104 and 105 also the double 2^333.

A value label on an extended missing value is keyed by its code, so labelOf finds it for a missing cell:

io.printf("%s\n", statfiles.missingCode($ds, "income", 3));   # .a
io.printf("%s\n", statfiles.labelOf($ds, "income", 3));       # refused

#Encodings

Formats 118 and 119 are UTF-8. Older files are in the character set of the machine that wrote them, most often Windows-1252, the default here (as in pandas and ReadStat). Options.encoding overrides both, and Dataset.encoding reports the one used:

def opts as statfiles.Options init statfiles.options();
$opts.encoding = "iso-8859-1";
def ds as statfiles.Dataset init statfiles.readWith(fs.readBytes("old.dta"), $opts);

Text that is not valid in the character set (a UTF-8 file read as ASCII, a Latin-1 file read as UTF-8) is an error, not a replacement character. A binary strL is decoded like text too.

#Limits and errors

A file is refused whose rows times variables exceed Options.maxCells (default DEFAULT_MAX_CELLS) with kind "limit", before anything is allocated. Everything else that is wrong throws kind "statfiles":

• a truncated file, including an old-format file that ends inside its data or inside a value label table (one that ends after its data or between two tables is complete)
• an unknown format, byte order or variable type
• a tagged file whose sections are not where its map says, a missing tag, bytes after </stata_dta>
• a strL reference no GSO record defines, a GSO of an unknown type or defined twice
• a value label table whose sizes disagree or whose text offset lies outside it

Numeric columns are read in bulk, one call per column; reading 10,000 rows of 20 variables (16 numeric, 4 string) takes about 0.5 s, most of it in the string columns.