Skip to content
@deckyard/ngs

Cheatsheet

Every export, by module. Jennifer does not declare return types, so the returns column gives them. Every error is an Error whose kind is ngs.<module> unless noted.

core

Errors, field parsing, Go-speed counting and shell quoting. Every other module imports it.

constantvaluenotes
core.ERROR_KIND"ngs"prefix of every error kind the deck raises
core.TAB"\t"field separator of every text format here
functionreturnsnotes
core.fail(what, message)-always raises kind ngs.<what>
core.countIn(haystack, needle)intnon-overlapping occurrences, one native scan; 0 for an empty needle
core.parseInt(what, label, raw)intraises ngs.<what> naming label when raw is not an integer
core.parseFloat(what, label, raw)floatraises ngs.<what> when raw is not a number
core.reraise(what, line, e)-re-raises e untouched if its kind starts ngs., else raises ngs.<what> naming line
core.splitTab(line)list of stringsplit on tabs
core.field(fields, i, fallback)stringcolumn i (0-based), or fallback when out of range
core.intField(what, fields, i, fallback)intfallback for a missing, empty or . column; raises ngs.<what> on a non-integer
core.floatField(what, fields, i, fallback)floatas intField, for floats
core.isSkippable(line, marker)booltrue for a blank line or one starting with marker
core.padNum(n, width)stringzero-padded to width; negative values unpadded
core.shellQuote(s)stringsingle-quoted for POSIX sh, embedded ' escaped
core.reverseComplement(seq)stringACGTUN and IUPAC codes, case kept; other characters unchanged
core.complementBase(b)stringcomplement of one base, or b when not a nucleotide code
core.gcCount(seq)intG, C, g and c, counted natively
core.nCount(seq)intN and n, counted natively
core.hasTool(name)boola regular file of that name on PATH (executable bit not checked); walks PATH with fs, no exec needed; a name with a separator is checked as a path
core.sameFile(a, b)boolboth exist and resolve, symlinks followed, to the same file

source

Streaming line sources and sinks, gzip in both directions (Streaming large files). Errors are ngs.source.

constantvaluenotes
source.GZIP_MAGIC10x1ffirst gzip magic byte (RFC 1952)
source.GZIP_MAGIC20x8bsecond gzip magic byte

source.Source

An open line source. Copies share the stream: file and proc are registry handles.

fieldnotes
file as fs.Filethe handle lines are read from
proc as os.Processthe decompressor, when piped
helper as stringFIFO directory or temp file removed on close, "" when none
piped as boolwhether a child process feeds the source
origin as stringthe path opened, or "<stdin>"

source.Sink

An open line sink.

fieldnotes
file as fs.Filethe handle lines are written to
proc as os.Processthe compressor, when piped
helper as stringFIFO directory removed on close, "" when none
piped as boolwhether a child process drains the sink
origin as stringthe path opened
functionreturnsnotes
source.isGzip(path)boolfirst two bytes are 1f 8b; raises when the file is missing
source.open(path)Sourcegzip detected from content
source.openPlain(path)Sourceraises when the file is missing
source.openStdin()Sourcevia /dev/stdin, Unix only; raises when it is missing, runtime when it cannot be reopened
source.openGzip(path)Sourcestreams gzip -dc through a FIFO; raises without exec or when mkfifo fails
source.openGzipBuffered(path)Sourcedecompresses the whole file in memory to a temp file; small inputs only
source.hasNext(s)boolfalse at end of stream
source.nextLine(s)stringnext line, terminator stripped; raises at end of stream
source.close(s)intdecompressor exit code, 0 without one; releases child and temp files even when it raises
source.closeStrict(s)-raises when the decompressor exited non-zero; use after reading to the end
source.create(path)Sinkgzip when the name ends in .gz
source.createPlain(path)Sinkcreates or truncates
source.createGzip(path)Sinkgzip -c reading a FIFO; raises without exec or when mkfifo fails
source.writeText(k, text)-writes text as is
source.writeLine(k, line)-writes line and "\n"
source.closeSink(k)intflushes, waits for the compressor; its exit code, 0 without one

fastq

FASTQ records, reader and writer (Formats). Errors are ngs.fastq.

constantvaluenotes
fastq.PHRED3333Sanger / Illumina 1.8+ quality offset
fastq.PHRED6464legacy Illumina 1.3-1.7 offset

fastq.Read

fieldnotes
id as stringheader up to the first space, without @
desc as stringheader after the first space, "" when none
seq as stringsequence
qual as stringraw ASCII qualities, same length as seq

fastq.Reader

fieldnotes
src as source.Sourcethe underlying line source
functionreturnsnotes
fastq.open(path)Readergzip transparent
fastq.openStdin()Readerstandard input
fastq.hasNext(r)boolanother record available
fastq.next(r)Readraises on a truncated or malformed record, naming its header
fastq.close(r)intdecompressor exit code
fastq.closeStrict(r)-raises when the decompressor failed
fastq.parse(header, seq, plus, qual)Readraises on a bad @ or + marker or a length mismatch
fastq.format(rec)stringfour lines, newline-terminated, separator +
fastq.toFasta(rec)stringtwo lines, newline-terminated
fastq.headerOf(rec)stringid, plus " " + desc when present
fastq.write(k, rec)-writes format(rec) to a sink
fastq.length(rec)intbases
fastq.qualities(rec, offset)list of intone Phred score per base; one step per base
fastq.meanQuality(rec, offset)floatarithmetic mean Phred, 0.0 for an empty read
fastq.minQuality(rec, offset)intlowest Phred, -1 for an empty read
fastq.gcPercent(rec)float0-100, 0.0 for an empty read
fastq.ambiguousCount(rec)intN bases
fastq.subseq(rec, start, end)Read0-based half-open slice of seq and qual; raises when out of range
fastq.reverseComplement(rec)Readqualities reversed to match

bed

BED as intervals.Interval (Formats). Errors are ngs.bed.

constantvaluenotes
bed.MISSING"."written for an absent name or strand

bed.Reader

fieldnotes
src as source.Sourcethe underlying line source

bed.Next

variantnotes
Featurethe next record, in value (intervals.Interval)
Doneend of stream
functionreturnsnotes
bed.open(path)Readergzip transparent
bed.openStdin()Readerstandard input
bed.next(r)Nextskips blank, #, track and browser lines
bed.close(r)intdecompressor exit code
bed.closeStrict(r)-raises when the decompressor failed
bed.parse(line)intervals.Intervalfirst six columns; raises on fewer than three or a non-numeric field
bed.format(iv)stringBED6, no newline; . for an empty name or strand
bed.readAll(path)list of intervals.Intervalwhole file, file order
bed.writeAll(path, ivs)-gzip when the name ends in .gz; raises when the compressor fails
bed.write(k, iv)-one BED6 line to an open sink

gff

GFF3 and GTF (Formats). Errors are ngs.gff.

constantvaluenotes
gff.MISSING"."placeholder for an absent column
gff.STYLE_GFF3"gff3"key=value; attributes, percent-escaped
gff.STYLE_GTF"gtf"key "value"; attributes

gff.Feature

fieldnotes
seqid as stringreference sequence
origin as stringthe source column
kind as stringthe type column
start as int1-based, inclusive, as written
end as int1-based, inclusive, as written
score as float0.0 when absent
strand as string+, - or .
phase as intCDS phase 0-2, -1 when absent
attributes as map of string to stringthe ninth column, in file order

gff.Reader

fieldnotes
src as source.Sourcethe underlying line source

gff.Next

variantnotes
Recordthe next feature, in value (Feature)
Doneend of stream, or a ##FASTA line
functionreturnsnotes
gff.open(path)Readergzip transparent
gff.openStdin()Readerstandard input
gff.next(r)Nextskips blank and # lines, stops at ##FASTA
gff.close(r)intdecompressor exit code
gff.closeStrict(r)-raises when the decompressor failed
gff.parse(line)Featurestyle detected per record; raises on fewer than nine columns
gff.parseAttributes(text)map of string to stringGFF3 values percent-decoded; empty for "" or .
gff.styleOf(text)stringSTYLE_GFF3 or STYLE_GTF, from the first pair
gff.attribute(f, key, fallback)stringvalue, or fallback
gff.identifier(f)stringID, else transcript_id, else gene_id, else ""
gff.name(f)stringName, else gene_name, else identifier(f)
gff.toInterval(f)intervals.Intervalstart - 1, end kept, named by name(f)
gff.fromInterval(iv, origin, kind)Featurestart + 1; ID from the interval's name; phase -1
gff.format(f, style)stringnine columns, no newline; score 0.0 written as .
gff.formatAttributes(attrs, style)string. for an empty map; GFF3 escapes only reserved characters
gff.readAll(path)list of Featurewhole file up to ##FASTA; closes without checking the decompressor
gff.readIntervals(path, kind)list of intervals.Intervalconverted features of type kind, "" for all

sam

SAM alignments (Formats). Errors are ngs.sam.

constantvaluenotes
sam.FLAG_PAIRED0x1paired in sequencing
sam.FLAG_PROPER_PAIR0x2each segment properly aligned
sam.FLAG_UNMAPPED0x4segment unmapped
sam.FLAG_MATE_UNMAPPED0x8mate unmapped
sam.FLAG_REVERSE0x10segment on the reverse strand
sam.FLAG_MATE_REVERSE0x20mate on the reverse strand
sam.FLAG_FIRST0x40first segment of the template
sam.FLAG_LAST0x80last segment of the template
sam.FLAG_SECONDARY0x100secondary alignment
sam.FLAG_QC_FAIL0x200failed platform or vendor QC
sam.FLAG_DUPLICATE0x400PCR or optical duplicate
sam.FLAG_SUPPLEMENTARY0x800supplementary alignment
sam.MISSING"*"placeholder for an absent column

sam.Record

fieldnotes
qname as stringquery template name
flag as intFLAG bits
rname as stringreference name, * when unmapped
pos as int1-based leftmost position, 0 when unmapped
mapq as intmapping quality
cigar as stringraw CIGAR, * when unavailable
rnext as stringmate's reference name
pnext as intmate's 1-based position
tlen as intobserved template length, signed
seq as stringsegment sequence, * when absent
qual as stringraw ASCII qualities, * when absent
tags as list of stringoptional TAG:TYPE:VALUE columns, verbatim

sam.CigarOp

fieldnotes
length as intbases covered
op as stringone of M I D N S H P = X

sam.Reader

fieldnotes
src as source.Sourcethe underlying line source
header as list of stringthe @ lines, empty on stdin

sam.Next

variantnotes
Alignmentthe next record, in value (Record)
Doneend of stream
functionreturnsnotes
sam.open(path)Readerreads the header in a separate pass
sam.openStdin()Readerstandard input; header empty
sam.next(r)Nextskips @ and blank lines
sam.close(r)intdecompressor exit code
sam.closeStrict(r)-raises when the decompressor failed
sam.parse(line)Recordraises on fewer than eleven columns or a non-numeric field
sam.format(rec)stringone record, no newline
sam.hasFlag(rec, bit)booltrue when any bit of bit is set
sam.isUnmapped(rec)boolFLAG 0x4
sam.isReverse(rec)boolFLAG 0x10
sam.isSecondary(rec)boolFLAG 0x100
sam.isSupplementary(rec)boolFLAG 0x800
sam.isDuplicate(rec)boolFLAG 0x400
sam.isPrimary(rec)boolnone of 0x4, 0x100, 0x800
sam.strandOf(rec)string+, -, or . when unmapped
sam.parseCigar(cigar)list of CigarOpempty for *; raises on a malformed CIGAR
sam.referenceLength(cigar)intsum of M D N = X
sam.queryLength(cigar)intsum of M I S = X
sam.toInterval(rec)intervals.Intervalpos - 1, end from CIGAR (sequence length for *); named by qname, scored by MAPQ; raises when unmapped
sam.tagValue(rec, tagName, fallback)stringvalue with TAG:TYPE: removed, or fallback
sam.referenceSizes(r)map of string to int@SQ SN to LN
sam.headerOf(path)list of stringthe @ lines only

vcf

VCF variants (Formats). Errors are ngs.vcf.

constantvaluenotes
vcf.MISSING"."placeholder for an absent column
vcf.FIXED_COLUMNS9columns before the first sample: eight fixed plus FORMAT

vcf.Variant

fieldnotes
chrom as stringreference sequence
pos as int1-based position of the first REF base
id as string"" when .
ref as stringreference allele
alt as list of stringalternate alleles, empty when .
qual as floatPhred-scaled quality, -1.0 when .
filter as stringFILTER verbatim
info as map of string to stringINFO; a flag maps to ""
format as list of stringFORMAT keys in order
samples as list of stringper-sample columns verbatim

vcf.Reader

fieldnotes
src as source.Sourcethe underlying line source
meta as list of string## lines, empty on stdin
samples as list of stringsample names from #CHROM, empty on stdin

vcf.Next

variantnotes
Recordthe next variant, in value (Variant)
Doneend of stream
functionreturnsnotes
vcf.open(path)Readerreads the header in a separate pass
vcf.openStdin()Readerstandard input; meta and samples empty
vcf.next(r)Nextskips # and blank lines
vcf.close(r)intdecompressor exit code
vcf.closeStrict(r)-raises when the decompressor failed
vcf.headerOf(path)list of stringthe leading # lines
vcf.parse(line)Variantraises on fewer than eight columns or a non-numeric POS or QUAL
vcf.parseInfo(text)map of string to stringvalue from the first =; flags map to ""; empty for .
vcf.info(v, key, fallback)stringINFO value, or fallback
vcf.hasInfo(v, key)boolkey present
vcf.passed(v)boolFILTER is PASS, . or empty
vcf.isSnv(v)boolREF and every ALT one of A C G T N
vcf.isIndel(v)boolan ALT of different length from REF; symbolic and breakend ALTs excluded
vcf.sampleValue(v, sampleIndex, key, fallback)stringFORMAT value for a 0-based sample, or fallback
vcf.genotype(v, sampleIndex)stringGT, ./. when unavailable
vcf.toInterval(v)intervals.Intervalpos - 1, spans REF; named by ID; QUAL as score, 0.0 when absent
vcf.format(v)stringone record, no newline; QUAL rendered as a float
vcf.formatInfo(entries)stringkey=value or bare flag, ;-joined; . when empty

intervals

Interval arithmetic, 0-based half-open (Interval arithmetic). Errors are ngs.intervals. Set operations return results in the order of sort(a).

constantvaluenotes
intervals.KEY_WIDTH12digits coordinates are zero-padded to in sort keys
intervals.MAX_COORDINATE1000000000000one past the largest allowed coordinate (10^12)

intervals.Interval

fieldnotes
chrom as stringreference sequence
start as int0-based, inclusive
end as int0-based, exclusive
name as string"" when unnamed
score as float0.0 when absent
strand as string+, - or .
functionreturnsnotes
intervals.make(chrom, start, end)Intervalunnamed, strand .; validated as named
intervals.named(chrom, start, end, name, score, strand)Intervalraises on a negative or inverted range or an end of MAX_COORDINATE or more
intervals.length(iv)intend - start
intervals.overlaps(a, b)boolshare a base; book-ended intervals do not
intervals.overlapLength(a, b)intshared bases, 0 when none
intervals.contains(outer, inner)boolsame chromosome, inner within outer
intervals.intersection(a, b)Intervalshared region with a's name, score and strand; raises when they do not overlap
intervals.sortKey(iv)stringchrom, start, end, tab-separated, coordinates zero-padded; exported as the func value for lists.sortBy
intervals.sort(ivs)list of Intervalby chromosome as text, start, end; sorted input returned as is
intervals.merge(ivs, maxGap)list of Intervalper chromosome; 0 joins overlapping and book-ended (bedtools merge -d 0); names, scores, strands cleared
intervals.totalLength(ivs)intbases in the union
intervals.intersect(a, b)list of Intervalone per overlapping pair, a's name (bedtools intersect)
intervals.countOverlaps(a, b)list of inttargets overlapping each query (intersect -c)
intervals.coverage(a, b)list of intbases of each query covered by merged b
intervals.subtract(a, b)list of Intervaluncovered pieces of each query, names kept (bedtools subtract)
intervals.complement(ivs, sizes)list of Intervalgaps on the chromosomes in sizes, sorted (bedtools complement)
intervals.slop(iv, left, right, limit)Intervalgrown, clamped at 0 and at limit when limit > 0
intervals.byChromosome(ivs)map of string to list of Intervalsorted groups
intervals.toBed3(iv)stringchrom\tstart\tend, no newline

qc

One-pass read QC (Read quality control).

constantvaluenotes
qc.MAX_PHRED94size of the quality histogram, Phred 0-93
qc.Q3030threshold for q30Percent
qc.DEFAULT_BATCH2000records per counting batch
qc.DEFAULT_BATCH_BASES4000000bases per counting batch
qc.ADAPTER_TRUSEQ"AGATCGGAAGAGC"Illumina TruSeq / NEBNext universal
qc.ADAPTER_NEXTERA"CTGTCTCTTATACACATCT"Nextera / transposase
qc.ADAPTER_SMALL_RNA"TGGAATTCTCGG"Illumina small-RNA 3'
qc.ARTEFACT_POLY_G"GGGGGGGGGG"poly-G of two-colour chemistry (NovaSeq, NextSeq)

qc.Options

The zero value is valid.

fieldnotes
maxReads as intstop after sampling this many; 0 is no limit
everyNth as intsample one read in N; 0 or 1 is every read
perBase as boolcollect per-position quality, one step per base
offset as intquality offset; 0 means 33
batchSize as intrecords per batch; 0 means DEFAULT_BATCH
maxBatchBases as intbases per batch; 0 means DEFAULT_BATCH_BASES
adapters as list of stringprobes to count; empty means defaultAdapters()

qc.Report

fieldnotes
reads as intrecords read
sampled as intrecords measured
bases as intbases in sampled records
minLength as intshortest sampled read, 0 when none
maxLength as intlongest sampled read
gcBases as intG and C bases
nBases as intN bases
qualHist as list of intbases per Phred score, indexed 0-93
lengthHist as map of int to intread length to read count
adapterHits as map of string to intprobe to occurrences
perBaseSum as list of intquality sum per position, perBase only
perBaseCount as list of intreads reaching each position, perBase only
functionreturnsnotes
qc.options()Optionsevery read, Phred+33, default batch bounds
qc.defaultAdapters()list of stringthe four probe constants
qc.analyze(path, opts)Reportone streaming pass; raises ngs.qc on qualities outside Phred 0-93
qc.meanQuality(rep)floatmean Phred over sampled bases
qc.q30Percent(rep)floatpercentage of bases at Q30 or above
qc.gcPercent(rep)floatpercentage G+C
qc.meanLength(rep)floatmean sampled read length
qc.perBaseMean(rep)list of floatmean quality per position, empty without perBase
qc.summary(rep)stringmulti-line text report

trim

Trimming, filtering and subsampling (Trimming and filtering). Errors are ngs.trim.

constantvaluenotes
trim.DEFAULT_WINDOW4sliding window width
trim.DEFAULT_OVERLAP6shortest partial 3' adapter match

trim.Options

The zero value trims nothing; its maxAmbiguous of 0 still drops reads holding an N.

fieldnotes
trimFront as intbases cut from the 5' end
trimTail as intbases cut from the 3' end
adapters as list of stringadapters to clip; empty disables
minOverlap as intshortest partial adapter; 0 means DEFAULT_OVERLAP
qualityThreshold as intwindow mean quality to keep; 0 disables
windowSize as intwindow width; 0 means DEFAULT_WINDOW
maxLength as intlength cap; 0 is none
minLength as intshorter reads dropped after trimming
maxAmbiguous as intreads with more N bases dropped; -1 disables
minMeanQuality as intreads below this mean dropped; 0 disables
offset as intquality offset; 0 means 33

trim.Stats

fieldnotes
readsIn as intrecords read
readsOut as intrecords written
basesIn as intbases read
basesOut as intbases written
adapterTrimmed as intrecords shortened by adapter clipping
qualityTrimmed as intrecords shortened by quality trimming
tooShort as intdropped by minLength
tooAmbiguous as intdropped by maxAmbiguous
lowQuality as intdropped by minMeanQuality
functionreturnsnotes
trim.options()Optionsbuilt-in adapters, Q20 window of 4, minLength 25, maxAmbiguous -1
trim.run(inPath, outPath, opts)Statsstreams; raises when input and output are the same file or the compressor fails
trim.trimRead(rec, opts)fastq.Readthe four trimming steps on one record
trim.clipFixed(rec, front, tail)fastq.Readempty when the cuts overlap
trim.clipAdapters(rec, adapters, minOverlap)fastq.Readcut at the earliest full or 3' partial adapter
trim.adapterStart(seq, adapters, minOverlap)int0-based cut position, -1 when none; minOverlap below 1 means 6
trim.clipQuality(rec, threshold, window, offset)fastq.Readcut at the first window below threshold; 0 disables
trim.qualityKeepLength(qual, threshold, window, offset)intbases kept from the 5' end
trim.truncate(rec, maxLength)fastq.Read5' end kept; 0 or less is no cap
trim.keeps(rec, opts)boolpasses length, N and mean-quality filters
trim.subsampleEvery(inPath, outPath, everyNth, maxReads)Statsevery Nth record; maxReads 0 is no limit
trim.subsampleFraction(inPath, outPath, fraction)Statsrandom, fraction in (0, 1]; seed with math.randSeed

pipeline

External tools (Pipeline orchestration). Errors are ngs.pipeline.

pipeline.Job

fieldnotes
name as stringlabel in logs and results
argv as list of stringcommand and arguments, run without a shell
inputs as list of stringfiles read
outputs as list of stringfiles produced
stdin as stringtext fed to standard input
allowFailure as boola non-zero exit is reported, not raised

pipeline.Result

fieldnotes
name as stringthe job's label
exitCode as intprocess exit code, 0 when skipped
stdout as stringcaptured standard output
stderr as stringcaptured standard error
skipped as boolthe job was up to date
allowFailure as boolcopied from the job
functionreturnsnotes
pipeline.job(name, argv)Jobraises on an empty argv
pipeline.shell(name, cmdline)Jobruns sh -c cmdline; quote untrusted values with core.shellQuote
pipeline.needing(j, inputs)Jobcopy with inputs set
pipeline.producing(j, outputs)Jobcopy with outputs set
pipeline.feeding(j, text)Jobcopy with stdin set
pipeline.tolerant(j)Jobcopy with allowFailure set
pipeline.needsRun(j)booltrue without outputs, with an output missing, or with an input not older than the oldest output; raises on a missing input
pipeline.run(j)Resultskips an up-to-date job; raises without exec
pipeline.runAll(jobs)list of Resultin order; raises at the first real failure
pipeline.runParallel(jobs, width)list of Resultwaves of width (0 or less: os.NCPU), input order; does not check
pipeline.check(res)-raises on a non-zero exit without allowFailure
pipeline.checkAll(results)-check on each, raising at the first failure
pipeline.allOk(results)boolno real failure
pipeline.hasTool(name)boolcore.hasTool
pipeline.requireTools(names)-raises listing every missing tool
pipeline.workDir(tag)stringfresh temp directory; the caller removes it
pipeline.cleanUp(dir)-recursive delete; raises for a root, the working directory, . or ..
pipeline.summary(results)stringone skip / ok / warn / fail line per job

ngs

The entry module. Errors are ngs.ngs.

constantvaluenotes
ngs.VERSION"0.1.0"the deck's version
ngs.NAME"@deckyard/ngs"the package name

ngs.Capabilities

fieldnotes
streamingGzip as boolthe host has exec, which streaming gzip needs (gzipOnPath is not consulted); false means source.openGzipBuffered
externalTools as boolpipeline can run programs
gzipOnPath as boola gzip executable is on PATH
functionreturnsnotes
ngs.capabilities()Capabilitieswhat this interpreter and host support
ngs.describe()stringname, version and capabilities on one line
ngs.requireFullHost()-raises without the exec capability