@deckyard/ngs

Streaming file formats, read QC, trimming, interval arithmetic and pipeline orchestration for next-generation sequencing data, written in Jennifer.

This deck is part of Deckyard, community decks for Jennifer.

Scope

Every reader is a pull cursor over one line at a time, and nothing in the deck loads a sequencing file into memory: a 50 GB input costs the same memory as a 50 MB one.

A tree-walking interpreter is orders of magnitude too slow to replace SIMD-optimised C, so the deck is the glue and light-I/O layer of a pipeline: string handling, bookkeeping and process management.

area	what you get
Format I/O	FASTQ, SAM, VCF, BED, GFF3/GTF: parse, filter and convert as a stream
QC	quality, length and GC distributions, adapter counts, in one pass
Preprocessing	adapter and quality trimming, length and content filters, subsampling
Interval operations	overlap, intersect, merge, subtract, coverage, complement
Pipeline orchestration	run the standard tools, skip finished steps, fan out over samples

Out of scope

Drive these through pipeline and parse their text output:

read alignment (BWA, Bowtie2, minimap2)
de-novo assembly (SPAdes)
variant calling (GATK)
BAM and CRAM. These also need BGZF (blocked gzip with random access), which the compress library does not provide; pipe samtools view -h output into the SAM reader instead.

The modules

Each module is imported on its own; importing fastq.j does not load the pipeline layer.

module	what it does
`source`	streaming line sources and sinks, gzip in both directions
`fastq`	FASTQ records, reader, writer, per-record accessors
`bed`	BED features as `intervals.Interval`
`gff`	GFF3 and GTF features, 1-based to 0-based conversion
`sam`	SAM alignments, FLAG and CIGAR handling
`vcf`	VCF variants, INFO and per-sample FORMAT access
`intervals`	overlap, merge, intersect, subtract, coverage, complement, slop
`qc`	one-pass FastQC-style statistics
`trim`	fastp-style trimming, filtering and subsampling
`pipeline`	run, skip, check and fan out external tools
`core`	errors, field parsing, Go-speed counting, shell quoting
`ngs`	the entry module: name, version, host capability check

Every export is in the cheatsheet.

Example

jennifer

use io;
import "@deckyard/ngs/fastq.j" as fastq;
import "@deckyard/ngs/qc.j" as qc;

def report as qc.Report init qc.analyze("sample.fastq.gz", qc.options());
io.printf("%s", qc.summary($report));

def r as fastq.Reader init fastq.open("sample.fastq.gz");
while (fastq.hasNext($r)) {
    def rec as fastq.Read init fastq.next($r);
    io.printf("%s\t%d\t%f|prec=1\n", $rec.id, fastq.length($rec), fastq.gcPercent($rec));
}
fastq.closeStrict($r);

Read Coordinate systems before writing interval code, and Installing and importing for what the host needs.

Licence

LGPL-3.0-only.

#@deckyard/ngs

#Scope

#Out of scope

#The modules

#Example

#Licence