4.8.2 smORF cluster¶

Function¶

smorf_cluster reduces the large candidate ORF set produced by smorf_scanner into non-redundant ORF families. It applies sequence-based quality filtering, removes ORFs that overlap annotated ORFs in the same reading frame, collapses exact transcript duplicates, and groups the remaining ORFs that share the same stop boundary into alternative-start families. Each family is represented by one primary ORF and described by a set of family-level summary columns.

The module is transcript-centric and does not use Ribo-seq evidence.

Workflow¶

Basic filtering → Annotated-overlap removal → Exact deduplication → Family clustering

Step	Purpose
1	Basic filtering
2	Annotated-overlap removal
3	Exact deduplication
4	Family clustering

Input files¶

Input	Used by	Description
ORF message table (`-i`)	`smorf_cluster`	Candidate ORF table generated by `smorf_scanner` (e.g. `gmx4.message.txt`).
genePred annotation (`-a`)	`smorf_cluster`	Transcript annotation used to check transcript structure, detect same-frame overlaps with annotated ORFs, and train the `auto` Kozak model.

Parameters¶

4.8.2.1 Basic options¶

Parameter	Required	Default	Description
`-i`, `--input`	Yes		Input ORF message table generated by `smorf_scanner`.
`-a`, `--annotation`	Yes		genePred/genePredExt annotation file.
`-o`, `--out-prefix`	No	`ORF.cluster`	Output file prefix.

4.8.2.2 Basic filtering arguments¶

Parameter	Required	Default	Description
`-s`, `--keep-start-codons`	No	`all`	Comma-separated start codons to retain; `all` keeps every start codon.
`-m`, `--min-aa`	No	`8`	Minimum peptide length.
`-M`, `--max-aa`	No	`10000`	Maximum peptide length.
`-c`, `--keep-categories`	No	`uORF,dORF,lncORF,iORF,same_frame_iORF,emORF,overlap_uORF,overlap_dORF,other_ORF,annotated_ORF,annotated_mORF`	Comma-separated category allowlist to keep.
`-r`, `--remove-categories`	No	`antisense_ORF`	Comma-separated category blacklist to remove.
`-A`, `--keep-antisense`	No	`False`	Retain antisense ORFs.
`--max-ambiguous-codons`	No	`0`	Maximum ambiguous codon count allowed.

4.8.2.3 Family clustering arguments¶

Parameter	Required	Default	Description
`--start-priority`	No	`ATG`	Start-codon priority used by the primary-ORF selection rule.

4.8.2.4 Optional Kozak arguments¶

Parameter	Required	Default	Description
`-k`, `--kozak-model`	No	`none`	Kozak model: `none`, `auto`, or a built-in model (`plant`, `vertebrate`, `drosophila`, `yeast`).
`--kozak-pwm`	No		Custom Kozak PWM file; overrides `--kozak-model`.
`-w`, `--min-kozak-score`	No	`0.0`	Minimum normalized Kozak score in [0, 1]. Only effective when a Kozak model is enabled.
`--export-kozak-pwm`	No		Optional output path to write the selected PWM.

4.8.2.5 Runtime arguments¶

Parameter	Required	Default	Description
`-t`, `--thread`	No	`0`	Worker processes. Zero uses the SLURM allocation or the CPU count; small inputs fall back to a single worker.

Output files¶

With -o gmx4.reliable four files are written:

File	Description
`<prefix>.family.message.txt`	One row per clustered family (the primary ORF row), containing all input columns plus filtering, annotated-overlap and cluster columns.
`<prefix>.family.members.txt`	One row per ORF member mapping it to its family, primary ORF and representative ORF.
`<prefix>.removed.message.txt`	One row per ORF removed during filtering, with filtering status and removal reasons.
`<prefix>.cluster_summary.txt`	Overall statistics, removal-reason counts, and primary-category counts.

family.message.txt¶

The family table keeps all scanner columns and appends the annotated-overlap columns, filtering columns and cluster columns:

Column group	Columns
Annotated overlap (4)	`matched_annotated_orf_id`, `annotated_overlap_relation`, `annotated_overlap_nt`, `annotated_overlap_codon`
Filtering (7)	`kozak_pwm_name`, `kozak_pwm_score`, `kozak_pwm_level`, `kozak_valid_ratio`, `structure_status`, `filter_status`, `filter_reason`
Family (12)	`family_id`, `family_role`, `family_type`, `family_size`, `family_representative_count`, `family_transcript_count`, `family_exact_duplicate_count`, `family_alt_start_count`, `family_collapsed_count`, `family_categories`, `family_start_codons`, `primary_selection_rule`

Key columns:

Column	Description
`family_id`	Unique family identifier, formatted as `SMORF_FAM%09d`.
`family_role`	`PRIMARY` for the row representing the family.
`family_type`	`SINGLETON`, `TRANSCRIPT_DUPLICATE`, `ALTERNATIVE_START`, or `MIXED`.
`family_size`	Total number of members in the family.
`family_representative_count`	Number of distinct representative ORFs.
`family_transcript_count`	Number of distinct transcripts.
`family_exact_duplicate_count`	Number of exact-duplicate members collapsed.
`family_alt_start_count`	Number of alternative-start members collapsed.
`family_collapsed_count`	Total number of collapsed members.
`family_categories`	Comma-separated union of member categories.
`family_start_codons`	Comma-separated union of member start codons.
`primary_selection_rule`	Rule string used to select the primary ORF.

Example row (truncated):

orf_id  gene_id transcript_id   chrom   strand  source_strand   category    ... structure_status    filter_status   filter_reason   family_id   family_role family_type family_size ... primary_selection_rule
SMORF0000001    Glyma.01G000100 TX_001  chr01   +   +   uORF    ... PASS    PASS    SMORF_FAM000000001  PRIMARY SINGLETON   1   ... annotated>start_codon>aa_length>kozak_score>transcript_length>input_order

family.members.txt¶

Eleven columns per member:

Column	Description
`family_id`	Family identifier.
`primary_orf_id`	Primary ORF of the family.
`representative_orf_id`	Representative ORF of the family.
`member_orf_id`	ORF id of this member.
`member_transcript_id`	Transcript of this member.
`member_category`	Category of this member.
`member_start_codon`	Start codon of this member.
`member_aa_length`	Peptide length of this member.
`member_transcript_length`	Transcript length of this member.
`family_role`	`PRIMARY` or `COLLAPSED`.
`collapse_reason`	`PRIMARY`, `ALTERNATIVE_START`, or `EXACT_TRANSCRIPT_DUPLICATE`.

removed.message.txt¶

The removed table keeps all scanner columns and appends the annotated-overlap columns and filtering columns (same groups as the family table, without the family columns). The filter_reason column lists the reasons for removal as a semicolon-separated string, for example category_not_allowed:emORF or contained_in_annotated_ORF_same_frame.

cluster_summary.txt¶

Three sections:

overall — input ORFs, basic-filter passed/removed, annotated-overlap removed, lncORF-mORF removed, exact duplicates collapsed, alternative starts collapsed, total collapsed, family count, singleton/multi-member family counts, genes processed, effective workers.
removal_reason — count of ORFs per removal reason.
primary_category — count of families per primary-ORF category.

Clustering rules¶

Basic filtering¶

For each candidate ORF:

Category: remove if the category is in --remove-categories; keep only categories in --keep-categories; antisense ORFs are removed unless --keep-antisense is set.
Length: aa_length must be within [--min-aa, --max-aa] and nt_length must equal aa_length * 3 + 3; a complete ORF must end with a valid stop codon.
Structure: exon blocks must be valid; the transcript must exist in the genePred annotation; gene, chrom and strand must match.
Ambiguous codons: the ambiguous-codon count must not exceed --max-ambiguous-codons (default 0).

Annotated-overlap removal¶

A non-annotated ORF is removed when it overlaps an annotated ORF in the same frame (phase % 3 == 0). The overlap relation is recorded as one of:

matches_annotated_ORF
contained_in_annotated_ORF_same_frame
extends_annotated_ORF_same_frame
partial_overlap_annotated_ORF_same_frame

Exact deduplication¶

ORFs are grouped by (chrom, strand, source_strand, blocks). Within each group one representative is kept; representatives are ranked by category level, then transcript length, then input order.

Family clustering¶

Representatives are bucketed by (chrom, strand, source_strand, stop_boundary, stop_codon) and joined with a union-find structure over a spliced suffix index. The resulting family_type is:

SINGLETON — a family with a single member.
TRANSCRIPT_DUPLICATE — members that are exact duplicates across transcripts.
ALTERNATIVE_START — members that share the same stop but use different start codons.
MIXED — a family containing both exact duplicates and alternative starts.

Primary selection¶

Within each family the primary ORF is selected by:

annotated > start_codon > aa_length > kozak_score > transcript_length > input_order

Annotated ORFs (annotated_ORF / annotated_mORF) are protected and always become primary.

Kozak model¶

Built-in models: plant (consensus ACAACAATGGC), vertebrate (GCCRCCATGG), drosophila (ATMAAMATGAMC), yeast (AAAAAAATGTCT).
auto: trains a species-specific PWM from the Kozak contexts of annotated ORFs in the input; at least 100 records are required, otherwise it falls back to plant.
Scoring: the normalized score is the mean PWM weight over informative positions, and requires a valid-ratio (fraction of informative positions containing A/C/G/T) of at least 0.5. Levels: strong ≥ 0.75, moderate ≥ 0.50, otherwise weak.

Examples¶

cd ./sce/5.smorf/02.cluster

smorf_cluster \
  -i ../01.smorf_scan/gmx4.message.txt \
  --annotation ~/gmx/norm/gmx4.genepred \
  -o gmx4.reliable \
  --min-aa 8 \
  --max-aa 10000 \
  --kozak-model auto \
  --keep-categories uORF,dORF,lncORF,annotated_ORF \
  --thread 20

Real run on soybean gmx4 data:

13,347,430 input ORFs → 6,629,760 passed basic filtering (6,717,670 removed).
250,133 ORFs removed by annotated-overlap checks; 2,605,356 exact duplicates and 2,722,340 alternative-start members collapsed.
1,051,931 families from 58,249 genes; 20 workers; ~80 seconds.

Excerpt of the generated cluster_summary.txt:

overall:
  input_orfs                  13347430
  basic_passed                 6629760
  basic_removed                6717670
  annotated_overlap_removed     250133
  lnc_morf_removed                2214
  post_annotation_retained     6379627
  exact_duplicates_collapsed   2605356
  alt_starts_collapsed         2722340
  total_collapsed              5327696
  family_count                 1051931
  singleton_families            276826
  multi_member_families         775105
  genes_processed                58249
  effective_workers                 20

removal_reason:
  category_not_allowed:emORF                 139014
  category_not_allowed:iORF                 5885393
  category_not_allowed:overlap_dORF          392445
  category_not_allowed:overlap_uORF          300818
  contained_in_annotated_ORF_same_frame      168707
  extends_annotated_ORF_same_frame             5254
  matches_annotated_ORF                        2333
  partial_overlap_annotated_ORF_same_frame     73839

primary_category:
  annotated_ORF   57952
  dORF           341871
  lncORF         386975
  uORF           265133

Notes¶

smorf_cluster performs sequence-based filtering and clustering only; it does not use Ribo-seq evidence.
--min-kozak-score takes effect only when a Kozak model is enabled (--kozak-model or --kozak-pwm).
Family identifiers are renumbered after parallel merging, so they are continuous across the final family table.
For small inputs (below the parallel threshold) a single worker is used regardless of --thread.