CryoNAV Overview Tomogram Processing: Difference between revisions
Bot edit via cryonavedit.py: migrate to sidebar-target titles |
Bot edit via cryonavedit.py: remove obsolete template-system language |
||
| Line 16: | Line 16: | ||
== Job submission == | == Job submission == | ||
At submission time | At submission time the user selects one or more tilt series, fills in the parameter form for the chosen pipeline step (defaults come from the catalog module definition), and picks an execution backend. CryoNAV constructs the appropriate command-line invocation, records the parameter set and command alongside the run for provenance, and submits the job. | ||
Steps can be submitted as a chain: the user selects the sequence of catalog modules to run, and each step is submitted automatically once its upstream dependencies complete. Failed steps halt the rest of the chain; downstream parameters can still be edited later before the chain is resumed. | |||
== Local and HPC execution == | == Local and HPC execution == | ||
Revision as of 22:36, 20 May 2026
CryoNAV's processing pipeline is built around IMOD's core utilities and extended with deep-learning denoising methods. The pipeline is accessed through two complementary approaches:
- Per-tilt-series -- the interface mirrors the step-by-step workflow familiar from IMOD's eTomo: navigate to a tilt series, run a processing step, review results (thumbnails, alignment residuals, CTF plots), and proceed to the next step with adjusted parameters if needed.
- Batch -- select a batch of tilt series and submit them all for processing with a single action, applying the same parameters across the set.
Pipeline steps
The processing pipeline for a typical tilt series:
- Motion correction -- correcting beam-induced specimen movement within individual tilt images.
- CTF estimation and correction -- measuring and fitting the contrast transfer function for defocus determination.
- Tilt series alignment -- aligning the tilt series using fiducial markers or patch tracking.
- Tomographic reconstruction -- computing the 3D volume via weighted back-projection or SIRT.
- Tomogram denoising (CryoCARE, DeepDeWedge) -- deep-learning denoising to improve contrast and signal-to-noise ratio. Denoising can also be applied to facility-imported data, requiring only the raw frames and reconstruction parameters.
Job submission
At submission time the user selects one or more tilt series, fills in the parameter form for the chosen pipeline step (defaults come from the catalog module definition), and picks an execution backend. CryoNAV constructs the appropriate command-line invocation, records the parameter set and command alongside the run for provenance, and submits the job.
Steps can be submitted as a chain: the user selects the sequence of catalog modules to run, and each step is submitted automatically once its upstream dependencies complete. Failed steps halt the rest of the chain; downstream parameters can still be edited later before the chain is resumed.
Local and HPC execution
CryoNAV supports both local workstation execution and HPC cluster submission (via SLURM) within the same deployment, and both modes are available simultaneously: a user might run a quick test alignment locally before submitting a full batch to the cluster. The execution backend is abstracted behind a common interface, so job submission, progress monitoring, log viewing, and error handling work identically regardless of where the job runs.
Progress monitoring
CryoNAV provides real-time progress tracking for running jobs through two complementary strategies, adapted from the approach used by eTomo:
- Log parsing -- processing commands (both IMOD utilities and denoising tools) that print progress messages to stdout (e.g., "Processing frame 15/60" from alignframes) are parsed with regular expressions to extract the current step and total steps, yielding a percentage completion.
- File size monitoring -- for steps that produce output files of predictable size, CryoNAV monitors intermediate file growth and compares it to the expected final size derived from input metadata.
The web interface displays progress bars with smooth interpolation between server updates and human-readable ETA estimates.
Immutable processing branches
Each processing run is stored as an immutable record. Re-running a step with different parameters creates a new parallel branch rather than overwriting the original, allowing side-by-side comparison of parameter choices. See Immutable processing branches for the full discussion.