About CryoNAV - Revision history

Admin: Bot edit via cryonavedit.py: redirect to sidebar-target title

2026-05-20T20:30:05Z

Bot edit via cryonavedit.py: redirect to sidebar-target title

← Older revision		Revision as of 22:30, 20 May 2026
Line 1:		Line 1:
	'''CryoNAV''' is a web-based platform for managing cryo-electron tomography (cryo-ET) data, processing pipelines, and quality assessment. It is designed as an organizational and management layer around established processing tools (IMOD, CryoCARE, DeepDeWedge), rather than as a replacement for them.		#REDIRECT [[CryoNAV About]]

	~~== Overview ==~~

	Cryo-ET generates large, complex datasets that require careful organization, multi-step computational processing, and systematic quality assessment before meaningful biological interpretation can begin. Current workflows typically involve manual file management, command-line scripting for job submission, and serial inspection of results across multiple disconnected tools.

	~~CryoNAV addresses this by:~~

	* Organizing data in a hierarchical model that mirrors the physical microscopy workflow (Project -> Grid -> Search Map -> Tilt Series -> Tomogram); see [[~~CryoEM Data Lifecycle]].~~
	* Integrating cryo-FIB milling records by linking FIB-SEM images to grids and lamellae observed in the TEM.
	* Automating import via SmartScan ([[Tilt Series Data Import]]), which detects acquisition folder structures, parses metadata, maps tilt series to their search map coordinates, and recognizes pre-processed outputs from facility pipelines.
	* Providing template-based job submission to both HPC (via SLURM) and local workstations, with real-time progress tracking ([[Tomogram Processing Overview]]).
	* Supporting deep-learning denoising (CryoCARE, DeepDeWedge) on CryoNAV~~-produced reconstructions or facility-imported tomograms.~~
	* Enabling rapid quality assessment ([[Key Concepts]]) through automatic thumbnail generation, star ratings, tagging, and combined filtering.

	~~CryoNAV is deployed as a single-server application designed for small collaborative research teams (5-10 users) within institutional networks, requiring minimal configuration.~~

	~~== The problems CryoNAV addresses ==~~

	~~Despite the maturity of cryo-ET processing tools, significant practical challenges remain in day-to-day data management:~~

	* '''Data organization.''' Acquisition software generates complex folder hierarchies with naming conventions that vary by platform. Raw data, metadata sidecars, search map images, and processing outputs become scattered across storage locations. Tracking which grid positions have been acquired, processed, or flagged as problematic requires manual bookkeeping that does not scale.
	* '''Disconnected sample preparation records.''' For cryo-FIB workflows, images from the FIB-SEM instrument (e.g., Aquilos) are stored separately from TEM acquisition data, making it difficult to trace data quality issues back to sample preparation conditions.
	* '''Quality assessment bottleneck.''' Evaluating whether a tilt series is worth further processing requires checking ice thickness, beam-induced motion, CTF fit quality, and alignment residuals -- information scattered across different file formats and visualization tools.
	* '''Computational workflow management.''' Chaining multiple command-line programs with precise parameters, writing SLURM batch scripts, and manually tracking job state is bookkeeping-heavy and error-prone.
	* '''Processing provenance.''' Manual or ad hoc scripting leaves no systematic record of which parameters were used for each tilt series, making results difficult to reproduce or revisit.
	* '''Collaboration friction.''' In multi-user microscope facilities, several researchers may share instrument time, storage, and computing resources. Without a centralized system, data ownership is unclear and processing templates are not shared.

	~~== Positioning ==~~

	~~Several existing platforms address aspects of cryo-EM data processing:~~

	* '''Scipion''' (de la Rosa-Trevin et al., 2016) provides a workflow engine integrating multiple processing packages.
	* '''CryoSPARC''' (Punjani et al., 2017) offers a web-based interface with GPU-accelerated algorithms, primarily optimized for single-particle analysis.
	* '''RELION''' (Scheres, 2012) includes its own project management.
	* '''Warp and M''' (Tegunov & Cramer, 2019; Tegunov et al., 2021) provide on-the-fly processing during data collection.
	* '''nextPYP''' (Liu et al., 2023) offers a web-based interface for both single-particle and tomography workflows.

	These tools are primarily designed around their own reconstruction algorithms and tend to treat data organization as secondary to computation. CryoNAV fills the remaining gap: importing heterogeneous data from acquisition software and facility pipelines, systematically deciding which tilt series merit reconstruction, and linking sample-preparation records to downstream results, as first-class concerns in a single lightweight platform.

	~~== Design rationale ==~~

	~~Several architectural decisions reflect deliberate tradeoffs:~~

	* '''Wrap IMOD, do not replace it.''' CryoNAV wraps IMOD's command-line utilities rather than implementing its own processing algorithms, inheriting a mature, well-validated reconstruction pipeline and allowing researchers to use familiar parameter conventions.
	* '''3D visualization via 3dmod.''' Rather than implement a web-based volume viewer, CryoNAV launches IMOD's 3dmod as an external application, preserving its full visualization and modeling capabilities.
	* '''SQLite over client-server database.''' Simplicity and ease of backup outweigh write-concurrency limitations at the target scale (5-10 users). This tradeoff would need revisiting for larger deployments.
	* '''Immutable processing branches.''' Re-running a step with different parameters creates a new branch rather than overwriting the original, enabling side-by-side comparison of parameter choices. Storage cost is mitigated by selective intermediate-file deletion while retaining the full parameter records.
	* '''Single institutional network.''' Deliberately scoped for small collaborative teams: no external cloud dependencies, simple deployment, matched to the operational reality of most cryo-ET laboratories.

	~~== Implementation overview ==~~

	CryoNAV is built as a client-server web application. The backend uses FastAPI (Python) with SQLAlchemy as the ORM and SQLite as the database. The frontend is a React/TypeScript single-page application using TanStack Query for server state management, including adaptive polling for near-real-time job status updates. MRC files are read using the mrcfile Python library for thumbnail extraction and search map overlay generation.

	The platform requires IMOD and the denoising tools CryoCARE and DeepDeWedge to be installed on the execution host(s). Job execution is abstracted behind an Executor interface with SLURM and local subprocess implementations, both of which can be active simultaneously.

	~~See [[Data Storage & Backup]] for storage and backup details and [[Integration with Cryo-EM Tools]] for the external tool integration.~~

	~~== Status and roadmap ==~~

	~~CryoNAV is in active development as a working prototype and is being tested with real cryo-ET datasets. Current limitations and planned work:~~

	* The processing pipeline is centered on the IMOD toolchain with CryoCARE and DeepDeWedge for denoising. Integration with additional packages (e.g., AreTomo for GPU-accelerated alignment) is planned.
	* The SmartScan import engine supports EPU and SerialEM folder structures; additional acquisition-software support could be added through the modular detection framework.
	* Export into RELION .star files is planned, to facilitate handoff to sub-tomogram averaging pipelines.
	* Automated parameter optimization (motion correction, patch tracking) using FRC-based scoring is planned to reduce trial-and-error tuning.
	* Single-server deployment is the current scope; scaling to multi-site facilities would require architectural changes.

	~~== See also ==~~

	* [[Core Concepts Home\|Core Concepts & Workflow]]
	* [[References]]

	~~[[Category:~~About]]

Admin: Bot edit via cryonavedit.py

2026-05-20T20:19:13Z

Bot edit via cryonavedit.py

New page

'''CryoNAV''' is a web-based platform for managing cryo-electron tomography (cryo-ET) data, processing pipelines, and quality assessment. It is designed as an organizational and management layer around established processing tools (IMOD, CryoCARE, DeepDeWedge), rather than as a replacement for them.

== Overview ==

Cryo-ET generates large, complex datasets that require careful organization, multi-step computational processing, and systematic quality assessment before meaningful biological interpretation can begin. Current workflows typically involve manual file management, command-line scripting for job submission, and serial inspection of results across multiple disconnected tools.

CryoNAV addresses this by:

* Organizing data in a hierarchical model that mirrors the physical microscopy workflow (Project -> Grid -> Search Map -> Tilt Series -> Tomogram); see [[CryoEM Data Lifecycle]].
* Integrating cryo-FIB milling records by linking FIB-SEM images to grids and lamellae observed in the TEM.
* Automating import via SmartScan ([[Tilt Series Data Import]]), which detects acquisition folder structures, parses metadata, maps tilt series to their search map coordinates, and recognizes pre-processed outputs from facility pipelines.
* Providing template-based job submission to both HPC (via SLURM) and local workstations, with real-time progress tracking ([[Tomogram Processing Overview]]).
* Supporting deep-learning denoising (CryoCARE, DeepDeWedge) on CryoNAV-produced reconstructions or facility-imported tomograms.
* Enabling rapid quality assessment ([[Key Concepts]]) through automatic thumbnail generation, star ratings, tagging, and combined filtering.

CryoNAV is deployed as a single-server application designed for small collaborative research teams (5-10 users) within institutional networks, requiring minimal configuration.

== The problems CryoNAV addresses ==

Despite the maturity of cryo-ET processing tools, significant practical challenges remain in day-to-day data management:

* '''Data organization.''' Acquisition software generates complex folder hierarchies with naming conventions that vary by platform. Raw data, metadata sidecars, search map images, and processing outputs become scattered across storage locations. Tracking which grid positions have been acquired, processed, or flagged as problematic requires manual bookkeeping that does not scale.
* '''Disconnected sample preparation records.''' For cryo-FIB workflows, images from the FIB-SEM instrument (e.g., Aquilos) are stored separately from TEM acquisition data, making it difficult to trace data quality issues back to sample preparation conditions.
* '''Quality assessment bottleneck.''' Evaluating whether a tilt series is worth further processing requires checking ice thickness, beam-induced motion, CTF fit quality, and alignment residuals -- information scattered across different file formats and visualization tools.
* '''Computational workflow management.''' Chaining multiple command-line programs with precise parameters, writing SLURM batch scripts, and manually tracking job state is bookkeeping-heavy and error-prone.
* '''Processing provenance.''' Manual or ad hoc scripting leaves no systematic record of which parameters were used for each tilt series, making results difficult to reproduce or revisit.
* '''Collaboration friction.''' In multi-user microscope facilities, several researchers may share instrument time, storage, and computing resources. Without a centralized system, data ownership is unclear and processing templates are not shared.

== Positioning ==

Several existing platforms address aspects of cryo-EM data processing:

* '''Scipion''' (de la Rosa-Trevin et al., 2016) provides a workflow engine integrating multiple processing packages.
* '''CryoSPARC''' (Punjani et al., 2017) offers a web-based interface with GPU-accelerated algorithms, primarily optimized for single-particle analysis.
* '''RELION''' (Scheres, 2012) includes its own project management.
* '''Warp and M''' (Tegunov & Cramer, 2019; Tegunov et al., 2021) provide on-the-fly processing during data collection.
* '''nextPYP''' (Liu et al., 2023) offers a web-based interface for both single-particle and tomography workflows.

These tools are primarily designed around their own reconstruction algorithms and tend to treat data organization as secondary to computation. CryoNAV fills the remaining gap: importing heterogeneous data from acquisition software and facility pipelines, systematically deciding which tilt series merit reconstruction, and linking sample-preparation records to downstream results, as first-class concerns in a single lightweight platform.

== Design rationale ==

Several architectural decisions reflect deliberate tradeoffs:

* '''Wrap IMOD, do not replace it.''' CryoNAV wraps IMOD's command-line utilities rather than implementing its own processing algorithms, inheriting a mature, well-validated reconstruction pipeline and allowing researchers to use familiar parameter conventions.
* '''3D visualization via 3dmod.''' Rather than implement a web-based volume viewer, CryoNAV launches IMOD's 3dmod as an external application, preserving its full visualization and modeling capabilities.
* '''SQLite over client-server database.''' Simplicity and ease of backup outweigh write-concurrency limitations at the target scale (5-10 users). This tradeoff would need revisiting for larger deployments.
* '''Immutable processing branches.''' Re-running a step with different parameters creates a new branch rather than overwriting the original, enabling side-by-side comparison of parameter choices. Storage cost is mitigated by selective intermediate-file deletion while retaining the full parameter records.
* '''Single institutional network.''' Deliberately scoped for small collaborative teams: no external cloud dependencies, simple deployment, matched to the operational reality of most cryo-ET laboratories.

== Implementation overview ==

CryoNAV is built as a client-server web application. The backend uses FastAPI (Python) with SQLAlchemy as the ORM and SQLite as the database. The frontend is a React/TypeScript single-page application using TanStack Query for server state management, including adaptive polling for near-real-time job status updates. MRC files are read using the mrcfile Python library for thumbnail extraction and search map overlay generation.

The platform requires IMOD and the denoising tools CryoCARE and DeepDeWedge to be installed on the execution host(s). Job execution is abstracted behind an Executor interface with SLURM and local subprocess implementations, both of which can be active simultaneously.

See [[Data Storage & Backup]] for storage and backup details and [[Integration with Cryo-EM Tools]] for the external tool integration.

== Status and roadmap ==

CryoNAV is in active development as a working prototype and is being tested with real cryo-ET datasets. Current limitations and planned work:

* The processing pipeline is centered on the IMOD toolchain with CryoCARE and DeepDeWedge for denoising. Integration with additional packages (e.g., AreTomo for GPU-accelerated alignment) is planned.
* The SmartScan import engine supports EPU and SerialEM folder structures; additional acquisition-software support could be added through the modular detection framework.
* Export into RELION .star files is planned, to facilitate handoff to sub-tomogram averaging pipelines.
* Automated parameter optimization (motion correction, patch tracking) using FRC-based scoring is planned to reduce trial-and-error tuning.
* Single-server deployment is the current scope; scaling to multi-site facilities would require architectural changes.

== See also ==

* [[Core Concepts Home|Core Concepts & Workflow]]
* [[References]]

[[Category:About]]