https://home.cryonav.org/index.php?action=history&feed=atom&title=CryoNAV_EPU_SerialEM_Folder_StructuresCryoNAV EPU SerialEM Folder Structures - Revision history2026-06-24T19:09:17ZRevision history for this page on the wikiMediaWiki 1.43.1https://home.cryonav.org/index.php?title=CryoNAV_EPU_SerialEM_Folder_Structures&diff=105&oldid=prevAdmin: Bot edit via cryonavedit.py: add reference pages and screenshots2026-05-20T20:57:38Z<p>Bot edit via cryonavedit.py: add reference pages and screenshots</p>
<p><b>New page</b></p><div>Cryo-EM acquisition software organises raw data in two very different ways. EPU (Thermo Fisher) enforces a rigid hierarchy that mirrors the physical grid (atlas -> grid square -> foil hole -> exposure), while SerialEM produces a mostly flat layout with rich text-based metadata. CryoNAV's '''SmartScan''' importer understands both, but you need to know what each acquisition tool writes to disk before pointing it at a session folder.<br />
<br />
== EPU folder structure ==<br />
<br />
EPU (Thermo Fisher) creates a fixed directory tree rooted at the session folder. Every high-magnification acquisition produces six files: summed image (.mrc/.tif), .jpg thumbnail, dose-fractionated movie (<code>_Fractions.mrc</code>, <code>_Fractions.tiff</code>, or <code>_EER.eer</code> on Falcon 4), two XMLs, and a <code>.dm5</code> transfer checksum.<br />
<br />
<pre><br />
&lt;SessionName&gt;/<br />
Metadata/<br />
EpuSession.dm # binary session config<br />
ScreeningSession.dm<br />
Atlas/<br />
Supervisor_&lt;date&gt;_&lt;id&gt;_Atlas/<br />
Tile_###_###_*.mrc<br />
Atlas.mrc *.xml<br />
Images-Disc1/<br />
GridSquare_&lt;8-digit-id&gt;/<br />
GridSquare_*.mrc .xml .jpg<br />
FoilHoles/<br />
FoilHole_&lt;id&gt;_&lt;date&gt;_&lt;time&gt;.mrc .xml<br />
Data/<br />
FoilHole_*_Data_*_*_&lt;date&gt;_&lt;time&gt;.mrc<br />
FoilHole_*_Data_*_*_Fractions.mrc / .tiff / .eer<br />
FoilHole_*_Data_*_*.xml<br />
FoilHole_*_Data_*_*.jpg<br />
FoilHole_*_Data_*_*.dm5<br />
&lt;GainReference&gt;.gain<br />
</pre><br />
<br />
Filenames encode internal (non-sequential) EPU IDs: <code>FoilHole_&lt;HoleID&gt;_Data_&lt;AcqID1&gt;_&lt;AcqID2&gt;_&lt;YYYYMMDD&gt;_&lt;HHMMSS&gt;</code>. Per-image XML holds beam-shift (critical for AFIS grouping), defocus, stage position, pixel size, dose, voltage, and detector mode. Multi-grid sessions (EPU Multigrid, up to 12 cartridges) produce one full session directory per grid.<br />
<br />
== SerialEM folder structure ==<br />
<br />
SerialEM imposes minimal structure; the user controls naming and layout. Metadata lives in '''.mdoc''' files (IMOD autodoc: human-readable, bracketed sections, key/value pairs).<br />
<br />
<pre><br />
&lt;session_root&gt;/<br />
GridName_LMM.st GridName_LMM.st.mdoc # low-mag atlas<br />
GridName_MMM.st GridName_MMM.st.mdoc # medium-mag montage<br />
GridName.mrc GridName.mrc.mdoc # acquired stack<br />
GridName.nav # Navigator<br />
Frames.mdoc # global frame metadata<br />
log.txt SerialEMsettings.txt<br />
frames/<br />
GridName_0001_Dec23_13.20.21.tif<br />
GridName_0001_Dec23_13.20.21.tif.mdoc<br />
</pre><br />
<br />
Key keys inside an .mdoc: <code>TiltAngle</code>, <code>StagePosition</code>, <code>Magnification</code>, <code>Defocus</code>, <code>ImageShift</code>, <code>ExposureDose</code>, <code>PixelSpacing</code>, <code>SubFramePath</code>, <code>NumSubFrames</code>, <code>DateTime</code>; montages also carry <code>PieceCoordinates</code>. The '''.nav''' file stores Navigator items: Type 0 (points), Type 1 (polygons), Type 2 (maps with <code>MapFile</code>, <code>MapID</code>, <code>MapSection</code>, and a scale matrix).<br />
<br />
== What SmartScan looks for ==<br />
<br />
When you point SmartScan at a session root it auto-detects the acquisition software and walks the tree accordingly:<br />
<br />
* '''EPU detection''' -- presence of <code>Metadata/EpuSession.dm</code> or <code>Images-Disc1/GridSquare_*/</code>. SmartScan recursively globs <code>Images-Disc1/GridSquare_*/Data/*_Fractions.{mrc,tiff,eer}</code> for movies, pairs each with its XML sibling, and parses beam-shift to reconstruct AFIS exposure groups. The atlas montage is read from <code>Atlas/Supervisor_*/Atlas.mrc</code>; per-square overviews come from <code>GridSquare_*.mrc</code>.<br />
* '''SerialEM detection''' -- presence of a <code>.nav</code> file or any <code>*.mdoc</code> in the root. SmartScan ingests the Navigator to build the grid map, then reads <code>frames/*.mdoc</code> (or the global <code>Frames.mdoc</code>) for per-tilt parameters. For tilt series, the stack <code>.mrc.mdoc</code> is the primary source of <code>TiltAngle</code>, dose, and <code>SubFramePath</code>.<br />
* '''Gain references''' -- <code>*.gain</code> at the EPU session root, or whatever path is recorded in the SerialEM <code>.mdoc</code>/log.<br />
* '''Multi-grid''' -- both formats are handled per-grid; SmartScan creates one CryoNAV Grid record per detected session or per autoloader-labelled subdirectory.<br />
<br />
Practical tips: do not rename or flatten EPU directories before import -- SmartScan relies on the <code>GridSquare</code> -> <code>FoilHoles</code> -> <code>Data</code> hierarchy to reconstruct grid topology. For SerialEM, keep the <code>frames/</code> directory adjacent to the <code>.nav</code> and <code>.mdoc</code> files so <code>SubFramePath</code> resolves correctly.<br />
<br />
== See also ==<br />
<br />
* [[CryoNAV Tilt Series Data Collection Import|Tilt Series Data Import]]<br />
* [[CryoNAV Metadata Management|Metadata Management]]<br />
<br />
[[Category:Reference]]</div>Admin