Update YZ sim to address large memory usage#905
Open
SFBayLaser wants to merge 11 commits into
Open
Conversation
…is broken into four "daq" modules, one per TPC. These files make that definition, for standard YZ sim and also for overlay
…yz sim uses 4 modules (daq0-daq3) while the standard sim continues to use only one (daq) so stage 0 will need to recognize this
… inherit from the standard one
There was a problem hiding this comment.
Pull request overview
This PR updates the ICARUS WireCell-based YZ simulation configuration to reduce peak memory usage by splitting the YZ TPC simulation into four per-TPC producer modules, while also refactoring the Jsonnet graph generation to avoid expensive g.uses() expansion/dedup behavior.
Changes:
- Add a new per-TPC YZ Jsonnet graph (
wcls-per-tpc-...) driven bytpc_idx, designed to be run by four separateWireCellToolkitmodules. - Refactor the existing multi-TPC YZ Jsonnet to use shared
YZMapnodes and a memoized DFS-basedusesresolver; updatesim.jsonnetto derive YZ pipeline counts fromtools.anodes. - Update detsim and stage0 FHiCL wiring for the per-TPC YZ module labels (
daq0..daq3) and adjust YZ SimChannel merge inputs accordingly.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| icaruscode/TPC/ICARUSWireCell/icarus/wcls-per-tpc-sim-drift-simchannel-yzsim.jsonnet | New per-TPC YZ simulation graph parameterized by tpc_idx to enable 4-module execution. |
| icaruscode/TPC/ICARUSWireCell/icarus/wcls-multitpc-sim-drift-simchannel-yzsim-refactored.jsonnet | Refactors YZ graph pieces (YZMap, sink, uses traversal) and speeds up configuration emission. |
| icaruscode/TPC/ICARUSWireCell/icarus/wcls-multitpc-sim-drift-simchannel-refactored.jsonnet | Replaces g.uses() emission with the same memoized DFS resolver (non-YZ path). |
| icaruscode/TPC/ICARUSWireCell/icarus/sim.jsonnet | Makes YZ transform/reframer/analog pipeline counts depend on nanodes for per-TPC slicing. |
| icaruscode/TPC/ICARUSWireCell/detsimmodules_wirecell_per_TPC_ICARUS.fcl | New detsim module definitions for running YZ as four modules (daq0..daq3) plus overlay variants. |
| icaruscode/TPC/ICARUSWireCell/detsimmodules_wirecell_ICARUS.fcl | Switches the base icarus_simwire_wirecell config to the YZ-sim Jsonnet. |
| fcl/standard/standard_mc_all_stage0_icarus.fcl | Changes the standard stage0 include to the YZ-specific stage0 config. |
| fcl/reco/Stage0/mc/stage0_run2_wcdnn_icarus_mc.fcl | Changes the stage0+DNN entry point to include the YZ-specific stage0 config. |
| fcl/reco/Stage0/mc/stage0_run2_wc_yz_icarus_mc.fcl | New stage0 config variant that overrides decoder fragment labels for per-TPC (daq0..daq3) inputs. |
| fcl/reco/Stage0/mc/stage0_run2_wc_raw_icarus_mc.fcl | Switches the “keep raw digits” stage0 config to include the YZ-specific stage0 config. |
| fcl/reco/Stage0/mc/stage0_run2_wc_icarus_mc_notriggersim.fcl | Switches the “no trigger sim” stage0 config to include the YZ-specific stage0 config. |
| fcl/g4/mergesimsources_icarus.fcl | Updates YZ SimChannel merge input labels to use daq0..daq3:simpleSC*. |
| fcl/detsim/detsim_2d_icarus_refactored.fcl | Includes the new per-TPC WireCell detsim module definitions (non-YZ still uses single daq). |
| fcl/detsim/detsim_2d_icarus_refactored_yzsim.fcl | Rewires YZ detsim to run daq0..daq3 instead of a single daq producer. |
| fcl/detsim/detsim_2d_icarus_refactored_yzsim_overlay.fcl | Updates overlay YZ detsim overrides to target daq0..daq3 producers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+24
to
+31
| local g = import 'pgraph.jsonnet'; | ||
| local f = import 'pgrapher/common/funcs.jsonnet'; | ||
| local wc = import 'wirecell.jsonnet'; | ||
|
|
||
| local io = import 'pgrapher/common/fileio.jsonnet'; | ||
| local tools_maker = import 'pgrapher/experiment/icarus/icarus_tools.jsonnet'; | ||
| local base = import 'pgrapher/experiment/icarus/simparams.jsonnet'; | ||
|
|
Comment on lines
+83
to
+87
| local tpc_idx = std.parseInt(std.extVar('tpc_idx')); | ||
| local apa_lo = tpc_idx * 2; // 2 anodes per TPC | ||
| local local_iota = std.range(0, 89); // 90 per-anode-plane-response branches | ||
| local volname = ["EE", "EW", "WE", "WW"]; | ||
|
|
| // product instances match the single-module reference run. | ||
| // --------------------------------------------------------------------------- | ||
| local overlay_drifter = std.extVar("overlay_drifter"); | ||
| local localeLiftime = [std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us,std.extVar('lifetime') * wc.us]; |
Comment on lines
+213
to
+217
| structs: { | ||
| DL: 4e-9 | ||
| DT: 8.8e-9 | ||
| coh_noise_scale: 1 | ||
| gain0: 1.705212e1 |
Comment on lines
+360
to
+364
| structs: { | ||
| DL: 4e-9 | ||
| DT: 8.8e-9 | ||
| coh_noise_scale: 1 | ||
| gain0: 1.705212e1 |
| @@ -1 +1 @@ | |||
| #include "stage0_run2_icarus_mc.fcl" | |||
| #include "stage0_run2_wc_yz_icarus_mc.fcl" | |||
| @@ -1,3 +1,3 @@ | |||
| #include "stage0_run2_wc_icarus_mc.fcl" | |||
| #include "stage0_run2_wc_yz_icarus_mc.fcl" | |||
| ## This fhicl file takes the standard stage 0 wc mc file and keeps the raw digits | ||
| ## | ||
| #include "stage0_run2_wc_icarus_mc.fcl" | ||
| #include "stage0_run2_wc_yz_icarus_mc.fcl" |
| @@ -1,4 +1,4 @@ | |||
| #include "stage0_run2_wc_icarus_mc.fcl" | |||
| #include "stage0_run2_wc_yz_icarus_mc.fcl" | |||
| plugins: ["WireCellPgraph", "WireCellGen","WireCellSio","WireCellRoot","WireCellLarsoft", "WireCellHio"] | ||
| // needs to be found via your WIRECELL_PATH | ||
| configs: ["pgrapher/experiment/icarus/wcls-multitpc-sim-drift-simchannel-refactored.jsonnet"] | ||
| configs: ["pgrapher/experiment/icarus/wcls-multitpc-sim-drift-simchannel-yzsim-refactored.jsonnet"] |
Contributor
Author
|
The above PR is compatible with larsoft v10_20_09 and later. It also requires icarus_data v10_20_03. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR provides the necessary updates to utilize improvements in the wirecell toolkit to address the ICARUS YZ simulation's large memory usage. While the main changes are in the toolkit itself we have modified the detector simulation at the ICARUS level to break the TPC simulation into four separate producer modules. At the same time we are maintaining the non-YZ simulation to a single producer module. This requires updates at the stage 0 processing level as well for the two cases.
At this point the comparison of non-YZ (top) to YZ (bottom) cpu and memory usage is:
Generally, we are now running the YZ sim with more than a 2x reduction in memory usage