justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 233301.0@dunegpschedd01.fnal.gov

Jobsub ID233301.0@dunegpschedd01.fnal.gov
Workflow ID8928
Stage ID1
User namejjo@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-13 13:53:54
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-10-13 17:24:22
From worker nodeHostnamedunegli-7231092-0-fnpc22020.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-13 13:57:11
Input filesmonte-carlo-008928-000001
JobscriptExit code0
Real time3h (12104s)
CPU time3h (11933s = 98%)
Max RSS bytes6926082048 (6605 MiB)
Outputting started2025-10-13 17:18:55
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/jjo/fnal/08928/1/001/monte-carlo-008928-000001_233301_0_1_20251013T135714Z_reco.root
Finished2025-10-13 17:24:22
Saved logsjustin-logs:233301.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

gproc0> call=38 save plane index: 0, Qtot=162142752 Qloss=-16586965, 2874 indices spanning [19738,22611] "gauss" 
[17:18:52.462] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=38 save plane index: 1, Qtot=169004832 Qloss=-24467078, 3582 indices spanning [22612,26193] "wiener" 
[17:18:52.545] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=38 save plane index: 1, Qtot=158068001 Qloss=-18223219, 2853 indices spanning [26194,29046] "gauss" 
[17:18:52.713] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=38 save plane index: 2, Qtot=171983865 Qloss=-9341776, 6142 indices spanning [29047,35188] "wiener" 
[17:18:52.873] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=38 save plane index: 2, Qtot=165971900 Qloss=-3364883, 4613 indices spanning [35189,39801] "gauss" 
[17:18:52.873] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=38 produce 39802 traces: 13371 wiener0, 0 decon_charge0, 10340 gauss0, frame tag: sigproc 
[17:18:52.873] D [sigproc ] <OmnibusSigProc:anode0sigproc0> call=38 output frame: frame: ident=20 time=12 tick=500 with 39802 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss0":10340 [0] "mp2_roi0":13430 [0] "mp3_roi0":2661 [0] "wiener0":13371 [13371] ] cmm:[ ] 
[17:18:52.922] D [  glue  ] <FrameFanin:nfsp> call=38 input 0: frame: ident=20 time=12 tick=500 with 39802 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss0":10340 [0] "mp2_roi0":13430 [0] "mp3_roi0":2661 [0] "wiener0":13371 [13371] ] cmm:[ ] input 1: frame: ident=20 time=12 tick=500 with 8016 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss1":2776 [0] "mp2_roi1":1362 [0] "mp3_roi1":568 [0] "wiener1":3310 [3310] ] cmm:[ ] input 2: frame: ident=20 time=12 tick=500 with 8834 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss2":2828 [0] "mp2_roi2":1583 [0] "mp3_roi2":760 [0] "wiener2":3663 [3663] ] cmm:[ ] input 3: frame: ident=20 time=12 tick=500 with 15685 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss3":4731 [0] "mp2_roi3":4222 [0] "mp3_roi3":1116 [0] "wiener3":5616 [5616] ] cmm:[ ] input 4: frame: ident=20 time=12 tick=500 with 26449 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss4":7467 [0] "mp2_roi4":7104 [0] "mp3_roi4":2303 [0] "wiener4":9575 [9575] ] cmm:[ ] input 5: frame: ident=20 time=12 tick=500 with 13818 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss5":4503 [0] "mp2_roi5":2873 [0] "mp3_roi5":1165 [0] "wiener5":5277 [5277] ] cmm:[ ] input 6: frame: ident=20 time=12 tick=500 with 8922 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss6":2906 [0] "mp2_roi6":1883 [0] "mp3_roi6":579 [0] "wiener6":3554 [3554] ] cmm:[ ] input 7: frame: ident=20 time=12 tick=500 with 23845 traces.  frame tags:[ "sigproc" ] 4 tagged trace sets:[ "gauss7":6994 [0] "mp2_roi7":6664 [0] "mp3_roi7":1876 [0] "wiener7":8311 [8311] ] cmm:[ ] output: frame: ident=20 time=12 tick=500 with 145371 traces.  frame tags:[ "framefanin" ] 16 tagged trace sets:[ "gauss0":10340 [0] "gauss1":2776 [0] "gauss2":2828 [0] "gauss3":4731 [0] "gauss4":7467 [0] "gauss5":4503 [0] "gauss6":2906 [0] "gauss7":6994 [0] "wiener0":13371 [13371] "wiener1":3310 [3310] "wiener2":3663 [3663] "wiener3":5616 [5616] "wiener4":9575 [9575] "wiener5":5277 [5277] "wiener6":3554 [3554] "wiener7":8311 [8311] ] cmm:[ ] 
Retagger: tagging trace set: wiener with 52677 traces, 52677 summary
Retagger: tagging trace set: gauss with 42545 traces, 0 summary
[17:18:52.929] D [  glue  ] sink frame: #20 @1.2e-05 with 145371 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[17:18:52.929] D [  glue  ] <FrameFanout:nfsp> call=39: see EOS 
[17:18:52.929] D [  glue  ] <ChannelSelector:chsel7> see EOS at call=19 
[17:18:52.929] D [sigproc ] <OmnibusSigProc:anode7sigproc7> EOS at call=39 anode=7 
[17:18:52.929] D [  glue  ] <ChannelSelector:chsel6> see EOS at call=19 
[17:18:52.929] D [sigproc ] <OmnibusSigProc:anode6sigproc6> EOS at call=39 anode=6 
[17:18:52.929] D [  glue  ] <ChannelSelector:chsel5> see EOS at call=19 
[17:18:52.929] D [sigproc ] <OmnibusSigProc:anode5sigproc5> EOS at call=39 anode=5 
[17:18:52.929] D [  glue  ] <ChannelSelector:chsel4> see EOS at call=19 
[17:18:52.929] D [sigproc ] <OmnibusSigProc:anode4sigproc4> EOS at call=39 anode=4 
[17:18:52.929] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=19 
[17:18:52.929] D [sigproc ] <OmnibusSigProc:anode3sigproc3> EOS at call=39 anode=3 
[17:18:52.929] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=19 
[17:18:52.929] D [sigproc ] <OmnibusSigProc:anode2sigproc2> EOS at call=39 anode=2 
[17:18:52.929] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=19 
[17:18:52.929] D [sigproc ] <OmnibusSigProc:anode1sigproc1> EOS at call=39 anode=1 
[17:18:52.929] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=19 
[17:18:52.929] D [sigproc ] <OmnibusSigProc:anode0sigproc0> EOS at call=39 anode=0 
[17:18:52.929] D [  glue  ] <FrameFanin:nfsp> EOS at call=39 with 8 
[17:18:52.929] D [  glue  ] frame sink sees EOS
[17:18:52.930] D [ pgraph ] <Pgrapher:> graph execution complete 
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.06 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.74 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.72 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.44 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.4 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.26 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.1 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.1 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:18:52.930] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:18:52.930] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[17:18:52.930] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[17:18:52.930] I [ timer  ] Timer: Total node execution : 27.859999932348728 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 42545 traces tagged "gauss"
FrameSaver: q=9.95074e+06 n=791312 tag=gauss
wclsFrameSaver: saving 52677 traces tagged "wiener"
FrameSaver: q=1.03881e+07 n=753313 tag=wiener
13-Oct-2025 17:18:54 UTC  Closed output file "monte-carlo-008928-000001_233301_0_1_20251013T135714Z_reco.root"
13-Oct-2025 17:18:54 UTC  Closed input file "detsim.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      23.8327       26.8896       29.7975       26.8493       1.37185        20     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000699048    0.0013678    0.00205267    0.00136683    0.000392261      20     
reco:rns:RandomNumberSaver                    2.9827e-05    4.28984e-05   0.000197312    3.424e-05    3.5555e-05       20     
reco:wclsdatavd:WireCellToolkit                 23.0396       26.1141       28.8754       26.1622       1.31446        20     
reco:gaushit:GausHitFinder                     0.380214      0.572407      0.696314      0.589828      0.0984208       20     
[art]:TriggerResults:TriggerResultInserter     1.06e-05     1.66755e-05   4.8141e-05    1.4062e-05    8.03388e-06      20     
end_path:out1:RootOutput                       2.624e-06    4.9887e-06    1.9757e-05    3.3715e-06    3.8789e-06       20     
end_path:out1:RootOutput(write)                0.139567      0.199693      0.247666       0.19776      0.0308216       20     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5379.65 MB
  Peak resident set size usage (VmHWM): 3681.48 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Job completed successfully.
-rw-r--r-- 1 dunegli fnalgrid 3.5G Oct 13 17:08 detsim.root
-rw-r--r-- 1 dunegli fnalgrid  519 Oct 13 17:08 detsim_single_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid  519 Oct 13 14:13 g4_protoDUNE_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5.0G Oct 13 14:07 g4_stage1.root
-rw-r--r-- 1 dunegli fnalgrid  12G Oct 13 14:13 g4_stage2.root
-rw-r--r-- 1 dunegli fnalgrid 964K Oct 13 13:58 gen.root
-rw-r--r-- 1 dunegli fnalgrid  519 Oct 13 13:58 gen_protodunevd_cosmics_hist.root
-rw-r--r-- 1 dunegli fnalgrid 3.6G Oct 13 17:18 monte-carlo-008928-000001_233301_0_1_20251013T135714Z_reco.root
-rw-r--r-- 1 dunegli fnalgrid  519 Oct 13 17:18 reco_protoDUNE_hist.root
justIN time: 2025-11-04 15:53:19 UTC       justIN version: 01.05.01