justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268896.116@dunegpschedd01.fnal.gov

Jobsub ID268896.116@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 17:53:51
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-17 21:01:40
From worker nodeHostnamedunegli-7943783-0-fnpc8209.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-17 19:12:27
Input filesusertests:pdhd_prod_beam__267379_139_1_20251212T162334Z_gen_g4_IonScintPDExt.root_262726_106_1_20251215T212441Z_PDInt.root
JobscriptExit code0
Real time1h (6536s)
CPU time1h (6212s = 95%)
Max RSS bytes6859890688 (6542 MiB)
Outputting started2025-12-17 21:01:24
Output files
Finished2025-12-17 21:01:40
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 56616 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":26742 [0] "wiener0":29874 [29874] ] cmm:[ bad:0 lf_noisy:206 ] input 1: frame: ident=189 time=0 tick=512 with 31611 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13290 [0] "wiener1":18321 [18321] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 39053 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":16356 [0] "wiener2":22697 [22697] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 40865 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":17069 [0] "wiener3":23796 [23796] ] cmm:[ bad:0 lf_noisy:1 ] output: frame: ident=189 time=0 tick=512 with 168145 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":26742 [0] "gauss1":13290 [0] "gauss2":16356 [0] "gauss3":17069 [0] "wiener0":29874 [29874] "wiener1":18321 [18321] "wiener2":22697 [22697] "wiener3":23796 [23796] ] cmm:[ bad:0 lf_noisy:207 ] 
Retagger: tagging trace set: wiener with 94688 traces, 94688 summary
Retagger: tagging trace set: gauss with 73457 traces, 0 summary
[21:00:22.271] D [  glue  ] sink frame: #189 @0 with 168145 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[21:00:22.271] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[21:00:22.271] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[21:00:22.271] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[21:00:22.271] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[21:00:22.271] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[21:00:22.271] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[21:00:22.271] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[21:00:22.271] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[21:00:22.271] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[21:00:22.271] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[21:00:22.271] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[21:00:22.271] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[21:00:22.271] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[21:00:22.271] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[21:00:22.271] D [  glue  ] frame sink sees EOS
[21:00:22.271] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 24.17 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.67 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.19 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.57 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.86 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.44 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.13 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.11 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[21:00:22.272] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:00:22.272] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:00:22.272] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:00:22.272] I [ timer  ] Timer: Total node execution : 77.22999962605536 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 73457 traces tagged "gauss"
FrameSaver: q=2.24248e+06 n=1099397 tag=gauss
wclsFrameSaver: saving 94688 traces tagged "wiener"
FrameSaver: q=2.90951e+06 n=1073908 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 26602 XUs and 36429 XVs -> 2447 XUVs
C:0 T:3 7 XUs and 16 XVs -> 0 XUVs
C:0 T:5 18900 XUs and 26071 XVs -> 3105 XUVs
C:0 T:6 24687 XUs and 39067 XVs -> 1296 XUVs
6848 XUVs total
1964 collection wire objects
6848 potential space points
Neighbour search...
465138 tests to find 277154 neighbours
Iterating with no regularization...
Begin: 1.83604e+07
0 1.63486e+07
1 1.61557e+07
2 1.61223e+07
3 1.61128e+07
Now with regularization...
Begin: 1.42828e+07
0 1.42765e+07
17-Dec-2025 21:01:19 UTC  Closed output file "pdhd_prod_beam__267379_139_1_20251212T162334Z_gen_g4_IonScintPDExt.root_262726_106_1_20251215T212441Z_PDInt.root_268896_116_1_20251217T191232Z_detsim_reco1.root"
17-Dec-2025 21:01:19 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           113.659       126.178       137.826       128.746       8.77962        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00127968     0.0677235     0.515585     0.00528648      0.15382        10     
reco:wclsdatahd:WireCellToolkit                      62.9722       73.5129       86.6876       70.9883        7.576         10     
reco:wirefilter:WireFilter                          0.024747      0.0358396     0.0486389     0.0359235     0.0062442       10     
reco:gaushit:GausHitFinder                          0.545877      0.845351       1.06289      0.897007      0.163899        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000150407   0.00033012    0.000546503   0.000324501   0.00011965       10     
reco:reco3d:SpacePointSolver                        0.248043      0.768495       1.70924      0.717411       0.40147        10     
reco:hitpdune:DisambigFromSpacePoints               0.232678      0.631213       1.08562      0.600557      0.279322        10     
reco:pandora:StandardPandora                         4.64903       9.8928        16.8165       8.87977       3.37941        10     
reco:pandoraWriter:StandardPandora                   0.14005      0.282555      0.429771      0.271225      0.0888945       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.679055       1.35769       2.0275        1.4027       0.465743        10     
reco:pandoraShower:LArPandoraShowerCreation         0.428274       1.32577       2.41728       1.24663      0.602685        10     
reco:pandoracalo:Calorimetry                        0.357125       0.72942       1.02104      0.753126      0.201978        10     
reco:pandoracalonosce:Calorimetry                   0.360142      0.703674       1.02265      0.714144      0.198013        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.373629       1.34399       2.42482       1.25336      0.604634        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.37591       1.31867       2.39935       1.20936      0.585599        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.036664      0.0691648      0.11238      0.061417      0.0243147       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0373784     0.0699982      0.11454      0.068011      0.024014        10     
reco:emtrkmichelid:EmTrackMichelId                   17.6311       30.7844       37.5696       31.9811       6.57291        10     
[art]:TriggerResults:TriggerResultInserter         3.9388e-05    5.7195e-05    0.000112248   4.8383e-05    2.06254e-05      10     
end_path:out1:RootOutput                            4.978e-06    9.7165e-06     2.506e-05    9.3405e-06    5.64017e-06      10     
end_path:out1:RootOutput(write)                      1.62289       2.38363       3.63363       2.40254      0.567355        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6002.15 MB
  Peak resident set size usage (VmHWM): 3838.21 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11835800
-rw-r--r-- 1 dunegli fnalgrid     231134 Dec 17 21:01 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        202 Dec 17 21:01 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 21:01 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 17 21:01 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 17 21:01 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 17 21:01 time.db
-rw-r--r-- 1 dunegli fnalgrid 6135904049 Dec 17 21:01 pdhd_prod_beam__267379_139_1_20251212T162334Z_gen_g4_IonScintPDExt.root_262726_106_1_20251215T212441Z_PDInt.root_268896_116_1_20251217T191232Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   33642310 Dec 17 21:00 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 17 20:36 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:36 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5949852391 Dec 17 20:36 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        123 Dec 17 19:12 all-input-dids.txt
justIN time: 2025-12-19 04:22:48 UTC       justIN version: 01.05.03