justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268896.145@dunegpschedd01.fnal.gov

Jobsub ID268896.145@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 17:53:51
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-17 21:03:24
From worker nodeHostnamedunegli-8100549-0-fnpc9059.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-17 19:31:42
Input filesusertests:pdhd_prod_beam__267379_129_1_20251212T162219Z_gen_g4_IonScintPDExt.root_262726_89_1_20251215T212433Z_PDInt.root
JobscriptExit code0
Real time1h (5491s)
CPU time1h (4800s = 87%)
Max RSS bytes7019008000 (6693 MiB)
Outputting started2025-12-17 21:03:14
Output files
Finished2025-12-17 21:03:24
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 82667 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":38682 [0] "wiener0":43985 [43985] ] cmm:[ bad:0 lf_noisy:156 ] input 1: frame: ident=189 time=0 tick=512 with 45399 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":19874 [0] "wiener1":25525 [25525] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 49674 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":20827 [0] "wiener2":28847 [28847] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 43376 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":18574 [0] "wiener3":24802 [24802] ] cmm:[ bad:0 ] output: frame: ident=189 time=0 tick=512 with 221116 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":38682 [0] "gauss1":19874 [0] "gauss2":20827 [0] "gauss3":18574 [0] "wiener0":43985 [43985] "wiener1":25525 [25525] "wiener2":28847 [28847] "wiener3":24802 [24802] ] cmm:[ bad:0 lf_noisy:156 ] 
Retagger: tagging trace set: wiener with 123159 traces, 123159 summary
Retagger: tagging trace set: gauss with 97957 traces, 0 summary
[21:02:12.906] D [  glue  ] sink frame: #189 @0 with 221116 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[21:02:12.906] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[21:02:12.906] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[21:02:12.906] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[21:02:12.906] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[21:02:12.906] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[21:02:12.906] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[21:02:12.906] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[21:02:12.906] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[21:02:12.906] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[21:02:12.906] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[21:02:12.906] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[21:02:12.906] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[21:02:12.906] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[21:02:12.906] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[21:02:12.906] D [  glue  ] frame sink sees EOS
[21:02:12.906] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.02 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.25 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.24 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.98 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.73 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.68 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.67 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.64 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:02:12.906] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:02:12.906] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:02:12.906] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:02:12.906] I [ timer  ] Timer: Total node execution : 52.29000003822148 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 97957 traces tagged "gauss"
FrameSaver: q=3.27723e+06 n=1637655 tag=gauss
wclsFrameSaver: saving 123159 traces tagged "wiener"
FrameSaver: q=4.10494e+06 n=1557906 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 32722 XUs and 18252 XVs -> 452 XUVs
C:0 T:3 9 XUs and 5 XVs -> 1 XUVs
C:0 T:4 30 XUs and 37 XVs -> 4 XUVs
C:0 T:5 33410 XUs and 43566 XVs -> 840 XUVs
C:0 T:6 25531 XUs and 19301 XVs -> 579 XUVs
1876 XUVs total
1300 collection wire objects
1876 potential space points
Neighbour search...
17838 tests to find 10114 neighbours
Iterating with no regularization...
Begin: 1.49569e+07
0 1.43056e+07
1 1.42657e+07
2 1.42616e+07
Now with regularization...
Begin: 1.32768e+07
0 1.32758e+07
17-Dec-2025 21:03:07 UTC  Closed output file "pdhd_prod_beam__267379_129_1_20251212T162219Z_gen_g4_IonScintPDExt.root_262726_89_1_20251215T212433Z_PDInt.root_268896_145_1_20251217T193146Z_detsim_reco1.root"
17-Dec-2025 21:03:07 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           81.3353       105.862       128.91        102.726       14.6537        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000925889   0.00365416    0.00522527    0.00420261    0.00144589       10     
reco:wclsdatahd:WireCellToolkit                      52.7257       55.3232       59.7022       54.3215       2.15422        10     
reco:wirefilter:WireFilter                          0.0282592     0.035026      0.0467048     0.0345098    0.00553929       10     
reco:gaushit:GausHitFinder                           0.40855      0.683522      0.845145      0.726744      0.116525        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000163396   0.000349156   0.000534059   0.000349203   0.000127298      10     
reco:reco3d:SpacePointSolver                        0.214238      0.594364      0.949028      0.541284      0.255882        10     
reco:hitpdune:DisambigFromSpacePoints               0.305675      0.694978       1.08928       0.65476      0.248958        10     
reco:pandora:StandardPandora                         5.05647       13.8426       31.6401       10.6239       7.42787        10     
reco:pandoraWriter:StandardPandora                  0.129754      0.276761      0.864686       0.20988      0.199643        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.553522       1.51119       2.12464       1.53534      0.446634        10     
reco:pandoraShower:LArPandoraShowerCreation         0.577061       1.41306       2.89604       1.30008      0.579889        10     
reco:pandoracalo:Calorimetry                        0.303831      0.797853       1.20818       0.80476      0.234076        10     
reco:pandoracalonosce:Calorimetry                   0.290448      0.776856       1.23112      0.796277      0.244053        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.481355       1.37378       2.71221       1.29702      0.537156        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.470156       1.30021       2.45756       1.28373      0.481621        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0383864     0.0535544     0.0661222     0.0550704    0.00772913       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0374174     0.0531959      0.06443      0.0532544    0.00746229       10     
reco:emtrkmichelid:EmTrackMichelId                   17.2888       25.8537       33.9798       25.7845       4.26083        10     
[art]:TriggerResults:TriggerResultInserter         2.6426e-05    3.78838e-05    7.892e-05    3.35685e-05   1.43181e-05      10     
end_path:out1:RootOutput                            6.749e-06     8.583e-06    2.2826e-05     6.937e-06    4.75286e-06      10     
end_path:out1:RootOutput(write)                     0.848394       1.22269       1.49214       1.2253       0.177342        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5921.96 MB
  Peak resident set size usage (VmHWM): 3766.96 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11985568
-rw-r--r-- 1 dunegli fnalgrid     231767 Dec 17 21:03 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Dec 17 21:03 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 21:03 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 17 21:03 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 17 21:03 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 17 21:03 time.db
-rw-r--r-- 1 dunegli fnalgrid 6219829245 Dec 17 21:03 pdhd_prod_beam__267379_129_1_20251212T162219Z_gen_g4_IonScintPDExt.root_262726_89_1_20251215T212433Z_PDInt.root_268896_145_1_20251217T193146Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   39573950 Dec 17 21:02 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 17 20:40 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:39 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6013161817 Dec 17 20:39 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Dec 17 19:31 all-input-dids.txt
justIN time: 2025-12-19 04:22:48 UTC       justIN version: 01.05.03