justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263108.0@dunegpschedd02.fnal.gov

Jobsub ID263108.0@dunegpschedd02.fnal.gov
Workflow ID11225
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 19:03:53
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_gk02_condce_mcore
Last heartbeat2025-12-17 20:50:51
From worker nodeHostnamemwt2-c097.campuscluster.illinois.edu
cpuinfoAMD EPYC 7443 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-17 19:13:20
Input filesusertests:pdhd_prod_beam__265003_98_1_20251208T181205Z_gen_g4_IonScintPDExt.root_265463_5_1_20251209T161324Z_PDInt.root
JobscriptExit code0
Real time1h (5830s)
CPU time1h (5215s = 89%)
Max RSS bytes6833815552 (6517 MiB)
Outputting started2025-12-17 20:50:30
Output files
Finished2025-12-17 20:50:51
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

=129 time=0 tick=512 with 53491 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":25308 [0] "wiener0":28183 [28183] ] cmm:[ bad:0 lf_noisy:280 ] 
[14:50:05.380] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=129 time=0 tick=512 with 53491 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":25308 [0] "wiener0":28183 [28183] ] cmm:[ bad:0 lf_noisy:280 ] input 1: frame: ident=129 time=0 tick=512 with 42615 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":19519 [0] "wiener1":23096 [23096] ] cmm:[ bad:0 ] input 2: frame: ident=129 time=0 tick=512 with 27631 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":12795 [0] "wiener2":14836 [14836] ] cmm:[ bad:0 ] input 3: frame: ident=129 time=0 tick=512 with 24199 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":11113 [0] "wiener3":13086 [13086] ] cmm:[ bad:0 ] output: frame: ident=129 time=0 tick=512 with 147936 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":25308 [0] "gauss1":19519 [0] "gauss2":12795 [0] "gauss3":11113 [0] "wiener0":28183 [28183] "wiener1":23096 [23096] "wiener2":14836 [14836] "wiener3":13086 [13086] ] cmm:[ bad:0 lf_noisy:280 ] 
Retagger: tagging trace set: wiener with 79201 traces, 79201 summary
Retagger: tagging trace set: gauss with 68735 traces, 0 summary
[14:50:05.391] D [  glue  ] sink frame: #129 @0 with 147936 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[14:50:05.391] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[14:50:05.391] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[14:50:05.391] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[14:50:05.391] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[14:50:05.391] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[14:50:05.391] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[14:50:05.391] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[14:50:05.391] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[14:50:05.391] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[14:50:05.391] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[14:50:05.391] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[14:50:05.391] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[14:50:05.391] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[14:50:05.391] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[14:50:05.391] D [  glue  ] frame sink sees EOS
[14:50:05.391] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.09 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.18 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.8 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.64 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.77 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.69 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.68 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.48 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:50:05.391] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[14:50:05.391] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:50:05.391] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:50:05.391] I [ timer  ] Timer: Total node execution : 42.36000016145408 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 68735 traces tagged "gauss"
FrameSaver: q=1.95197e+06 n=1114709 tag=gauss
wclsFrameSaver: saving 79201 traces tagged "wiener"
FrameSaver: q=2.42551e+06 n=1091671 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 24643 XUs and 17805 XVs -> 610 XUVs
C:0 T:5 6458 XUs and 6837 XVs -> 203 XUVs
C:0 T:6 5010 XUs and 4740 XVs -> 192 XUVs
1005 XUVs total
866 collection wire objects
1005 potential space points
Neighbour search...
6849 tests to find 3414 neighbours
Iterating with no regularization...
Begin: 1.24321e+07
0 1.21892e+07
1 1.21865e+07
Now with regularization...
Begin: 1.15462e+07
0 1.15461e+07
17-Dec-2025 14:50:29 CST  Closed output file "pdhd_prod_beam__265003_98_1_20251208T181205Z_gen_g4_IonScintPDExt.root_265463_5_1_20251209T161324Z_PDInt.root_263108_0_1_20251217T191531Z_detsim_reco1.root"
17-Dec-2025 14:50:29 CST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           67.0693       82.807        92.5291       83.4285       7.43386        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000849263    0.0095697     0.0653338     0.0037246     0.0186454       10     
reco:wclsdatahd:WireCellToolkit                      43.0299       45.5557       47.3028       45.6858       1.28164        10     
reco:wirefilter:WireFilter                          0.0126915     0.0185672     0.024568      0.0180208    0.00366591       10     
reco:gaushit:GausHitFinder                          0.404932      0.671284      0.806925      0.690563      0.122252        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000192524   0.000267623   0.000506807   0.000245193   8.50942e-05      10     
reco:reco3d:SpacePointSolver                        0.146367      0.509992       1.15934      0.514287      0.267435        10     
reco:hitpdune:DisambigFromSpacePoints               0.199643      0.480411      0.871726      0.434767      0.183373        10     
reco:pandora:StandardPandora                         3.54029       8.11197       13.3597       7.99433       3.03957        10     
reco:pandoraWriter:StandardPandora                  0.0928781     0.221203      0.305918      0.230588      0.0538003       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.258897      0.778548       1.06639      0.871444      0.233957        10     
reco:pandoraShower:LArPandoraShowerCreation         0.251918       0.74108       1.06699      0.808787      0.214849        10     
reco:pandoracalo:Calorimetry                         0.16013      0.450908      0.573256      0.500698      0.131108        10     
reco:pandoracalonosce:Calorimetry                   0.156328      0.415917      0.547982      0.436172      0.126947        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.289352      0.768613       1.18553      0.807042      0.234772        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.229743      0.744011       1.26401      0.787911      0.265104        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0203379     0.050754      0.0811119      0.0502       0.0191947       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0204985     0.0506651     0.0783956     0.0493376     0.0176307       10     
reco:emtrkmichelid:EmTrackMichelId                   15.7546       21.9555       26.6961       21.8882       2.93626        10     
[art]:TriggerResults:TriggerResultInserter         2.5348e-05    4.22238e-05   0.000115057   3.5883e-05    2.45507e-05      10     
end_path:out1:RootOutput                            7.434e-06    1.26308e-05   3.9495e-05    1.0299e-05    9.04086e-06      10     
end_path:out1:RootOutput(write)                     0.959928       1.21554       1.48991       1.22504      0.159551        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5769 MB
  Peak resident set size usage (VmHWM): 3589.44 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12193476
-rw-r--r-- 1 dune osgvo     231061 Dec 17 14:50 jobscript.log
-rw-r--r-- 1 dune osgvo        193 Dec 17 14:50 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Dec 17 14:50 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Dec 17 14:50 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     139264 Dec 17 14:50 mem.db
-rw-r--r-- 1 dune osgvo      28672 Dec 17 14:50 time.db
-rw-r--r-- 1 dune osgvo 6320209965 Dec 17 14:50 pdhd_prod_beam__265003_98_1_20251208T181205Z_gen_g4_IonScintPDExt.root_265463_5_1_20251209T161324Z_PDInt.root_263108_0_1_20251217T191531Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   35042720 Dec 17 14:50 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Dec 17 14:35 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Dec 17 14:34 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 6130417537 Dec 17 14:34 temp_detsim.root
-rw-r--r-- 1 dune osgvo        120 Dec 17 13:15 all-input-dids.txt
justIN time: 2025-12-18 20:20:51 UTC       justIN version: 01.05.03