justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234967.3@dunegpschedd02.fnal.gov

Jobsub ID234967.3@dunegpschedd02.fnal.gov
Workflow ID9455
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:54:48
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_condce_mcore
Last heartbeat2025-10-29 20:48:36
From worker nodeHostnameiut2-c377.iu.edu
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:02:51
Input filesusertests:pdhd_prod_beam__226242_58_1_20251008T182908Z_gen_g4_IonScintPDExt.root_227038_152_1_20251010T165520Z_PDInt.root
JobscriptExit code0
Real time1h (5938s)
CPU time1h (5671s = 95%)
Max RSS bytes6800228352 (6485 MiB)
Outputting started2025-10-29 20:41:50
Output files
Finished2025-10-29 20:48:36
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

40:45.729] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=89 time=0 tick=512 with 55756 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":25641 [0] "wiener0":30115 [30115] ] cmm:[ bad:0 lf_noisy:92 ] input 1: frame: ident=89 time=0 tick=512 with 37978 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16185 [0] "wiener1":21793 [21793] ] cmm:[ bad:0 ] input 2: frame: ident=89 time=0 tick=512 with 69461 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":28676 [0] "wiener2":40785 [40785] ] cmm:[ bad:0 lf_noisy:2 ] input 3: frame: ident=89 time=0 tick=512 with 43473 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":18228 [0] "wiener3":25245 [25245] ] cmm:[ bad:0 ] output: frame: ident=89 time=0 tick=512 with 206668 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":25641 [0] "gauss1":16185 [0] "gauss2":28676 [0] "gauss3":18228 [0] "wiener0":30115 [30115] "wiener1":21793 [21793] "wiener2":40785 [40785] "wiener3":25245 [25245] ] cmm:[ bad:0 lf_noisy:94 ] 
Retagger: tagging trace set: wiener with 117938 traces, 117938 summary
Retagger: tagging trace set: gauss with 88730 traces, 0 summary
[16:40:45.751] D [  glue  ] sink frame: #89 @0 with 206668 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[16:40:45.751] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[16:40:45.751] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[16:40:45.752] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[16:40:45.752] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[16:40:45.752] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[16:40:45.752] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[16:40:45.752] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[16:40:45.752] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[16:40:45.752] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[16:40:45.752] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[16:40:45.752] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[16:40:45.752] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[16:40:45.752] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[16:40:45.752] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[16:40:45.752] D [  glue  ] frame sink sees EOS
[16:40:45.752] D [ pgraph ] <Pgrapher:> graph execution complete 
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.05 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.68 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.45 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.22 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.91 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.78 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.77 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.69 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:40:45.752] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[16:40:45.752] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[16:40:45.752] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:40:45.752] I [ timer  ] Timer: Total node execution : 44.61999971233308 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 88730 traces tagged "gauss"
FrameSaver: q=2.88983e+06 n=1411965 tag=gauss
wclsFrameSaver: saving 117938 traces tagged "wiener"
FrameSaver: q=3.63538e+06 n=1356273 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 49945 XUs and 59061 XVs -> 1396 XUVs
C:0 T:3 180 XUs and 147 XVs -> 2 XUVs
C:0 T:4 66 XUs and 34 XVs -> 7 XUVs
C:0 T:5 88016 XUs and 66366 XVs -> 2134 XUVs
C:0 T:6 60495 XUs and 55842 XVs -> 1319 XUVs
C:0 T:7 1 XUs and 2 XVs -> 0 XUVs
4858 XUVs total
2541 collection wire objects
4858 potential space points
Neighbour search...
104020 tests to find 60600 neighbours
Iterating with no regularization...
Begin: 5.28434e+07
0 4.90746e+07
1 4.8955e+07
2 4.89389e+07
Now with regularization...
Begin: 4.6272e+07
0 4.62641e+07
29-Oct-2025 16:41:49 EDT  Closed output file "pdhd_prod_beam__226242_58_1_20251008T182908Z_gen_g4_IonScintPDExt.root_227038_152_1_20251010T165520Z_PDInt.root_234967_3_1_20251029T190257Z_detsim_reco1.root"
29-Oct-2025 16:41:49 EDT  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           72.234        84.9421       108.61        84.3418       9.48049        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00124794    0.00357417    0.00533729    0.00352356    0.00144032       10     
reco:wclsdatahd:WireCellToolkit                      45.5009       47.2828       49.4651       47.039        1.18929        10     
reco:wirefilter:WireFilter                          0.0176422     0.0199228     0.0230485     0.0192728    0.00184137       10     
reco:gaushit:GausHitFinder                          0.328622      0.547246      0.776574      0.564097       0.11097        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000165298   0.000308167   0.000491303   0.00030626    8.71728e-05      10     
reco:reco3d:SpacePointSolver                        0.237342      0.636858       1.66552      0.438382      0.465359        10     
reco:hitpdune:DisambigFromSpacePoints               0.201245      0.422664      0.914215      0.380166      0.198719        10     
reco:pandora:StandardPandora                         3.08757       7.33543       15.0205       6.74379       3.13679        10     
reco:pandoraWriter:StandardPandora                  0.132038      0.233741      0.353834      0.224642      0.0642992       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.350632      0.965011       1.8063       0.898786      0.377091        10     
reco:pandoraShower:LArPandoraShowerCreation         0.337448      0.830407       2.22706      0.763864      0.500741        10     
reco:pandoracalo:Calorimetry                        0.191388      0.483885      0.907887      0.456713      0.183405        10     
reco:pandoracalonosce:Calorimetry                    0.17656      0.479347      0.857983       0.44761      0.195639        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.337296      0.817339       2.07078      0.708349      0.449368        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.330708      0.795727       2.02858      0.686115      0.443276        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0338358     0.0584753     0.0835103     0.0577827     0.0157815       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0381063     0.0571489     0.0805498     0.0557898     0.0126879       10     
reco:emtrkmichelid:EmTrackMichelId                   15.846        22.5894       31.3514       22.0562       4.13049        10     
[art]:TriggerResults:TriggerResultInserter         2.8984e-05    4.2084e-05    0.000106681   3.32675e-05   2.23468e-05      10     
end_path:out1:RootOutput                            8.226e-06    1.09488e-05   3.0708e-05    8.7615e-06    6.59795e-06      10     
end_path:out1:RootOutput(write)                     0.998715       1.30978       1.7431        1.25956      0.201531        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5765.98 MB
  Peak resident set size usage (VmHWM): 3584.2 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11781628
-rw-r--r--. 1 dune osgvo     231044 Oct 29 16:41 jobscript.log
-rw-r--r--. 1 dune osgvo        201 Oct 29 16:41 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Oct 29 16:41 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Oct 29 16:41 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     139264 Oct 29 16:41 mem.db
-rw-r--r--. 1 dune osgvo      28672 Oct 29 16:41 time.db
-rw-r--r--. 1 dune osgvo 6107188936 Oct 29 16:41 pdhd_prod_beam__226242_58_1_20251008T182908Z_gen_g4_IonScintPDExt.root_227038_152_1_20251010T165520Z_PDInt.root_234967_3_1_20251029T190257Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   34355110 Oct 29 16:41 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Oct 29 16:25 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Oct 29 16:25 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 5922394682 Oct 29 16:25 temp_detsim.root
-rw-r--r--. 1 dune osgvo        122 Oct 29 15:02 all-input-dids.txt
justIN time: 2025-11-03 18:43:32 UTC       justIN version: 01.05.01