justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 260519.2@dunegpschedd02.fnal.gov

Jobsub ID260519.2@dunegpschedd02.fnal.gov
Workflow ID11142
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-10 20:55:21
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-10 23:39:55
From worker nodeHostnamedunegli-7872658-0-fnpc18002.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-10 20:56:44
Input filesusertests:pdhd_prod_beam__248282_53_1_20251113T000319Z_gen_g4_IonScintPDExt.root_258178_42_1_20251205T213722Z_PDInt.root
JobscriptExit code0
Real time2h (9611s)
CPU time2h (8972s = 93%)
Max RSS bytes7333998592 (6994 MiB)
Outputting started2025-12-10 23:36:56
Output filesusertests:pdhd_prod_beam__248282_53_1_20251113T000319Z_gen_g4_IonScintPDExt.root_258178_42_1_20251205T213722Z_PDInt.root_260519_2_1_20251210T205703Z_detsim_reco1.root
Finished2025-12-10 23:39:55
Saved logsjustin-logs:260519.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

with 57390 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":27269 [0] "wiener0":30121 [30121] ] cmm:[ bad:0 lf_noisy:208 ] input 1: frame: ident=139 time=0 tick=512 with 30471 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13554 [0] "wiener1":16917 [16917] ] cmm:[ bad:0 lf_noisy:4 ] input 2: frame: ident=139 time=0 tick=512 with 40510 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":17291 [0] "wiener2":23219 [23219] ] cmm:[ bad:0 lf_noisy:2 ] input 3: frame: ident=139 time=0 tick=512 with 35238 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15554 [0] "wiener3":19684 [19684] ] cmm:[ bad:0 ] output: frame: ident=139 time=0 tick=512 with 163609 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":27269 [0] "gauss1":13554 [0] "gauss2":17291 [0] "gauss3":15554 [0] "wiener0":30121 [30121] "wiener1":16917 [16917] "wiener2":23219 [23219] "wiener3":19684 [19684] ] cmm:[ bad:0 lf_noisy:214 ] 
Retagger: tagging trace set: wiener with 89941 traces, 89941 summary
Retagger: tagging trace set: gauss with 73668 traces, 0 summary
[23:35:44.783] D [  glue  ] sink frame: #139 @0 with 163609 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[23:35:44.784] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[23:35:44.784] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[23:35:44.784] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[23:35:44.784] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[23:35:44.784] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[23:35:44.784] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[23:35:44.784] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[23:35:44.784] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[23:35:44.784] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[23:35:44.784] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[23:35:44.784] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[23:35:44.784] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[23:35:44.784] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[23:35:44.784] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[23:35:44.784] D [  glue  ] frame sink sees EOS
[23:35:44.784] D [ pgraph ] <Pgrapher:> graph execution complete 
[23:35:44.784] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 22.73 sec
[23:35:44.784] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 16.07 sec
[23:35:44.784] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.85 sec
[23:35:44.784] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.74 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 5.46 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 5.27 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.95 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.58 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[23:35:44.785] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[23:35:44.785] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[23:35:44.785] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[23:35:44.785] I [ timer  ] Timer: Total node execution : 88.75999913923442 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 73668 traces tagged "gauss"
FrameSaver: q=2.27838e+06 n=1136678 tag=gauss
wclsFrameSaver: saving 89941 traces tagged "wiener"
FrameSaver: q=2.94167e+06 n=1107912 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 25292 XUs and 19187 XVs -> 1973 XUVs
C:0 T:3 127 XUs and 59 XVs -> 10 XUVs
C:0 T:4 3 XUs and 10 XVs -> 0 XUVs
C:0 T:5 26180 XUs and 16842 XVs -> 419 XUVs
C:0 T:6 16541 XUs and 18646 XVs -> 618 XUVs
C:0 T:7 16 XUs and 18 XVs -> 0 XUVs
3020 XUVs total
1161 collection wire objects
3020 potential space points
Neighbour search...
222450 tests to find 109808 neighbours
Iterating with no regularization...
Begin: 1.71589e+07
0 1.55347e+07
1 1.54186e+07
2 1.5393e+07
3 1.53832e+07
Now with regularization...
Begin: 1.4276e+07
0 1.42692e+07
10-Dec-2025 23:36:50 UTC  Closed output file "pdhd_prod_beam__248282_53_1_20251113T000319Z_gen_g4_IonScintPDExt.root_258178_42_1_20251205T213722Z_PDInt.root_260519_2_1_20251210T205703Z_detsim_reco1.root"
10-Dec-2025 23:36:50 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           118.242       166.907       219.91        164.411       32.5857        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00381144     0.0228691      0.17685     0.00596025     0.0513482       10     
reco:wclsdatahd:WireCellToolkit                      61.6825       83.8955       95.6026       85.2453       8.6762         10     
reco:wirefilter:WireFilter                          0.0424374     0.0604989     0.0744691     0.0602445     0.0109282       10     
reco:gaushit:GausHitFinder                          0.632367      0.964173       1.29322      0.932878      0.241748        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000506078   0.000723518   0.000966618   0.00071571    0.000135737      10     
reco:reco3d:SpacePointSolver                        0.293047       1.15692       2.49639       1.10469       0.77095        10     
reco:hitpdune:DisambigFromSpacePoints                0.13209      0.677625       1.47095      0.500024      0.435941        10     
reco:pandora:StandardPandora                         5.96575       17.7151       45.1465       14.5221       11.359         10     
reco:pandoraWriter:StandardPandora                  0.170585      0.363586      0.504301      0.376075       0.10701        10     
reco:pandoraTrack:LArPandoraTrackCreation            0.77865       2.34761       3.25478       2.64164      0.864652        10     
reco:pandoraShower:LArPandoraShowerCreation         0.428851       2.57552       5.0529        2.21436       1.37981        10     
reco:pandoracalo:Calorimetry                        0.301486       1.18183       2.06104       1.09216      0.561099        10     
reco:pandoracalonosce:Calorimetry                   0.306081       1.07815       1.83042       1.03171      0.486472        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.324863       2.26691       5.17511       1.67481       1.5372         10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.364167       2.31738       5.33861       1.75081       1.56052        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0408721     0.0917954     0.142184      0.0872246     0.0301733       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0360784     0.0910762     0.140804      0.085229      0.0318334       10     
reco:emtrkmichelid:EmTrackMichelId                   24.5347       48.2067       73.9635       46.1935       12.9056        10     
[art]:TriggerResults:TriggerResultInserter         4.9588e-05    8.67443e-05   0.000129348   9.0551e-05    2.53375e-05      10     
end_path:out1:RootOutput                           1.0616e-05    1.63153e-05    3.569e-05    1.25715e-05   8.75757e-06      10     
end_path:out1:RootOutput(write)                      1.25429       1.77403       2.24476       1.8584       0.286087        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6111.51 MB
  Peak resident set size usage (VmHWM): 3961.9 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11692780
-rw-r--r-- 1 dunegli fnalgrid     231578 Dec 10 23:36 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        194 Dec 10 23:36 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 10 23:36 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 10 23:36 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 10 23:36 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 10 23:36 time.db
-rw-r--r-- 1 dunegli fnalgrid 6060995639 Dec 10 23:36 pdhd_prod_beam__248282_53_1_20251113T000319Z_gen_g4_IonScintPDExt.root_258178_42_1_20251205T213722Z_PDInt.root_260519_2_1_20251210T205703Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   33667720 Dec 10 23:35 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 10 23:05 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 10 23:04 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5878287334 Dec 10 23:04 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 10 20:57 all-input-dids.txt
justIN time: 2026-02-07 13:29:34 UTC       justIN version: 01.06.00