justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263123.0@dunegpschedd02.fnal.gov

Jobsub ID263123.0@dunegpschedd02.fnal.gov
Workflow ID11228
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 20:35:58
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw7
Last heartbeat2025-12-17 23:02:29
From worker nodeHostnamemh-7763-3.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-17 20:37:22
Input filesusertests:pdhd_prod_beam__259199_71_1_20251208T212958Z_gen_g4_IonScintPDExt.root_260263_110_1_20251210T194945Z_PDInt.root
JobscriptExit code0
Real time2h (8497s)
CPU time1h (4270s = 50%)
Max RSS bytes7785312256 (7424 MiB)
Outputting started2025-12-17 22:59:00
Output filesusertests:pdhd_prod_beam__259199_71_1_20251208T212958Z_gen_g4_IonScintPDExt.root_260263_110_1_20251210T194945Z_PDInt.root_263123_0_1_20251217T203727Z_detsim_reco1.root
Finished2025-12-17 23:02:29
Saved logsjustin-logs:263123.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

: ident=169 time=0 tick=512 with 50156 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":23749 [0] "wiener0":26407 [26407] ] cmm:[ bad:0 lf_noisy:229 ] input 1: frame: ident=169 time=0 tick=512 with 36289 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16307 [0] "wiener1":19982 [19982] ] cmm:[ bad:0 ] input 2: frame: ident=169 time=0 tick=512 with 35154 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15138 [0] "wiener2":20016 [20016] ] cmm:[ bad:0 ] input 3: frame: ident=169 time=0 tick=512 with 40906 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":17262 [0] "wiener3":23644 [23644] ] cmm:[ bad:0 ] output: frame: ident=169 time=0 tick=512 with 162505 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23749 [0] "gauss1":16307 [0] "gauss2":15138 [0] "gauss3":17262 [0] "wiener0":26407 [26407] "wiener1":19982 [19982] "wiener2":20016 [20016] "wiener3":23644 [23644] ] cmm:[ bad:0 lf_noisy:229 ] 
Retagger: tagging trace set: wiener with 90049 traces, 90049 summary
Retagger: tagging trace set: gauss with 72456 traces, 0 summary
[14:58:36.912] D [  glue  ] sink frame: #169 @0 with 162505 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[14:58:36.912] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[14:58:36.912] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[14:58:36.912] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[14:58:36.912] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[14:58:36.912] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[14:58:36.912] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[14:58:36.912] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[14:58:36.912] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[14:58:36.912] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[14:58:36.912] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[14:58:36.912] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[14:58:36.912] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[14:58:36.912] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[14:58:36.912] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[14:58:36.912] D [  glue  ] frame sink sees EOS
[14:58:36.912] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.81 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.86 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.86 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.74 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.26 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.24 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.23 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.22 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:58:36.912] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:58:36.912] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[14:58:36.912] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:58:36.912] I [ timer  ] Timer: Total node execution : 35.24000050500035 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72456 traces tagged "gauss"
FrameSaver: q=2.05637e+06 n=1092735 tag=gauss
wclsFrameSaver: saving 90049 traces tagged "wiener"
FrameSaver: q=2.59805e+06 n=1051630 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 18511 XUs and 9664 XVs -> 219 XUVs
C:0 T:3 8 XUs and 6 XVs -> 0 XUVs
C:0 T:4 14 XUs and 9 XVs -> 0 XUVs
C:0 T:5 36668 XUs and 13433 XVs -> 254 XUVs
C:0 T:6 33169 XUs and 48599 XVs -> 875 XUVs
C:0 T:7 0 XUs and 3 XVs -> 0 XUVs
1348 XUVs total
754 collection wire objects
1348 potential space points
Neighbour search...
39230 tests to find 20304 neighbours
Iterating with no regularization...
Begin: 1.27285e+07
0 1.16801e+07
1 1.1649e+07
2 1.16435e+07
Now with regularization...
Begin: 1.12529e+07
0 1.12507e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Dec-2025 14:58:59 PST  Closed output file "pdhd_prod_beam__259199_71_1_20251208T212958Z_gen_g4_IonScintPDExt.root_260263_110_1_20251210T194945Z_PDInt.root_263123_0_1_20251217T203727Z_detsim_reco1.root"
17-Dec-2025 14:58:59 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           54.0181       66.9832       95.1856       62.4555       12.2037        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000975494   0.00260499    0.00475249    0.00249718    0.00120001       10     
reco:wclsdatahd:WireCellToolkit                      36.6189       37.4728       38.5934       37.4191      0.510713        10     
reco:wirefilter:WireFilter                          0.0101161     0.012185      0.0165223     0.011775     0.00184469       10     
reco:gaushit:GausHitFinder                          0.301318      0.458925      0.697709      0.420947      0.126297        10     
reco:nhitsfilter:NumberOfHitsFilter                0.00011088    0.000201792   0.000487177   0.000153754   0.000111582      10     
reco:reco3d:SpacePointSolver                        0.145885      0.328958      0.652775      0.233771      0.184556        10     
reco:hitpdune:DisambigFromSpacePoints               0.123736      0.332108       0.79298      0.324308      0.214079        10     
reco:pandora:StandardPandora                         2.4788        6.25028       14.8135       5.24738       3.46432        10     
reco:pandoraWriter:StandardPandora                  0.0763794      0.12577      0.212987      0.109103      0.0429042       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.25079      0.578453       1.22763      0.488163       0.26793        10     
reco:pandoraShower:LArPandoraShowerCreation         0.271883      0.712299       1.8287       0.571973      0.464412        10     
reco:pandoracalo:Calorimetry                        0.165346      0.369325      0.782241       0.31055      0.167561        10     
reco:pandoracalonosce:Calorimetry                   0.158782      0.357594      0.755857      0.298991      0.162626        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.256714       0.78222       2.1278       0.624887      0.566411        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.252024      0.772727       2.10366      0.618221      0.563483        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0211633     0.0366746     0.0623741     0.0323899     0.0124563       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0205427     0.0361904     0.0658437     0.0321453     0.0130656       10     
reco:emtrkmichelid:EmTrackMichelId                   11.4399       17.4511       29.2014       15.3056       5.50373        10     
[art]:TriggerResults:TriggerResultInserter          1.787e-05    2.91758e-05    7.57e-05     2.35395e-05   1.60774e-05      10     
end_path:out1:RootOutput                            4.75e-06     1.3137e-05     5.856e-05     7.21e-06     1.54417e-05      10     
end_path:out1:RootOutput(write)                     0.584825      0.870422       1.36957      0.793388      0.235748        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6213.46 MB
  Peak resident set size usage (VmHWM): 3944.28 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12054872
-rw-r--r--. 1 cuser cuser     231458 Dec 17 14:59 jobscript.log
-rw-r--r--. 1 cuser cuser        201 Dec 17 14:59 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser        519 Dec 17 14:58 reco_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser       3574 Dec 17 14:58 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser     139264 Dec 17 14:58 mem.db
-rw-r--r--. 1 cuser cuser      28672 Dec 17 14:58 time.db
-rw-r--r--. 1 cuser cuser 6252649367 Dec 17 14:58 pdhd_prod_beam__259199_71_1_20251208T212958Z_gen_g4_IonScintPDExt.root_260263_110_1_20251210T194945Z_PDInt.root_263123_0_1_20251217T203727Z_detsim_reco1.root
-rw-r--r--. 1 cuser cuser   37131730 Dec 17 14:58 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser          0 Dec 17 14:46 debugprod.log
-rw-r--r--. 1 cuser cuser        519 Dec 17 14:46 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser 6053882748 Dec 17 14:46 temp_detsim.root
-rw-r--r--. 1 cuser cuser        122 Dec 17 12:37 all-input-dids.txt
justIN time: 2025-12-18 20:20:52 UTC       justIN version: 01.05.03