justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 260296.1@dunegpschedd02.fnal.gov

Jobsub ID260296.1@dunegpschedd02.fnal.gov
Workflow ID11103
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-10 15:01:04
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-10 20:48:51
From worker nodeHostnamedunegli-8028875-0-fnpc23009.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-10 19:56:47
Input filesusertests:pdhd_prod_beam__247502_9_1_20251112T020214Z_gen_g4_IonScintPDExt.root_263961_87_1_20251205T175601Z_PDInt.root
JobscriptExit code0
Real time51m (3071s)
CPU time50m (3029s = 98%)
Max RSS bytes5958868992 (5682 MiB)
Outputting started2025-12-10 20:47:59
Output filesusertests:pdhd_prod_beam__247502_9_1_20251112T020214Z_gen_g4_IonScintPDExt.root_263961_87_1_20251205T175601Z_PDInt.root_260296_1_1_20251210T195651Z_detsim_reco1.root
Finished2025-12-10 20:48:51
Saved logsjustin-logs:260296.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

> call=18 input 0: frame: ident=109 time=0 tick=512 with 47501 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":22700 [0] "wiener0":24801 [24801] ] cmm:[ bad:0 lf_noisy:178 ] input 1: frame: ident=109 time=0 tick=512 with 37611 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16129 [0] "wiener1":21482 [21482] ] cmm:[ bad:0 ] input 2: frame: ident=109 time=0 tick=512 with 28576 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":13114 [0] "wiener2":15462 [15462] ] cmm:[ bad:0 ] input 3: frame: ident=109 time=0 tick=512 with 35838 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16241 [0] "wiener3":19597 [19597] ] cmm:[ bad:0 ] output: frame: ident=109 time=0 tick=512 with 149526 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":22700 [0] "gauss1":16129 [0] "gauss2":13114 [0] "gauss3":16241 [0] "wiener0":24801 [24801] "wiener1":21482 [21482] "wiener2":15462 [15462] "wiener3":19597 [19597] ] cmm:[ bad:0 lf_noisy:178 ] 
Retagger: tagging trace set: wiener with 81342 traces, 81342 summary
Retagger: tagging trace set: gauss with 68184 traces, 0 summary
[20:47:37.775] D [  glue  ] sink frame: #109 @0 with 149526 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:47:37.775] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[20:47:37.775] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[20:47:37.775] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[20:47:37.775] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[20:47:37.775] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[20:47:37.775] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[20:47:37.775] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[20:47:37.775] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[20:47:37.775] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[20:47:37.775] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[20:47:37.775] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[20:47:37.775] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[20:47:37.775] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[20:47:37.775] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[20:47:37.775] D [  glue  ] frame sink sees EOS
[20:47:37.775] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.11 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.56 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.46 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.44 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.99 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.99 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.97 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.96 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:47:37.775] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:47:37.776] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[20:47:37.776] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:47:37.776] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:47:37.776] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:47:37.776] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[20:47:37.776] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:47:37.776] I [ timer  ] Timer: Total node execution : 32.47999978065491 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 68184 traces tagged "gauss"
FrameSaver: q=2.06222e+06 n=1033011 tag=gauss
wclsFrameSaver: saving 81342 traces tagged "wiener"
FrameSaver: q=2.72739e+06 n=1028009 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 20531 XUs and 18264 XVs -> 912 XUVs
C:0 T:3 4 XUs and 6 XVs -> 2 XUVs
C:0 T:4 2 XUs and 6 XVs -> 0 XUVs
C:0 T:5 17651 XUs and 9358 XVs -> 476 XUVs
C:0 T:6 25222 XUs and 60381 XVs -> 1698 XUVs
C:0 T:7 4 XUs and 4 XVs -> 0 XUVs
3088 XUVs total
1481 collection wire objects
3088 potential space points
Neighbour search...
127794 tests to find 54306 neighbours
Iterating with no regularization...
Begin: 4.25014e+07
0 3.17877e+07
1 3.04695e+07
2 3.04329e+07
3 3.04265e+07
Now with regularization...
Begin: 2.86816e+07
0 2.86745e+07
10-Dec-2025 20:47:58 UTC  Closed output file "pdhd_prod_beam__247502_9_1_20251112T020214Z_gen_g4_IonScintPDExt.root_263961_87_1_20251205T175601Z_PDInt.root_260296_1_1_20251210T195651Z_detsim_reco1.root"
10-Dec-2025 20:47:58 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           44.0639       52.3989       59.379        53.101        4.25363        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000594076   0.00245881     0.0040711     0.0024249    0.00108493       10     
reco:wclsdatahd:WireCellToolkit                      33.2505       34.0175       34.7771       34.0583      0.504981        10     
reco:wirefilter:WireFilter                         0.00784055    0.00951656     0.0112422    0.00977595    0.00115113       10     
reco:gaushit:GausHitFinder                          0.166326      0.304173      0.408157      0.313179      0.0645103       10     
reco:nhitsfilter:NumberOfHitsFilter                7.7017e-05    0.000195589   0.000536536   0.000169855   0.00012177       10     
reco:reco3d:SpacePointSolver                        0.092859      0.196754      0.286019      0.201469       0.05757        10     
reco:hitpdune:DisambigFromSpacePoints               0.0774268     0.167283      0.279425      0.150992      0.0707433       10     
reco:pandora:StandardPandora                         1.2075        3.45179       5.5929        3.14803       1.32539        10     
reco:pandoraWriter:StandardPandora                  0.0480145     0.0893116     0.124358      0.0958743     0.0218905       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.194427       0.37066      0.514184      0.339994      0.103522        10     
reco:pandoraShower:LArPandoraShowerCreation         0.130674      0.280574      0.466505      0.271448      0.0990189       10     
reco:pandoracalo:Calorimetry                        0.115281      0.230228      0.317301      0.205925      0.0630178       10     
reco:pandoracalonosce:Calorimetry                   0.109777      0.219172      0.303548      0.198604      0.058528        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.121259      0.291011      0.517939      0.281825      0.118271        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.120336      0.285979       0.50542      0.276167      0.115119        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0142997     0.0224249     0.0303167     0.0206192    0.00522633       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0141564     0.0217133     0.0285685     0.020325     0.00470742       10     
reco:emtrkmichelid:EmTrackMichelId                   7.25813       11.8024       15.1742       12.2427       2.2173         10     
[art]:TriggerResults:TriggerResultInserter         1.6021e-05    3.09453e-05   9.8399e-05    2.47425e-05   2.28765e-05      10     
end_path:out1:RootOutput                            4.839e-06    1.12836e-05   3.4005e-05    9.0625e-06    7.75266e-06      10     
end_path:out1:RootOutput(write)                     0.447769      0.609134      0.735875      0.607032      0.0823486       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5705.39 MB
  Peak resident set size usage (VmHWM): 3553.21 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10258820
-rw-r--r-- 1 dunegli fnalgrid     231031 Dec 10 20:47 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        199 Dec 10 20:47 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 10 20:47 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 10 20:47 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 10 20:47 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 10 20:47 time.db
-rw-r--r-- 1 dunegli fnalgrid 5317199872 Dec 10 20:47 pdhd_prod_beam__247502_9_1_20251112T020214Z_gen_g4_IonScintPDExt.root_263961_87_1_20251205T175601Z_PDInt.root_260296_1_1_20251210T195651Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   28264300 Dec 10 20:47 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 10 20:38 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 10 20:38 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5159110084 Dec 10 20:38 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Dec 10 19:56 all-input-dids.txt
justIN time: 2026-02-07 16:20:56 UTC       justIN version: 01.06.00