justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.24@dunegpschedd02.fnal.gov

Jobsub ID263305.24@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-18 21:48:11
From worker nodeHostnamedunegli-7951903-0-fnpc23032.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:52:04
Input filesusertests:pdhd_prod_beam__261612_68_1_20251212T155933Z_gen_g4_IonScintPDExt.root_268535_33_1_20251215T214808Z_PDInt.root
JobscriptExit code0
Real time55m (3317s)
CPU time54m (3256s = 98%)
Max RSS bytes6353104896 (6058 MiB)
Outputting started2025-12-18 21:47:22
Output filesusertests:pdhd_prod_beam__261612_68_1_20251212T155933Z_gen_g4_IonScintPDExt.root_268535_33_1_20251215T214808Z_PDInt.root_263305_24_1_20251218T205208Z_detsim_reco1.root
Finished2025-12-18 21:48:11
Saved logsjustin-logs:263305.24-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 33465 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":16129 [0] "wiener0":17336 [17336] ] cmm:[ bad:0 lf_noisy:88 ] input 1: frame: ident=179 time=0 tick=512 with 33394 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":14994 [0] "wiener1":18400 [18400] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 43744 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":18310 [0] "wiener2":25434 [25434] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 25385 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":11355 [0] "wiener3":14030 [14030] ] cmm:[ bad:0 lf_noisy:4 ] output: frame: ident=179 time=0 tick=512 with 135988 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":16129 [0] "gauss1":14994 [0] "gauss2":18310 [0] "gauss3":11355 [0] "wiener0":17336 [17336] "wiener1":18400 [18400] "wiener2":25434 [25434] "wiener3":14030 [14030] ] cmm:[ bad:0 lf_noisy:92 ] 
Retagger: tagging trace set: wiener with 75200 traces, 75200 summary
Retagger: tagging trace set: gauss with 60788 traces, 0 summary
[21:47:05.951] D [  glue  ] sink frame: #179 @0 with 135988 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[21:47:05.951] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[21:47:05.951] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[21:47:05.951] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[21:47:05.951] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[21:47:05.951] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[21:47:05.951] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[21:47:05.951] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[21:47:05.951] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[21:47:05.951] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[21:47:05.951] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[21:47:05.951] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[21:47:05.951] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[21:47:05.951] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[21:47:05.951] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[21:47:05.951] D [  glue  ] frame sink sees EOS
[21:47:05.951] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.6 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.28 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.16 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.01 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.92 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.92 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.92 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.91 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:47:05.951] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[21:47:05.951] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:47:05.951] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:47:05.951] I [ timer  ] Timer: Total node execution : 30.730000028386712 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 60788 traces tagged "gauss"
FrameSaver: q=1.63498e+06 n=902092 tag=gauss
wclsFrameSaver: saving 75200 traces tagged "wiener"
FrameSaver: q=2.01642e+06 n=874595 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 7993 XUs and 11202 XVs -> 646 XUVs
C:0 T:3 4 XUs and 15 XVs -> 1 XUVs
C:0 T:5 22186 XUs and 17916 XVs -> 1069 XUVs
C:0 T:6 3818 XUs and 6137 XVs -> 207 XUVs
C:0 T:7 12 XUs and 9 XVs -> 0 XUVs
1923 XUVs total
951 collection wire objects
1923 potential space points
Neighbour search...
28353 tests to find 17126 neighbours
Iterating with no regularization...
Begin: 2.52171e+07
0 2.40863e+07
1 2.39932e+07
2 2.39865e+07
Now with regularization...
Begin: 2.22799e+07
0 2.22771e+07
18-Dec-2025 21:47:21 UTC  Closed output file "pdhd_prod_beam__261612_68_1_20251212T155933Z_gen_g4_IonScintPDExt.root_268535_33_1_20251215T214808Z_PDInt.root_263305_24_1_20251218T205208Z_detsim_reco1.root"
18-Dec-2025 21:47:21 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           46.7969       53.3077       61.1404       53.3529       4.65095        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000413042   0.00241686    0.00369496    0.00257219    0.00107993       10     
reco:wclsdatahd:WireCellToolkit                      31.3683       31.9326       33.2093       31.8667      0.534997        10     
reco:wirefilter:WireFilter                         0.00746131    0.00946458     0.0112068    0.00941111    0.00116587       10     
reco:gaushit:GausHitFinder                          0.269573      0.336888      0.397563       0.34519      0.0458764       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000100383   0.000173522   0.000376692   0.000161845   7.48538e-05      10     
reco:reco3d:SpacePointSolver                        0.0949537     0.256214      0.571547      0.171011       0.16803        10     
reco:hitpdune:DisambigFromSpacePoints               0.0975993     0.208208      0.330119      0.181625      0.0894036       10     
reco:pandora:StandardPandora                         2.07045       4.33208       6.01829       4.57987       1.21622        10     
reco:pandoraWriter:StandardPandora                  0.0691701     0.094742      0.136011      0.095564      0.0220297       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.320291      0.417555      0.491098       0.42787      0.0631973       10     
reco:pandoraShower:LArPandoraShowerCreation          0.22619      0.410264      0.634478      0.444661      0.147147        10     
reco:pandoracalo:Calorimetry                        0.180137      0.259063      0.322078      0.268089      0.0461681       10     
reco:pandoracalonosce:Calorimetry                   0.171046       0.25247      0.341104      0.257917      0.0484487       10     
reco:pandoraShowercalo:ShowerCalorimetry             0.19151      0.425107      0.686412      0.453674      0.161124        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.18603      0.417144      0.673302      0.447844      0.157809        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0184694     0.0257625     0.0353207     0.0253198    0.00475027       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0179582     0.024759      0.0307533     0.0243794    0.00389615       10     
reco:emtrkmichelid:EmTrackMichelId                   10.1309       13.1884       17.0331       13.0951       2.2848         10     
[art]:TriggerResults:TriggerResultInserter         1.3927e-05    2.40692e-05   9.5553e-05    1.5615e-05    2.39074e-05      10     
end_path:out1:RootOutput                            4.068e-06    8.1377e-06    3.0609e-05    5.1645e-06    7.64907e-06      10     
end_path:out1:RootOutput(write)                     0.502026      0.689939      0.831046      0.678469      0.0942534       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5612.32 MB
  Peak resident set size usage (VmHWM): 3445.06 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11418760
-rw-r--r-- 1 dunegli fnalgrid     231278 Dec 18 21:47 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 18 21:47 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:47 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 21:47 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 21:47 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 21:47 time.db
-rw-r--r-- 1 dunegli fnalgrid 5918787307 Dec 18 21:47 pdhd_prod_beam__261612_68_1_20251212T155933Z_gen_g4_IonScintPDExt.root_268535_33_1_20251215T214808Z_PDInt.root_263305_24_1_20251218T205208Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   32756920 Dec 18 21:47 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 21:37 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:37 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5740803715 Dec 18 21:37 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 18 20:52 all-input-dids.txt
justIN time: 2025-12-18 23:05:56 UTC       justIN version: 01.05.03