justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261491.1@dunegpschedd02.fnal.gov

Jobsub ID261491.1@dunegpschedd02.fnal.gov
Workflow ID11129
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-12 04:54:59
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-12 13:11:32
From worker nodeHostnamedunegli-7877340-0-fnpc19125.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-12 11:37:11
Input filesusertests:pdhd_prod_beam__241570_23_1_20251111T230939Z_gen_g4_IonScintPDExt.root_264771_0_1_20251207T234530Z_PDInt.root
JobscriptExit code0
Real time1h (5517s)
CPU time1h (5238s = 94%)
Max RSS bytes7378817024 (7036 MiB)
Outputting started2025-12-12 13:09:09
Output filesusertests:pdhd_prod_beam__241570_23_1_20251111T230939Z_gen_g4_IonScintPDExt.root_264771_0_1_20251207T234530Z_PDInt.root_261491_1_1_20251212T113716Z_detsim_reco1.root
Finished2025-12-12 13:11:32
Saved logsjustin-logs:261491.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":23187 [0] "wiener0":25002 [25002] ] cmm:[ bad:0 lf_noisy:210 ] input 1: frame: ident=119 time=0 tick=512 with 62748 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":27113 [0] "wiener1":35635 [35635] ] cmm:[ bad:0 lf_noisy:27 ] input 2: frame: ident=119 time=0 tick=512 with 31392 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":13746 [0] "wiener2":17646 [17646] ] cmm:[ bad:0 ] input 3: frame: ident=119 time=0 tick=512 with 55174 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":22390 [0] "wiener3":32784 [32784] ] cmm:[ bad:0 lf_noisy:2 ] output: frame: ident=119 time=0 tick=512 with 197503 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23187 [0] "gauss1":27113 [0] "gauss2":13746 [0] "gauss3":22390 [0] "wiener0":25002 [25002] "wiener1":35635 [35635] "wiener2":17646 [17646] "wiener3":32784 [32784] ] cmm:[ bad:0 lf_noisy:239 ] 
Retagger: tagging trace set: wiener with 111067 traces, 111067 summary
Retagger: tagging trace set: gauss with 86436 traces, 0 summary
[13:08:01.391] D [  glue  ] sink frame: #119 @0 with 197503 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[13:08:01.391] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[13:08:01.391] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[13:08:01.391] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[13:08:01.391] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[13:08:01.391] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[13:08:01.391] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[13:08:01.391] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[13:08:01.391] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[13:08:01.391] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[13:08:01.391] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[13:08:01.391] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[13:08:01.391] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[13:08:01.391] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[13:08:01.391] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[13:08:01.391] D [  glue  ] frame sink sees EOS
[13:08:01.391] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:08:01.391] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.4 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.78 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.58 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.96 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.53 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.49 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.48 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.46 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:08:01.392] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:08:01.392] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:08:01.392] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:08:01.392] I [ timer  ] Timer: Total node execution : 43.71999982744455 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 86436 traces tagged "gauss"
FrameSaver: q=3.11776e+06 n=1372504 tag=gauss
wclsFrameSaver: saving 111067 traces tagged "wiener"
FrameSaver: q=3.86365e+06 n=1322199 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 157963 XUs and 173635 XVs -> 13807 XUVs
C:0 T:3 96 XUs and 93 XVs -> 8 XUVs
C:0 T:5 14234 XUs and 18317 XVs -> 607 XUVs
C:0 T:6 41071 XUs and 57025 XVs -> 2044 XUVs
C:0 T:7 89 XUs and 110 XVs -> 33 XUVs
16499 XUVs total
4507 collection wire objects
16499 potential space points
Neighbour search...
4773775 tests to find 1053396 neighbours
Iterating with no regularization...
Begin: 1.3158e+08
0 9.06569e+07
1 8.57793e+07
2 8.49057e+07
3 8.46135e+07
4 8.44813e+07
5 8.44117e+07
Now with regularization...
Begin: 7.15422e+07
0 7.12184e+07
1 7.10999e+07
2 7.10332e+07
12-Dec-2025 13:09:07 UTC  Closed output file "pdhd_prod_beam__241570_23_1_20251111T230939Z_gen_g4_IonScintPDExt.root_264771_0_1_20251207T234530Z_PDInt.root_261491_1_1_20251212T113716Z_detsim_reco1.root"
12-Dec-2025 13:09:07 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           66.9863       93.0792       126.096       89.3314       20.9953        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.0015983    0.00458375    0.00926455    0.00394043    0.00244361       10     
reco:wclsdatahd:WireCellToolkit                      44.9731       51.0175       57.8673       50.7308       3.95023        10     
reco:wirefilter:WireFilter                          0.0178179      0.03564      0.0577421     0.0371498     0.0136081       10     
reco:gaushit:GausHitFinder                           0.30274      0.511109      0.741773      0.490636      0.135836        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000217091   0.000624213   0.00148946    0.000544248   0.000361032      10     
reco:reco3d:SpacePointSolver                        0.162775      0.636169       1.89881      0.429033      0.497982        10     
reco:hitpdune:DisambigFromSpacePoints               0.0989141     0.404609      0.786538      0.369763      0.223809        10     
reco:pandora:StandardPandora                         2.77259       9.87224       22.3208       7.74117       6.21232        10     
reco:pandoraWriter:StandardPandora                   0.12157      0.232085      0.384636      0.226118      0.0783553       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.459949       1.31868       2.36635       1.32293      0.665412        10     
reco:pandoraShower:LArPandoraShowerCreation         0.189147       1.46754       3.50921       1.33808       1.03258        10     
reco:pandoracalo:Calorimetry                         0.24268      0.739498       1.29889      0.734589      0.398133        10     
reco:pandoracalonosce:Calorimetry                   0.220662      0.678389       1.23394      0.699559      0.348663        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.151864       1.33412       2.91923       1.30974      0.910987        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.150901       1.29869       2.75688       1.27499      0.872107        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0266031     0.0519592     0.0745609     0.0546674     0.0160166       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0263637     0.0502104     0.0712621     0.0534803     0.0157102       10     
reco:emtrkmichelid:EmTrackMichelId                   12.0198       22.1972       33.8486       20.3451       7.94596        10     
[art]:TriggerResults:TriggerResultInserter          2.107e-05    4.64693e-05   0.000130541   3.58805e-05   3.14581e-05      10     
end_path:out1:RootOutput                             7.8e-06     1.1325e-05     3.218e-05     8.91e-06     7.00643e-06      10     
end_path:out1:RootOutput(write)                      0.66885       1.12141       1.69448       1.06135      0.303005        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5781.41 MB
  Peak resident set size usage (VmHWM): 3627.24 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11582140
-rw-r--r-- 1 dunegli fnalgrid     236591 Dec 12 13:09 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        181 Dec 12 13:09 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 13:09 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 12 13:09 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 12 13:09 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 12 13:09 time.db
-rw-r--r-- 1 dunegli fnalgrid 6009547916 Dec 12 13:09 pdhd_prod_beam__241570_23_1_20251111T230939Z_gen_g4_IonScintPDExt.root_264771_0_1_20251207T234530Z_PDInt.root_261491_1_1_20251212T113716Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   36692720 Dec 12 13:08 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 12 12:51 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 12:51 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5813402974 Dec 12 12:51 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Dec 12 11:37 all-input-dids.txt
justIN time: 2026-02-07 14:58:55 UTC       justIN version: 01.06.00