justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.14@dunegpschedd01.fnal.gov

Jobsub ID269119.14@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-18 20:27:50
From worker nodeHostnamedunegli-8107756-0-fnpc23017.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 19:30:50
Input filesusertests:pdhd_prod_beam__261612_30_1_20251212T155553Z_gen_g4_IonScintPDExt.root_262727_27_1_20251215T214419Z_PDInt.root
JobscriptExit code0
Real time56m (3374s)
CPU time55m (3310s = 98%)
Max RSS bytes7000387584 (6676 MiB)
Outputting started2025-12-18 20:27:04
Output filesusertests:pdhd_prod_beam__261612_30_1_20251212T155553Z_gen_g4_IonScintPDExt.root_262727_27_1_20251215T214419Z_PDInt.root_269119_14_1_20251218T193053Z_detsim_reco1.root
Finished2025-12-18 20:27:50
Saved logsjustin-logs:269119.14-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 49735 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":23498 [0] "wiener0":26237 [26237] ] cmm:[ bad:0 lf_noisy:241 ] input 1: frame: ident=179 time=0 tick=512 with 58701 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":23994 [0] "wiener1":34707 [34707] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 28098 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":13041 [0] "wiener2":15057 [15057] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 52755 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":21233 [0] "wiener3":31522 [31522] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 189289 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23498 [0] "gauss1":23994 [0] "gauss2":13041 [0] "gauss3":21233 [0] "wiener0":26237 [26237] "wiener1":34707 [34707] "wiener2":15057 [15057] "wiener3":31522 [31522] ] cmm:[ bad:0 lf_noisy:241 ] 
Retagger: tagging trace set: wiener with 107523 traces, 107523 summary
Retagger: tagging trace set: gauss with 81766 traces, 0 summary
[20:26:36.781] D [  glue  ] sink frame: #179 @0 with 189289 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:26:36.781] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[20:26:36.781] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[20:26:36.781] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[20:26:36.781] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[20:26:36.781] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[20:26:36.781] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[20:26:36.781] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[20:26:36.781] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[20:26:36.781] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[20:26:36.781] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[20:26:36.781] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[20:26:36.781] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[20:26:36.781] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[20:26:36.781] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[20:26:36.781] D [  glue  ] frame sink sees EOS
[20:26:36.781] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.6 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.42 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.21 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.95 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.93 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.92 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.91 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.89 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:26:36.781] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[20:26:36.781] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:26:36.781] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:26:36.781] I [ timer  ] Timer: Total node execution : 30.839999685063958 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 81766 traces tagged "gauss"
FrameSaver: q=2.71513e+06 n=1271999 tag=gauss
wclsFrameSaver: saving 107523 traces tagged "wiener"
FrameSaver: q=3.31144e+06 n=1219753 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 51866 XUs and 48582 XVs -> 1677 XUVs
C:0 T:3 3 XUs and 4 XVs -> 0 XUVs
C:0 T:5 6298 XUs and 4745 XVs -> 174 XUVs
C:0 T:6 89635 XUs and 82838 XVs -> 3169 XUVs
C:0 T:7 26 XUs and 50 XVs -> 7 XUVs
5027 XUVs total
1689 collection wire objects
5027 potential space points
Neighbour search...
226619 tests to find 116456 neighbours
Iterating with no regularization...
Begin: 4.40386e+07
0 3.87824e+07
1 3.84794e+07
2 3.84432e+07
Now with regularization...
Begin: 3.5677e+07
0 3.56661e+07
18-Dec-2025 20:27:04 UTC  Closed output file "pdhd_prod_beam__261612_30_1_20251212T155553Z_gen_g4_IonScintPDExt.root_262727_27_1_20251215T214419Z_PDInt.root_269119_14_1_20251218T193053Z_detsim_reco1.root"
18-Dec-2025 20:27:04 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           44.0138       53.0374       59.8526       53.3231       4.52361        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00039757    0.00227045    0.00403571    0.00241215     0.0010485       10     
reco:wclsdatahd:WireCellToolkit                      30.4669       31.2231       31.7935       31.2432      0.389529        10     
reco:wirefilter:WireFilter                          0.0077405    0.00901878     0.0109276     0.0088866    0.000985675      10     
reco:gaushit:GausHitFinder                          0.232855      0.359644      0.485049      0.365132      0.0637994       10     
reco:nhitsfilter:NumberOfHitsFilter                8.0194e-05    0.000150467   0.000406077   0.000127063   8.68982e-05      10     
reco:reco3d:SpacePointSolver                        0.0946171     0.307247      0.514372      0.302711      0.136499        10     
reco:hitpdune:DisambigFromSpacePoints               0.0966366     0.215554      0.383475      0.178375      0.0808874       10     
reco:pandora:StandardPandora                         2.37679       4.27145       5.66178       4.2058        1.21653        10     
reco:pandoraWriter:StandardPandora                  0.0795599     0.0998082     0.137699      0.096146      0.0167635       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.215654      0.390345      0.538917      0.414054      0.0971875       10     
reco:pandoraShower:LArPandoraShowerCreation         0.201078      0.396977      0.530009      0.408856      0.104616        10     
reco:pandoracalo:Calorimetry                        0.136592      0.249575      0.334326      0.259611      0.0554132       10     
reco:pandoracalonosce:Calorimetry                   0.131508      0.239319      0.320796      0.249581      0.0530535       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.204943       0.42905      0.572427      0.437058      0.111222        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.203384      0.420402      0.538677      0.430747      0.107159        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.019482      0.0243936     0.0334146     0.0236508    0.00408851       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0187261     0.0237319     0.0288448     0.0232915    0.00327852       10     
reco:emtrkmichelid:EmTrackMichelId                   8.99996       13.6766       17.7528       13.4501       2.29851        10     
[art]:TriggerResults:TriggerResultInserter         1.4688e-05    2.64276e-05   9.3149e-05    1.9542e-05    2.23252e-05      10     
end_path:out1:RootOutput                            9.218e-06    1.21064e-05   3.5528e-05    9.5185e-06    7.81038e-06      10     
end_path:out1:RootOutput(write)                     0.504977      0.682921      0.813682      0.699399      0.0877824       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5980.93 MB
  Peak resident set size usage (VmHWM): 3808.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11629584
-rw-r--r-- 1 dunegli fnalgrid     231144 Dec 18 20:27 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 18 20:27 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 20:27 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 20:27 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 20:27 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 20:27 time.db
-rw-r--r-- 1 dunegli fnalgrid 6028913243 Dec 18 20:27 pdhd_prod_beam__261612_30_1_20251212T155553Z_gen_g4_IonScintPDExt.root_262727_27_1_20251215T214419Z_PDInt.root_269119_14_1_20251218T193053Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   34226520 Dec 18 20:26 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 20:17 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 20:17 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5844896911 Dec 18 20:17 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 18 19:30 all-input-dids.txt
justIN time: 2025-12-19 04:23:38 UTC       justIN version: 01.05.03