justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.59@dunegpschedd01.fnal.gov

Jobsub ID269119.59@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UChicago
EntryEngage_US_MWT2_uiuc_condce_mcore
Last heartbeat2025-12-18 21:43:04
From worker nodeHostnamemwt2-c071.campuscluster.illinois.edu
cpuinfoAMD EPYC 7443 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:12:00
Input filesusertests:pdhd_prod_beam__267378_29_1_20251212T160741Z_gen_g4_IonScintPDExt.root_262727_83_1_20251215T214920Z_PDInt.root
JobscriptExit code0
Real time1h (5353s)
CPU time1h (5151s = 96%)
Max RSS bytes7123845120 (6793 MiB)
Outputting started2025-12-18 21:41:13
Output filesusertests:pdhd_prod_beam__267378_29_1_20251212T160741Z_gen_g4_IonScintPDExt.root_262727_83_1_20251215T214920Z_PDInt.root_269119_59_1_20251218T201209Z_detsim_reco1.root
Finished2025-12-18 21:43:04
Saved logsjustin-logs:269119.59-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

r0":23370 [23370] ] cmm:[ bad:0 lf_noisy:92 ] 
[15:40:42.431] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 44378 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":21008 [0] "wiener0":23370 [23370] ] cmm:[ bad:0 lf_noisy:92 ] input 1: frame: ident=179 time=0 tick=512 with 28875 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13107 [0] "wiener1":15768 [15768] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 39621 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":16970 [0] "wiener2":22651 [22651] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 37659 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15992 [0] "wiener3":21667 [21667] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 150533 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":21008 [0] "gauss1":13107 [0] "gauss2":16970 [0] "gauss3":15992 [0] "wiener0":23370 [23370] "wiener1":15768 [15768] "wiener2":22651 [22651] "wiener3":21667 [21667] ] cmm:[ bad:0 lf_noisy:92 ] 
Retagger: tagging trace set: wiener with 83456 traces, 83456 summary
Retagger: tagging trace set: gauss with 67077 traces, 0 summary
[15:40:42.442] D [  glue  ] sink frame: #179 @0 with 150533 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[15:40:42.442] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[15:40:42.442] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[15:40:42.443] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[15:40:42.443] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[15:40:42.443] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[15:40:42.443] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[15:40:42.443] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[15:40:42.443] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[15:40:42.443] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[15:40:42.443] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[15:40:42.443] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[15:40:42.443] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[15:40:42.443] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[15:40:42.443] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[15:40:42.443] D [  glue  ] frame sink sees EOS
[15:40:42.443] D [ pgraph ] <Pgrapher:> graph execution complete 
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.55 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.95 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.27 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.13 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.23 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.16 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.66 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.37 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:40:42.443] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[15:40:42.443] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[15:40:42.443] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[15:40:42.443] I [ timer  ] Timer: Total node execution : 48.360000647604465 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 67077 traces tagged "gauss"
FrameSaver: q=1.94258e+06 n=1012609 tag=gauss
wclsFrameSaver: saving 83456 traces tagged "wiener"
FrameSaver: q=2.53834e+06 n=994781 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 6743 XUs and 3669 XVs -> 105 XUVs
C:0 T:4 3 XUs and 1 XVs -> 0 XUVs
C:0 T:5 21801 XUs and 19426 XVs -> 996 XUVs
C:0 T:6 18377 XUs and 23212 XVs -> 2733 XUVs
C:0 T:7 14 XUs and 30 XVs -> 7 XUVs
3841 XUVs total
1311 collection wire objects
3841 potential space points
Neighbour search...
189931 tests to find 114738 neighbours
Iterating with no regularization...
Begin: 1.40163e+07
0 1.27564e+07
1 1.26303e+07
2 1.26067e+07
3 1.25989e+07
Now with regularization...
Begin: 1.12116e+07
0 1.12066e+07
18-Dec-2025 15:41:12 CST  Closed output file "pdhd_prod_beam__267378_29_1_20251212T160741Z_gen_g4_IonScintPDExt.root_262727_83_1_20251215T214920Z_PDInt.root_269119_59_1_20251218T201209Z_detsim_reco1.root"
18-Dec-2025 15:41:12 CST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           74.8517       88.3284       103.684       85.2778       11.1144        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000999866   0.00359157    0.00686504     0.0030789     0.0018375       10     
reco:wclsdatahd:WireCellToolkit                      43.6058       48.7862       52.6205       50.2232       3.10898        10     
reco:wirefilter:WireFilter                          0.0120284     0.0175177     0.0279559     0.0155345    0.00551169       10     
reco:gaushit:GausHitFinder                          0.379086      0.619707      0.782678      0.624504      0.117744        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000162124   0.000267959   0.000402855   0.000248987   6.79439e-05      10     
reco:reco3d:SpacePointSolver                        0.224659      0.483824       1.30426      0.401752      0.312635        10     
reco:hitpdune:DisambigFromSpacePoints               0.194551       0.41591      0.768839      0.349912      0.178823        10     
reco:pandora:StandardPandora                         4.45813       7.75712       13.8778       6.52655       2.8344         10     
reco:pandoraWriter:StandardPandora                  0.163787      0.213324      0.267867      0.202823      0.033335        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.344783      0.814457       1.40905      0.737305      0.285727        10     
reco:pandoraShower:LArPandoraShowerCreation         0.414722      0.775879       1.32709      0.649392      0.304914        10     
reco:pandoracalo:Calorimetry                         0.2894       0.447751       0.63833      0.404711      0.125163        10     
reco:pandoracalonosce:Calorimetry                   0.187627       0.43278      0.800457      0.393335      0.155795        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.350129      0.740263       1.37265      0.685476      0.275306        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.351722      0.731992       1.37826      0.706158      0.272213        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0334125     0.051447      0.0824947     0.0472457     0.015456        10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0294357     0.0476047     0.0775397     0.0447971     0.0145773       10     
reco:emtrkmichelid:EmTrackMichelId                   19.0786       24.6706       35.2015       23.1021       5.28203        10     
[art]:TriggerResults:TriggerResultInserter         2.9225e-05    4.49293e-05   0.000115186   3.7656e-05    2.40519e-05      10     
end_path:out1:RootOutput                            9.267e-06    1.34641e-05    3.702e-05    1.02895e-05   8.04335e-06      10     
end_path:out1:RootOutput(write)                     0.777021       1.24874       1.57889       1.20447      0.230179        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6086.5 MB
  Peak resident set size usage (VmHWM): 3893.05 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11900168
-rw-r--r-- 1 dune osgvo     231247 Dec 18 15:41 jobscript.log
-rw-r--r-- 1 dune osgvo        200 Dec 18 15:41 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Dec 18 15:41 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Dec 18 15:41 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     139264 Dec 18 15:41 mem.db
-rw-r--r-- 1 dune osgvo      28672 Dec 18 15:41 time.db
-rw-r--r-- 1 dune osgvo 6170205251 Dec 18 15:41 pdhd_prod_beam__267378_29_1_20251212T160741Z_gen_g4_IonScintPDExt.root_262727_83_1_20251215T214920Z_PDInt.root_269119_59_1_20251218T201209Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   35057680 Dec 18 15:40 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Dec 18 15:24 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Dec 18 15:24 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 5980056192 Dec 18 15:24 temp_detsim.root
-rw-r--r-- 1 dune osgvo        121 Dec 18 14:12 all-input-dids.txt
justIN time: 2025-12-19 13:26:08 UTC       justIN version: 01.05.03