justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.31@dunegpschedd02.fnal.gov

Jobsub ID263305.31@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_gk02_condce_mcore
Last heartbeat2025-12-18 22:26:57
From worker nodeHostnameuct2-c594.mwt2.org
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 21:01:25
Input filesusertests:pdhd_prod_beam__261612_43_1_20251212T155655Z_gen_g4_IonScintPDExt.root_268566_0_1_20251215T230702Z_PDInt.root
JobscriptExit code0
Real time1h (5025s)
CPU time1h (4832s = 96%)
Max RSS bytes6432251904 (6134 MiB)
Outputting started2025-12-18 22:25:10
Output filesusertests:pdhd_prod_beam__261612_43_1_20251212T155655Z_gen_g4_IonScintPDExt.root_268566_0_1_20251215T230702Z_PDInt.root_263305_31_1_20251218T210135Z_detsim_reco1.root
Finished2025-12-18 22:26:57
Saved logsjustin-logs:263305.31-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

:[ bad:0 lf_noisy:198 ] 
[16:24:31.075] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 73689 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":34622 [0] "wiener0":39067 [39067] ] cmm:[ bad:0 lf_noisy:198 ] input 1: frame: ident=179 time=0 tick=512 with 31713 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13955 [0] "wiener1":17758 [17758] ] cmm:[ bad:0 lf_noisy:3 ] input 2: frame: ident=179 time=0 tick=512 with 45228 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":19798 [0] "wiener2":25430 [25430] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 45525 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":19356 [0] "wiener3":26169 [26169] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 196155 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":34622 [0] "gauss1":13955 [0] "gauss2":19798 [0] "gauss3":19356 [0] "wiener0":39067 [39067] "wiener1":17758 [17758] "wiener2":25430 [25430] "wiener3":26169 [26169] ] cmm:[ bad:0 lf_noisy:201 ] 
Retagger: tagging trace set: wiener with 108424 traces, 108424 summary
Retagger: tagging trace set: gauss with 87731 traces, 0 summary
[16:24:31.092] D [  glue  ] sink frame: #179 @0 with 196155 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[16:24:31.093] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[16:24:31.093] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[16:24:31.093] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[16:24:31.093] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[16:24:31.093] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[16:24:31.093] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[16:24:31.093] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[16:24:31.093] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[16:24:31.093] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[16:24:31.093] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[16:24:31.093] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[16:24:31.093] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[16:24:31.093] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[16:24:31.093] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[16:24:31.093] D [  glue  ] frame sink sees EOS
[16:24:31.093] D [ pgraph ] <Pgrapher:> graph execution complete 
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.17 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.23 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.12 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.97 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.95 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.87 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.73 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.71 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:24:31.093] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:24:31.093] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[16:24:31.093] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:24:31.093] I [ timer  ] Timer: Total node execution : 47.80999928340316 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 87731 traces tagged "gauss"
FrameSaver: q=2.89571e+06 n=1445816 tag=gauss
wclsFrameSaver: saving 108424 traces tagged "wiener"
FrameSaver: q=3.84497e+06 n=1420792 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 21007 XUs and 24552 XVs -> 422 XUVs
C:0 T:3 2 XUs and 6 XVs -> 0 XUVs
C:0 T:5 29800 XUs and 39559 XVs -> 878 XUVs
C:0 T:6 42688 XUs and 32553 XVs -> 1753 XUVs
C:0 T:7 52 XUs and 115 XVs -> 38 XUVs
3091 XUVs total
1404 collection wire objects
3091 potential space points
Neighbour search...
138057 tests to find 83088 neighbours
Iterating with no regularization...
Begin: 2.58636e+07
0 2.50462e+07
1 2.49768e+07
2 2.49663e+07
Now with regularization...
Begin: 2.35763e+07
0 2.35718e+07
18-Dec-2025 16:25:09 CST  Closed output file "pdhd_prod_beam__261612_43_1_20251212T155655Z_gen_g4_IonScintPDExt.root_268566_0_1_20251215T230702Z_PDInt.root_263305_31_1_20251218T210135Z_detsim_reco1.root"
18-Dec-2025 16:25:09 CST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           64.6407       75.658        86.907        76.084        6.90925        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00112139    0.00328836    0.00708876    0.00345159    0.00157975       10     
reco:wclsdatahd:WireCellToolkit                      42.5979       46.5461       49.8216       46.7433       2.1159         10     
reco:wirefilter:WireFilter                            0.011       0.0145871     0.0194594     0.0150639    0.00266482       10     
reco:gaushit:GausHitFinder                          0.322839      0.507842      0.794634      0.471874      0.118733        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000117661   0.000208895   0.000426083   0.000206244   8.22547e-05      10     
reco:reco3d:SpacePointSolver                        0.112078      0.296412       0.56293      0.280776      0.144355        10     
reco:hitpdune:DisambigFromSpacePoints               0.115385      0.277326      0.541317      0.236834      0.131423        10     
reco:pandora:StandardPandora                         2.95971       5.23621       7.23478       5.49985       1.40698        10     
reco:pandoraWriter:StandardPandora                  0.109516      0.151317      0.195771      0.153182      0.025863        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.268053      0.633042       1.14754      0.661347      0.262324        10     
reco:pandoraShower:LArPandoraShowerCreation          0.21115      0.515262      0.907813      0.482306      0.216849        10     
reco:pandoracalo:Calorimetry                        0.172973      0.329557      0.641247      0.308462      0.130038        10     
reco:pandoracalonosce:Calorimetry                   0.168672      0.319222      0.625731      0.280653      0.136329        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.154901      0.520133      0.922238      0.487761      0.240126        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.155919      0.479644      0.758257      0.455784      0.194903        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0264484     0.0403432     0.0703781     0.0366223     0.0122687       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0278639     0.0372629     0.049667      0.0376233    0.00596841       10     
reco:emtrkmichelid:EmTrackMichelId                   13.5149       18.6504       25.3334       17.8896       3.71672        10     
[art]:TriggerResults:TriggerResultInserter          2.691e-05    3.57302e-05   7.8448e-05    3.16545e-05   1.44626e-05      10     
end_path:out1:RootOutput                            6.693e-06    9.5459e-06    2.5718e-05     7.915e-06    5.43216e-06      10     
end_path:out1:RootOutput(write)                      0.80272       1.06362       1.28506       1.06008      0.123993        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5844.54 MB
  Peak resident set size usage (VmHWM): 3672.75 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10545548
-rw-r--r--. 1 dune osgvo     230975 Dec 18 16:25 jobscript.log
-rw-r--r--. 1 dune osgvo        199 Dec 18 16:25 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Dec 18 16:25 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Dec 18 16:25 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     139264 Dec 18 16:25 mem.db
-rw-r--r--. 1 dune osgvo      28672 Dec 18 16:25 time.db
-rw-r--r--. 1 dune osgvo 5466957477 Dec 18 16:25 pdhd_prod_beam__261612_43_1_20251212T155655Z_gen_g4_IonScintPDExt.root_268566_0_1_20251215T230702Z_PDInt.root_263305_31_1_20251218T210135Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   29336140 Dec 18 16:24 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Dec 18 16:10 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Dec 18 16:10 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 5301895217 Dec 18 16:10 temp_detsim.root
-rw-r--r--. 1 dune osgvo        120 Dec 18 15:01 all-input-dids.txt
justIN time: 2025-12-18 23:06:48 UTC       justIN version: 01.05.03