justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.56@dunegpschedd01.fnal.gov

Jobsub ID269119.56@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UChicago
EntryEngage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat2025-12-18 21:46:09
From worker nodeHostnamemwt2-c197.campuscluster.illinois.edu
cpuinfoAMD EPYC 7402 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:04:59
Input filesusertests:pdhd_prod_beam__261612_9_1_20251212T155456Z_gen_g4_IonScintPDExt.root_268535_60_1_20251215T215350Z_PDInt.root
JobscriptExit code0
Real time1h (5950s)
CPU time1h (5763s = 96%)
Max RSS bytes7600058368 (7247 MiB)
Outputting started2025-12-18 21:44:10
Output filesusertests:pdhd_prod_beam__261612_9_1_20251212T155456Z_gen_g4_IonScintPDExt.root_268535_60_1_20251215T215350Z_PDInt.root_269119_56_1_20251218T200508Z_detsim_reco1.root
Finished2025-12-18 21:46:09
Saved logsjustin-logs:269119.56-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[17721] ] cmm:[ bad:0 lf_noisy:70 ] 
[15:43:24.045] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 34207 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":16486 [0] "wiener0":17721 [17721] ] cmm:[ bad:0 lf_noisy:70 ] input 1: frame: ident=179 time=0 tick=512 with 31814 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13867 [0] "wiener1":17947 [17947] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 48115 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":20321 [0] "wiener2":27794 [27794] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 35228 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15591 [0] "wiener3":19637 [19637] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 149364 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":16486 [0] "gauss1":13867 [0] "gauss2":20321 [0] "gauss3":15591 [0] "wiener0":17721 [17721] "wiener1":17947 [17947] "wiener2":27794 [27794] "wiener3":19637 [19637] ] cmm:[ bad:0 lf_noisy:70 ] 
Retagger: tagging trace set: wiener with 83099 traces, 83099 summary
Retagger: tagging trace set: gauss with 66265 traces, 0 summary
[15:43:24.057] D [  glue  ] sink frame: #179 @0 with 149364 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[15:43:24.057] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[15:43:24.057] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[15:43:24.057] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[15:43:24.057] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[15:43:24.057] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[15:43:24.057] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[15:43:24.057] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[15:43:24.057] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[15:43:24.057] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[15:43:24.057] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[15:43:24.057] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[15:43:24.057] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[15:43:24.057] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[15:43:24.057] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[15:43:24.057] D [  glue  ] frame sink sees EOS
[15:43:24.057] D [ pgraph ] <Pgrapher:> graph execution complete 
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.75 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.4 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.92 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.27 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.9 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.85 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.56 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:43:24.057] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[15:43:24.057] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[15:43:24.057] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[15:43:24.057] I [ timer  ] Timer: Total node execution : 56.66999961808324 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 66265 traces tagged "gauss"
FrameSaver: q=1.92001e+06 n=1021162 tag=gauss
wclsFrameSaver: saving 83099 traces tagged "wiener"
FrameSaver: q=2.43454e+06 n=987326 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 12254 XUs and 10259 XVs -> 486 XUVs
C:0 T:3 1 XUs and 5 XVs -> 0 XUVs
C:0 T:4 8 XUs and 8 XVs -> 0 XUVs
C:0 T:5 41821 XUs and 26582 XVs -> 1100 XUVs
C:0 T:6 16120 XUs and 14912 XVs -> 739 XUVs
C:0 T:7 2 XUs and 13 XVs -> 1 XUVs
2326 XUVs total
1429 collection wire objects
2326 potential space points
Neighbour search...
35854 tests to find 20006 neighbours
Iterating with no regularization...
Begin: 1.85032e+07
0 1.77527e+07
1 1.77113e+07
2 1.7707e+07
Now with regularization...
Begin: 1.63571e+07
0 1.63563e+07
18-Dec-2025 15:44:08 CST  Closed output file "pdhd_prod_beam__261612_9_1_20251212T155456Z_gen_g4_IonScintPDExt.root_268535_60_1_20251215T215350Z_PDInt.root_269119_56_1_20251218T200508Z_detsim_reco1.root"
18-Dec-2025 15:44:08 CST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           84.2351       106.355       135.559       102.355       13.5985        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.0014968    0.00406021     0.0078642    0.00384961    0.00190467       10     
reco:wclsdatahd:WireCellToolkit                      55.1516       59.2455       61.8934       59.5955       2.27716        10     
reco:wirefilter:WireFilter                          0.0162471     0.0270598     0.0399778     0.0247637    0.00740033       10     
reco:gaushit:GausHitFinder                          0.349857      0.635556      0.975892      0.605338      0.188734        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000182173   0.000372994   0.000640467   0.000372282   0.000127504      10     
reco:reco3d:SpacePointSolver                        0.198807      0.515086       1.09144      0.495282       0.24036        10     
reco:hitpdune:DisambigFromSpacePoints               0.193497      0.434769      0.889196      0.361098      0.224943        10     
reco:pandora:StandardPandora                         3.88481       9.24211       18.3476       8.45781       3.73607        10     
reco:pandoraWriter:StandardPandora                   0.18415      0.269363      0.394355      0.257918      0.0610116       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.534641       1.00121       1.40981       1.02535      0.249967        10     
reco:pandoraShower:LArPandoraShowerCreation         0.433207       1.02352       1.94063      0.947122       0.47296        10     
reco:pandoracalo:Calorimetry                        0.321197      0.590823       1.12756      0.553649      0.205073        10     
reco:pandoracalonosce:Calorimetry                   0.291176      0.574657       0.9555       0.532457      0.188823        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.317987       1.01943       1.96236      0.883294       0.50693        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.31465       1.01141       2.25081      0.904792      0.554954        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0346959     0.0583988     0.0910701     0.0612554     0.0172034       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0349951     0.0580276     0.0947827     0.0590474     0.0178569       10     
reco:emtrkmichelid:EmTrackMichelId                   14.5687       29.2303       41.1137       28.4302       7.21675        10     
[art]:TriggerResults:TriggerResultInserter         2.5518e-05    5.01194e-05   0.00013306    3.74255e-05   2.96064e-05      10     
end_path:out1:RootOutput                            8.446e-06    1.3239e-05    3.4495e-05     9.693e-06    7.91085e-06      10     
end_path:out1:RootOutput(write)                     0.957826        1.33         1.96032       1.32716      0.257145        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5834.25 MB
  Peak resident set size usage (VmHWM): 3640.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12057144
-rw-r--r-- 1 dune osgvo     231252 Dec 18 15:44 jobscript.log
-rw-r--r-- 1 dune osgvo        199 Dec 18 15:44 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Dec 18 15:44 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Dec 18 15:44 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     139264 Dec 18 15:44 mem.db
-rw-r--r-- 1 dune osgvo      28672 Dec 18 15:44 time.db
-rw-r--r-- 1 dune osgvo 6248924302 Dec 18 15:44 pdhd_prod_beam__261612_9_1_20251212T155456Z_gen_g4_IonScintPDExt.root_268535_60_1_20251215T215350Z_PDInt.root_269119_56_1_20251218T200508Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   34810510 Dec 18 15:43 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Dec 18 15:24 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Dec 18 15:23 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 6062322653 Dec 18 15:23 temp_detsim.root
-rw-r--r-- 1 dune osgvo        120 Dec 18 14:05 all-input-dids.txt
justIN time: 2025-12-19 13:10:10 UTC       justIN version: 01.05.03