justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 235051.0@dunegpschedd02.fnal.gov

Jobsub ID235051.0@dunegpschedd02.fnal.gov
Workflow ID9455
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 23:06:57
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_condce_mcore
Last heartbeat2025-10-30 00:22:51
From worker nodeHostnameiut2-c413.iu.edu
cpuinfoAMD EPYC 7443 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-29 23:07:34
Input filesusertests:pdhd_prod_beam__226242_58_1_20251008T182908Z_gen_g4_IonScintPDExt.root_227038_152_1_20251010T165520Z_PDInt.root
JobscriptExit code0
Real time1h (4405s)
CPU time1h (4268s = 96%)
Max RSS bytes6614605824 (6308 MiB)
Outputting started2025-10-30 00:21:00
Output filesusertests:pdhd_prod_beam__226242_58_1_20251008T182908Z_gen_g4_IonScintPDExt.root_227038_152_1_20251010T165520Z_PDInt.root_235051_0_1_20251029T230740Z_detsim_reco1.root
Finished2025-10-30 00:22:51
Saved logsjustin-logs:235051.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[20:19:50.855] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=89 time=0 tick=512 with 55830 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":25817 [0] "wiener0":30013 [30013] ] cmm:[ bad:0 lf_noisy:85 ] input 1: frame: ident=89 time=0 tick=512 with 38304 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16448 [0] "wiener1":21856 [21856] ] cmm:[ bad:0 ] input 2: frame: ident=89 time=0 tick=512 with 69495 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":28750 [0] "wiener2":40745 [40745] ] cmm:[ bad:0 lf_noisy:1 ] input 3: frame: ident=89 time=0 tick=512 with 43362 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":18008 [0] "wiener3":25354 [25354] ] cmm:[ bad:0 ] output: frame: ident=89 time=0 tick=512 with 206991 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":25817 [0] "gauss1":16448 [0] "gauss2":28750 [0] "gauss3":18008 [0] "wiener0":30013 [30013] "wiener1":21856 [21856] "wiener2":40745 [40745] "wiener3":25354 [25354] ] cmm:[ bad:0 lf_noisy:86 ] 
Retagger: tagging trace set: wiener with 117968 traces, 117968 summary
Retagger: tagging trace set: gauss with 89023 traces, 0 summary
[20:19:50.872] D [  glue  ] sink frame: #89 @0 with 206991 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:19:50.872] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[20:19:50.872] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[20:19:50.872] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[20:19:50.872] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[20:19:50.872] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[20:19:50.872] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[20:19:50.872] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[20:19:50.873] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[20:19:50.873] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[20:19:50.873] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[20:19:50.873] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[20:19:50.873] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[20:19:50.873] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[20:19:50.873] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[20:19:50.873] D [  glue  ] frame sink sees EOS
[20:19:50.873] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.02 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.12 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.42 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.35 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.91 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.86 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.85 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.85 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:19:50.873] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:19:50.873] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:19:50.873] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:19:50.873] I [ timer  ] Timer: Total node execution : 50.43000059016049 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 89023 traces tagged "gauss"
FrameSaver: q=2.89138e+06 n=1411796 tag=gauss
wclsFrameSaver: saving 117968 traces tagged "wiener"
FrameSaver: q=3.65388e+06 n=1360986 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 50780 XUs and 59316 XVs -> 1480 XUVs
C:0 T:4 70 XUs and 40 XVs -> 8 XUVs
C:0 T:5 87143 XUs and 65731 XVs -> 2038 XUVs
C:0 T:6 60671 XUs and 55841 XVs -> 1353 XUVs
4879 XUVs total
2553 collection wire objects
4879 potential space points
Neighbour search...
105303 tests to find 61010 neighbours
Iterating with no regularization...
Begin: 5.69638e+07
0 5.34778e+07
1 5.33112e+07
2 5.32914e+07
Now with regularization...
Begin: 5.04921e+07
0 5.04813e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:20:58 EDT  Closed output file "pdhd_prod_beam__226242_58_1_20251008T182908Z_gen_g4_IonScintPDExt.root_227038_152_1_20251010T165520Z_PDInt.root_235051_0_1_20251029T230740Z_detsim_reco1.root"
29-Oct-2025 20:20:58 EDT  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           72.3895       93.2558       119.223       92.0181       11.5833        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00124916    0.00446174     0.0068862    0.00450963    0.00170499       10     
reco:wclsdatahd:WireCellToolkit                      48.8414       51.774        55.5449       51.3522       1.98347        10     
reco:wirefilter:WireFilter                          0.0219702     0.0295704     0.0384554     0.0283767    0.00482274       10     
reco:gaushit:GausHitFinder                          0.411582      0.623959      0.896056      0.636079       0.12183        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000220703   0.000350549   0.000517723   0.000323429   0.000101226      10     
reco:reco3d:SpacePointSolver                        0.276684      0.791269       2.12041      0.499071      0.589902        10     
reco:hitpdune:DisambigFromSpacePoints               0.273078      0.474629      0.953705       0.45065      0.187745        10     
reco:pandora:StandardPandora                         3.99921       8.70266       17.3012       7.78921       3.73035        10     
reco:pandoraWriter:StandardPandora                  0.170555      0.248288      0.338434      0.246847      0.0436784       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.450404       1.07661       1.74268       1.0134        0.34102        10     
reco:pandoraShower:LArPandoraShowerCreation         0.437256      0.983559       2.29875      0.911603      0.478732        10     
reco:pandoracalo:Calorimetry                        0.286568      0.612507      0.904388      0.611067       0.16755        10     
reco:pandoracalonosce:Calorimetry                   0.277507      0.579565      0.869778      0.582331      0.151403        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.41916      0.951765       1.82606      0.911837      0.356642        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.387923      0.943132       1.75487      0.916185      0.346196        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0421713     0.0613577     0.0741705     0.0607981     0.0100571       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.042517      0.059342      0.0739473     0.0594663     0.0091432       10     
reco:emtrkmichelid:EmTrackMichelId                   14.8991       23.9623       33.6707       24.2075       4.66863        10     
[art]:TriggerResults:TriggerResultInserter         3.0718e-05    4.91478e-05   0.000114595   4.2725e-05    2.2792e-05       10     
end_path:out1:RootOutput                            9.267e-06    1.34243e-05   3.4946e-05    1.0996e-05    7.25799e-06      10     
end_path:out1:RootOutput(write)                     0.936055       1.29043       1.6352        1.30122      0.171222        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5748.85 MB
  Peak resident set size usage (VmHWM): 3562.02 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11782016
-rw-r--r-- 1 dune osgvo     230975 Oct 29 20:21 jobscript.log
-rw-r--r-- 1 dune osgvo        201 Oct 29 20:21 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Oct 29 20:20 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Oct 29 20:20 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     139264 Oct 29 20:20 mem.db
-rw-r--r-- 1 dune osgvo      28672 Oct 29 20:20 time.db
-rw-r--r-- 1 dune osgvo 6107541328 Oct 29 20:20 pdhd_prod_beam__226242_58_1_20251008T182908Z_gen_g4_IonScintPDExt.root_227038_152_1_20251010T165520Z_PDInt.root_235051_0_1_20251029T230740Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   34361270 Oct 29 20:20 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Oct 29 20:03 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Oct 29 20:03 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 5922438830 Oct 29 20:03 temp_detsim.root
-rw-r--r-- 1 dune osgvo        122 Oct 29 19:07 all-input-dids.txt
justIN time: 2025-11-04 15:52:12 UTC       justIN version: 01.05.01