justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242701.1@dunegpschedd01.fnal.gov

Jobsub ID242701.1@dunegpschedd01.fnal.gov
Workflow ID9567
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-04 00:31:27
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-04 01:27:41
From worker nodeHostnamedunegli-7473162-0-fnpc22005.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 00:32:51
Input filesusertests:pdhd_prod_beam__231902_55_1_20251008T212443Z_gen_g4_IonScintPDExt.root_227040_79_1_20251010T170328Z_PDInt.root
JobscriptExit code0
Real time54m (3241s)
CPU time53m (3185s = 98%)
Max RSS bytes6509383680 (6207 MiB)
Outputting started2025-11-04 01:26:52
Output filesusertests:pdhd_prod_beam__231902_55_1_20251008T212443Z_gen_g4_IonScintPDExt.root_227040_79_1_20251010T170328Z_PDInt.root_242701_1_1_20251104T003254Z_detsim_reco1.root
Finished2025-11-04 01:27:41
Saved logsjustin-logs:242701.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t=99 time=0 tick=512 with 41544 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":19909 [0] "wiener0":21635 [21635] ] cmm:[ bad:0 lf_noisy:138 ] input 1: frame: ident=99 time=0 tick=512 with 44191 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":18919 [0] "wiener1":25272 [25272] ] cmm:[ bad:0 lf_noisy:1 ] input 2: frame: ident=99 time=0 tick=512 with 44935 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":18745 [0] "wiener2":26190 [26190] ] cmm:[ bad:0 ] input 3: frame: ident=99 time=0 tick=512 with 30631 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":13728 [0] "wiener3":16903 [16903] ] cmm:[ bad:0 ] output: frame: ident=99 time=0 tick=512 with 161301 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":19909 [0] "gauss1":18919 [0] "gauss2":18745 [0] "gauss3":13728 [0] "wiener0":21635 [21635] "wiener1":25272 [25272] "wiener2":26190 [26190] "wiener3":16903 [16903] ] cmm:[ bad:0 lf_noisy:139 ] 
Retagger: tagging trace set: wiener with 90000 traces, 90000 summary
Retagger: tagging trace set: gauss with 71301 traces, 0 summary
[01:26:29.128] D [  glue  ] sink frame: #99 @0 with 161301 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[01:26:29.128] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[01:26:29.128] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[01:26:29.128] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[01:26:29.128] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[01:26:29.128] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[01:26:29.128] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[01:26:29.128] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[01:26:29.128] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[01:26:29.128] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[01:26:29.128] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[01:26:29.128] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[01:26:29.128] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[01:26:29.128] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[01:26:29.128] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[01:26:29.128] D [  glue  ] frame sink sees EOS
[01:26:29.128] D [ pgraph ] <Pgrapher:> graph execution complete 
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.8 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.34 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.33 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.06 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.01 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.98 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.97 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[01:26:29.129] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[01:26:29.129] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[01:26:29.129] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[01:26:29.129] I [ timer  ] Timer: Total node execution : 31.52000024728477 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 71301 traces tagged "gauss"
FrameSaver: q=2.12264e+06 n=1103149 tag=gauss
wclsFrameSaver: saving 90000 traces tagged "wiener"
FrameSaver: q=2.72958e+06 n=1082238 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 36150 XUs and 39654 XVs -> 1353 XUVs
C:0 T:3 41 XUs and 52 XVs -> 7 XUVs
C:0 T:4 92 XUs and 51 XVs -> 2 XUVs
C:0 T:5 70794 XUs and 52486 XVs -> 4338 XUVs
C:0 T:6 8535 XUs and 9330 XVs -> 1070 XUVs
C:0 T:7 1 XUs and 9 XVs -> 0 XUVs
6770 XUVs total
1462 collection wire objects
6770 potential space points
Neighbour search...
647090 tests to find 377244 neighbours
Iterating with no regularization...
Begin: 3.09967e+07
0 2.67969e+07
1 2.64767e+07
2 2.6394e+07
3 2.63652e+07
4 2.63507e+07
Now with regularization...
Begin: 2.47846e+07
0 2.47714e+07
04-Nov-2025 01:26:51 UTC  Closed output file "pdhd_prod_beam__231902_55_1_20251008T212443Z_gen_g4_IonScintPDExt.root_227040_79_1_20251010T170328Z_PDInt.root_242701_1_1_20251104T003254Z_detsim_reco1.root"
04-Nov-2025 01:26:51 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           43.3655       51.4556       61.7402       50.1859       4.72089        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00039002    0.00226965    0.00378919    0.00243385    0.00103138       10     
reco:wclsdatahd:WireCellToolkit                      31.8883       32.3968       33.2079       32.321       0.397711        10     
reco:wirefilter:WireFilter                         0.00748474    0.00891138     0.0105205    0.00878403    0.000830678      10     
reco:gaushit:GausHitFinder                          0.210145      0.306709      0.438946      0.293751      0.0613351       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000109959    0.0001657    0.000433382   0.000140852   9.03319e-05      10     
reco:reco3d:SpacePointSolver                        0.0846381     0.254229      0.682342      0.169026      0.171023        10     
reco:hitpdune:DisambigFromSpacePoints               0.0707826     0.162887      0.267914      0.153828      0.0524914       10     
reco:pandora:StandardPandora                         1.32232       3.25131       6.3206        2.96565       1.25411        10     
reco:pandoraWriter:StandardPandora                  0.0517046     0.0855846     0.135789      0.0792895     0.021313        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.158205      0.386931      0.580209      0.357539      0.121963        10     
reco:pandoraShower:LArPandoraShowerCreation         0.112724      0.348647      0.621696      0.319607      0.130176        10     
reco:pandoracalo:Calorimetry                        0.102434      0.245224      0.360667      0.234642      0.0719312       10     
reco:pandoracalonosce:Calorimetry                   0.0982136     0.236079      0.355829      0.227037      0.070382        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.115749      0.372755      0.668085      0.350872      0.144005        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.111298      0.365583      0.655996      0.341768      0.141778        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0163749     0.024017      0.036227      0.024245     0.00505227       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0161558     0.0228739     0.0299615     0.0234927    0.00376028       10     
reco:emtrkmichelid:EmTrackMichelId                   8.38653       12.3216       16.8434       11.9747       2.3016         10     
[art]:TriggerResults:TriggerResultInserter         1.3916e-05    2.74009e-05   0.000107323   1.7303e-05    2.68652e-05      10     
end_path:out1:RootOutput                            6.623e-06    1.23816e-05    3.674e-05    9.5935e-06    8.26649e-06      10     
end_path:out1:RootOutput(write)                     0.470335      0.647039      0.850133      0.627525       0.09555        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5825.1 MB
  Peak resident set size usage (VmHWM): 3663.99 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11115268
-rw-r--r-- 1 dunegli fnalgrid     230495 Nov  4 01:26 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Nov  4 01:26 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Nov  4 01:26 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Nov  4 01:26 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Nov  4 01:26 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Nov  4 01:26 time.db
-rw-r--r-- 1 dunegli fnalgrid 5761046879 Nov  4 01:26 pdhd_prod_beam__231902_55_1_20251008T212443Z_gen_g4_IonScintPDExt.root_227040_79_1_20251010T170328Z_PDInt.root_242701_1_1_20251104T003254Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   30248370 Nov  4 01:26 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Nov  4 01:17 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Nov  4 01:17 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5590282081 Nov  4 01:17 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Nov  4 00:32 all-input-dids.txt
justIN time: 2025-11-05 12:15:32 UTC       justIN version: 01.05.01