justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261191.0@dunegpschedd02.fnal.gov

Jobsub ID261191.0@dunegpschedd02.fnal.gov
Workflow ID11142
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 19:16:28
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-12 12:50:53
From worker nodeHostnamedunegli-8033255-0-fnpc23022.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-12 11:37:37
Input filesusertests:pdhd_prod_beam__248282_2_1_20251112T235913Z_gen_g4_IonScintPDExt.root_258178_62_1_20251205T214141Z_PDInt.root
JobscriptExit code0
Real time1h (4313s)
CPU time1h (3976s = 92%)
Max RSS bytes7336660992 (6996 MiB)
Outputting started2025-12-12 12:49:30
Output filesusertests:pdhd_prod_beam__248282_2_1_20251112T235913Z_gen_g4_IonScintPDExt.root_258178_62_1_20251205T214141Z_PDInt.root_261191_0_1_20251212T113740Z_detsim_reco1.root
Finished2025-12-12 12:50:53
Saved logsjustin-logs:261191.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":25094 [0] "wiener0":28979 [28979] ] cmm:[ bad:0 lf_noisy:141 ] input 1: frame: ident=139 time=0 tick=512 with 42817 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":18568 [0] "wiener1":24249 [24249] ] cmm:[ bad:0 ] input 2: frame: ident=139 time=0 tick=512 with 25322 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":11618 [0] "wiener2":13704 [13704] ] cmm:[ bad:0 ] input 3: frame: ident=139 time=0 tick=512 with 45570 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":19094 [0] "wiener3":26476 [26476] ] cmm:[ bad:0 lf_noisy:1 ] output: frame: ident=139 time=0 tick=512 with 167782 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":25094 [0] "gauss1":18568 [0] "gauss2":11618 [0] "gauss3":19094 [0] "wiener0":28979 [28979] "wiener1":24249 [24249] "wiener2":13704 [13704] "wiener3":26476 [26476] ] cmm:[ bad:0 lf_noisy:142 ] 
Retagger: tagging trace set: wiener with 93408 traces, 93408 summary
Retagger: tagging trace set: gauss with 74374 traces, 0 summary
[12:49:07.494] D [  glue  ] sink frame: #139 @0 with 167782 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[12:49:07.494] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[12:49:07.494] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[12:49:07.494] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[12:49:07.494] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[12:49:07.494] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[12:49:07.494] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[12:49:07.494] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[12:49:07.494] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[12:49:07.494] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[12:49:07.494] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[12:49:07.494] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[12:49:07.494] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[12:49:07.494] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[12:49:07.494] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[12:49:07.494] D [  glue  ] frame sink sees EOS
[12:49:07.494] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.84 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.88 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.86 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.62 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.03 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.99 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.99 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.97 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:49:07.494] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:49:07.494] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:49:07.494] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:49:07.494] I [ timer  ] Timer: Total node execution : 34.200000304728746 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 74374 traces tagged "gauss"
FrameSaver: q=2.14612e+06 n=1110780 tag=gauss
wclsFrameSaver: saving 93408 traces tagged "wiener"
FrameSaver: q=3.04271e+06 n=1067068 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 32178 XUs and 17903 XVs -> 605 XUVs
C:0 T:3 13 XUs and 12 XVs -> 0 XUVs
C:0 T:4 4 XUs and 3 XVs -> 0 XUVs
C:0 T:5 5152 XUs and 4495 XVs -> 212 XUVs
C:0 T:6 32200 XUs and 16461 XVs -> 509 XUVs
C:0 T:7 29 XUs and 25 XVs -> 0 XUVs
1326 XUVs total
955 collection wire objects
1326 potential space points
Neighbour search...
14358 tests to find 8204 neighbours
Iterating with no regularization...
Begin: 1.14554e+07
0 1.09717e+07
1 1.09365e+07
2 1.09331e+07
Now with regularization...
Begin: 9.92761e+06
0 9.92677e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
12-Dec-2025 12:49:29 UTC  Closed output file "pdhd_prod_beam__248282_2_1_20251112T235913Z_gen_g4_IonScintPDExt.root_258178_62_1_20251205T214141Z_PDInt.root_261191_0_1_20251212T113740Z_detsim_reco1.root"
12-Dec-2025 12:49:29 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           52.8707       69.4867       91.5038       70.6721       10.3807        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00102263    0.00419328    0.00612385    0.00469532    0.00160315       10     
reco:wclsdatahd:WireCellToolkit                      35.1257       38.273        48.4369       37.5618       3.60429        10     
reco:wirefilter:WireFilter                          0.0102818     0.0163749     0.0243332     0.0142773    0.00447667       10     
reco:gaushit:GausHitFinder                          0.226489      0.383669      0.482906      0.427408      0.0775855       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000212866   0.000414534   0.000949351   0.00032271    0.000242048      10     
reco:reco3d:SpacePointSolver                        0.149109      0.346934      0.577801      0.337031      0.138355        10     
reco:hitpdune:DisambigFromSpacePoints               0.087327      0.242351      0.384541      0.236887      0.0968253       10     
reco:pandora:StandardPandora                         2.19135       6.13956       8.51499       6.76664       1.96137        10     
reco:pandoraWriter:StandardPandora                  0.105594      0.217306      0.335449       0.21394      0.0729446       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.287142      0.762528       1.1843       0.745267      0.305052        10     
reco:pandoraShower:LArPandoraShowerCreation         0.225993      0.604675       0.85082      0.656621      0.205616        10     
reco:pandoracalo:Calorimetry                        0.166454      0.396521      0.737201      0.383193      0.151994        10     
reco:pandoracalonosce:Calorimetry                    0.15555      0.347808      0.504997       0.3456       0.0996748       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.204147      0.637861       1.28088      0.596778      0.291853        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.201991      0.577814      0.865023      0.583474      0.204615        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0227588     0.0329588     0.0452328     0.0309396     0.0069274       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0243822     0.0317819     0.0407534     0.0317292     0.0054602       10     
reco:emtrkmichelid:EmTrackMichelId                   10.6802       19.4358       28.1827       20.3905       4.95135        10     
[art]:TriggerResults:TriggerResultInserter         2.8324e-05    4.91398e-05   0.000121472   3.70005e-05   2.83854e-05      10     
end_path:out1:RootOutput                           1.2033e-05    1.82298e-05   4.3613e-05    1.39815e-05   9.32156e-06      10     
end_path:out1:RootOutput(write)                     0.540132      0.934175       1.46888       0.9414       0.250291        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5964.17 MB
  Peak resident set size usage (VmHWM): 3811.44 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12926232
-rw-r--r-- 1 dunegli fnalgrid     231405 Dec 12 12:49 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        193 Dec 12 12:49 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 12:49 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 12 12:49 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 12 12:49 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 12 12:49 time.db
-rw-r--r-- 1 dunegli fnalgrid 6697308594 Dec 12 12:49 pdhd_prod_beam__248282_2_1_20251112T235913Z_gen_g4_IonScintPDExt.root_258178_62_1_20251205T214141Z_PDInt.root_261191_0_1_20251212T113740Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   35728900 Dec 12 12:49 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 12 12:36 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 12:36 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6502966547 Dec 12 12:36 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Dec 12 11:37 all-input-dids.txt
justIN time: 2026-02-07 13:30:28 UTC       justIN version: 01.06.00