justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269170.0@dunegpschedd01.fnal.gov

Jobsub ID269170.0@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 23:10:17
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-19 01:15:57
From worker nodeHostnamedunegli-7950916-0-fnpc18002.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 23:11:32
Input filesusertests:pdhd_prod_beam__267379_115_1_20251212T162136Z_gen_g4_IonScintPDExt.root_262726_87_1_20251215T212443Z_PDInt.root
JobscriptExit code0
Real time2h (7370s)
CPU time1h (6953s = 94%)
Max RSS bytes7165128704 (6833 MiB)
Outputting started2025-12-19 01:14:23
Output filesusertests:pdhd_prod_beam__267379_115_1_20251212T162136Z_gen_g4_IonScintPDExt.root_262726_87_1_20251215T212443Z_PDInt.root_269170_0_1_20251218T231137Z_detsim_reco1.root
Finished2025-12-19 01:15:57
Saved logsjustin-logs:269170.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 55997 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":26738 [0] "wiener0":29259 [29259] ] cmm:[ bad:0 lf_noisy:67 ] input 1: frame: ident=189 time=0 tick=512 with 34834 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":15262 [0] "wiener1":19572 [19572] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 36453 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15632 [0] "wiener2":20821 [20821] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 39029 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16991 [0] "wiener3":22038 [22038] ] cmm:[ bad:0 ] output: frame: ident=189 time=0 tick=512 with 166313 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":26738 [0] "gauss1":15262 [0] "gauss2":15632 [0] "gauss3":16991 [0] "wiener0":29259 [29259] "wiener1":19572 [19572] "wiener2":20821 [20821] "wiener3":22038 [22038] ] cmm:[ bad:0 lf_noisy:67 ] 
Retagger: tagging trace set: wiener with 91690 traces, 91690 summary
Retagger: tagging trace set: gauss with 74623 traces, 0 summary
[01:13:20.232] D [  glue  ] sink frame: #189 @0 with 166313 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[01:13:20.232] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[01:13:20.232] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[01:13:20.232] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[01:13:20.232] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[01:13:20.232] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[01:13:20.232] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[01:13:20.232] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[01:13:20.232] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[01:13:20.232] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[01:13:20.232] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[01:13:20.232] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[01:13:20.232] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[01:13:20.232] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[01:13:20.232] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[01:13:20.232] D [  glue  ] frame sink sees EOS
[01:13:20.233] D [ pgraph ] <Pgrapher:> graph execution complete 
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 16.9 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.13 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.11 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.85 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.56 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.47 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.37 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.37 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:13:20.233] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[01:13:20.233] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[01:13:20.233] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[01:13:20.233] I [ timer  ] Timer: Total node execution : 67.78999927453697 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 74623 traces tagged "gauss"
FrameSaver: q=2.27203e+06 n=1125984 tag=gauss
wclsFrameSaver: saving 91690 traces tagged "wiener"
FrameSaver: q=3.09564e+06 n=1104584 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 25177 XUs and 26507 XVs -> 1140 XUVs
C:0 T:3 17 XUs and 2 XVs -> 0 XUVs
C:0 T:4 2 XUs and 1 XVs -> 0 XUVs
C:0 T:5 22711 XUs and 40842 XVs -> 797 XUVs
C:0 T:6 26443 XUs and 34601 XVs -> 761 XUVs
2698 XUVs total
1586 collection wire objects
2698 potential space points
Neighbour search...
89336 tests to find 42300 neighbours
Iterating with no regularization...
Begin: 2.88093e+07
0 2.52491e+07
1 2.4926e+07
2 2.4896e+07
3 2.48901e+07
Now with regularization...
Begin: 2.36259e+07
0 2.36221e+07
19-Dec-2025 01:14:21 UTC  Closed output file "pdhd_prod_beam__267379_115_1_20251212T162136Z_gen_g4_IonScintPDExt.root_262726_87_1_20251215T212443Z_PDInt.root_269170_0_1_20251218T231137Z_detsim_reco1.root"
19-Dec-2025 01:14:21 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           88.9942       118.615       142.387       124.197       17.0474        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00128517    0.00394197    0.00684617    0.00419593    0.00163504       10     
reco:wclsdatahd:WireCellToolkit                      53.0111       58.4697       70.9193       55.8392       6.03411        10     
reco:wirefilter:WireFilter                          0.0187115     0.0234561     0.0301048     0.0235495    0.00356046       10     
reco:gaushit:GausHitFinder                          0.500003      0.781347       1.10176      0.800826      0.181946        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000236975   0.000357635   0.000517532   0.000360562   8.09411e-05      10     
reco:reco3d:SpacePointSolver                        0.275375      0.535086      0.908397       0.57386      0.192486        10     
reco:hitpdune:DisambigFromSpacePoints                0.16975      0.476061      0.735326      0.485984      0.187241        10     
reco:pandora:StandardPandora                         5.00124       10.4862       17.0414       9.35677       4.18829        10     
reco:pandoraWriter:StandardPandora                  0.171161      0.273093      0.399375      0.261171      0.0654131       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.579271       1.20818       2.29847       1.18195      0.497411        10     
reco:pandoraShower:LArPandoraShowerCreation         0.653668       1.36628       2.82405       1.17294      0.682312        10     
reco:pandoracalo:Calorimetry                        0.249279       0.61892      0.944002      0.613046      0.214468        10     
reco:pandoracalonosce:Calorimetry                   0.294035      0.590869      0.890771      0.562175      0.189254        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.652413       1.23886       2.62089       1.05688      0.602783        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.613717       1.25391       2.59561       1.03077      0.622393        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0440966     0.0684332     0.118007      0.0666988     0.0206052       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0400586     0.0677544     0.118361      0.0691035     0.0216237       10     
reco:emtrkmichelid:EmTrackMichelId                   25.5008       39.4781       54.893        40.9122       8.57852        10     
[art]:TriggerResults:TriggerResultInserter         2.9543e-05    4.70388e-05   0.000124948   3.87275e-05   2.62738e-05      10     
end_path:out1:RootOutput                           1.1594e-05    1.70067e-05   4.1905e-05    1.4318e-05    8.44711e-06      10     
end_path:out1:RootOutput(write)                      1.1214        1.62082       2.5581        1.62163      0.364345        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5958.7 MB
  Peak resident set size usage (VmHWM): 3794.89 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12176696
-rw-r--r-- 1 dunegli fnalgrid     231509 Dec 19 01:14 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Dec 19 01:14 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 19 01:14 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 19 01:14 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 19 01:14 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 19 01:14 time.db
-rw-r--r-- 1 dunegli fnalgrid 6314570844 Dec 19 01:14 pdhd_prod_beam__267379_115_1_20251212T162136Z_gen_g4_IonScintPDExt.root_262726_87_1_20251215T212443Z_PDInt.root_269170_0_1_20251218T231137Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   36569520 Dec 19 01:13 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 19 00:50 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 19 00:49 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6117328463 Dec 19 00:49 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Dec 18 23:11 all-input-dids.txt
justIN time: 2026-02-04 09:33:30 UTC       justIN version: 01.06.00