justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269125.0@dunegpschedd01.fnal.gov

Jobsub ID269125.0@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 21:10:11
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-18 23:11:30
From worker nodeHostnamedunegli-8107737-0-fnpc17126.fnal.gov
cpuinfoIntel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 21:24:25
Input filesusertests:pdhd_prod_beam__267379_156_1_20251212T162420Z_gen_g4_IonScintPDExt.root_262726_162_1_20251215T213957Z_PDInt.root
JobscriptExit code0
Real time1h (6365s)
CPU time1h (5137s = 80%)
Max RSS bytes6938107904 (6616 MiB)
Outputting started2025-12-18 23:10:30
Output filesusertests:pdhd_prod_beam__267379_156_1_20251212T162420Z_gen_g4_IonScintPDExt.root_262726_162_1_20251215T213957Z_PDInt.root_269125_0_1_20251218T212428Z_detsim_reco1.root
Finished2025-12-18 23:11:30
Saved logsjustin-logs:269125.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 49049 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":23624 [0] "wiener0":25425 [25425] ] cmm:[ bad:0 lf_noisy:333 ] input 1: frame: ident=189 time=0 tick=512 with 41383 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":17489 [0] "wiener1":23894 [23894] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 39355 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":17886 [0] "wiener2":21469 [21469] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 34784 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14994 [0] "wiener3":19790 [19790] ] cmm:[ bad:0 ] output: frame: ident=189 time=0 tick=512 with 164571 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23624 [0] "gauss1":17489 [0] "gauss2":17886 [0] "gauss3":14994 [0] "wiener0":25425 [25425] "wiener1":23894 [23894] "wiener2":21469 [21469] "wiener3":19790 [19790] ] cmm:[ bad:0 lf_noisy:333 ] 
Retagger: tagging trace set: wiener with 90578 traces, 90578 summary
Retagger: tagging trace set: gauss with 73993 traces, 0 summary
[23:09:34.552] D [  glue  ] sink frame: #189 @0 with 164571 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[23:09:34.552] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[23:09:34.552] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[23:09:34.552] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[23:09:34.552] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[23:09:34.552] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[23:09:34.552] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[23:09:34.552] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[23:09:34.553] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[23:09:34.553] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[23:09:34.553] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[23:09:34.553] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[23:09:34.553] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[23:09:34.553] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[23:09:34.553] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[23:09:34.553] D [  glue  ] frame sink sees EOS
[23:09:34.553] D [ pgraph ] <Pgrapher:> graph execution complete 
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.62 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.67 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.58 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.58 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.43 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.35 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.35 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.32 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[23:09:34.553] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[23:09:34.553] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[23:09:34.553] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[23:09:34.553] I [ timer  ] Timer: Total node execution : 47.949999617412686 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 73993 traces tagged "gauss"
FrameSaver: q=2.17856e+06 n=1156605 tag=gauss
wclsFrameSaver: saving 90578 traces tagged "wiener"
FrameSaver: q=2.91745e+06 n=1136653 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 41916 XUs and 56364 XVs -> 824 XUVs
C:0 T:3 6 XUs and 12 XVs -> 0 XUVs
C:0 T:5 22955 XUs and 19533 XVs -> 718 XUVs
C:0 T:6 22962 XUs and 36582 XVs -> 570 XUVs
C:0 T:7 14 XUs and 10 XVs -> 0 XUVs
2112 XUVs total
1216 collection wire objects
2112 potential space points
Neighbour search...
39586 tests to find 22570 neighbours
Iterating with no regularization...
Begin: 2.05135e+07
0 1.87756e+07
1 1.86484e+07
2 1.86404e+07
Now with regularization...
Begin: 1.78483e+07
0 1.7846e+07
18-Dec-2025 23:10:05 UTC  Closed output file "pdhd_prod_beam__267379_156_1_20251212T162420Z_gen_g4_IonScintPDExt.root_262726_162_1_20251215T213957Z_PDInt.root_269125_0_1_20251218T212428Z_detsim_reco1.root"
18-Dec-2025 23:10:05 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           71.4376       82.7291       97.0191       80.2844       8.13892        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000520026   0.00621814     0.0387219    0.00273718     0.0108821       10     
reco:wclsdatahd:WireCellToolkit                      49.048        50.792        52.7027       51.1471       1.19713        10     
reco:wirefilter:WireFilter                          0.0149963     0.0171449     0.0200274     0.0170491    0.00154916       10     
reco:gaushit:GausHitFinder                          0.420902      0.554519      0.712806      0.526491      0.0903498       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000212435   0.00028505    0.000393648   0.000256802   5.50387e-05      10     
reco:reco3d:SpacePointSolver                        0.236717      0.527258      0.941172      0.441603      0.229749        10     
reco:hitpdune:DisambigFromSpacePoints                0.26599      0.494095       0.85929      0.476267      0.189841        10     
reco:pandora:StandardPandora                         4.44044       7.68747       12.9318       5.97098       2.94078        10     
reco:pandoraWriter:StandardPandora                  0.128154      0.163601      0.213548      0.155478      0.028715        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.677816      0.882558       1.21522      0.857754      0.190309        10     
reco:pandoraShower:LArPandoraShowerCreation         0.637955      0.925861       1.48984       0.80329       0.29985        10     
reco:pandoracalo:Calorimetry                         0.37067       0.46707      0.642801      0.433654      0.0920043       10     
reco:pandoracalonosce:Calorimetry                   0.360669      0.453372      0.625159      0.420898      0.0910737       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.605432      0.884954       1.36913      0.758131      0.285954        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.60509      0.874734       1.35667        0.745       0.283837        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0331557     0.0429586     0.0539988     0.041697     0.00701859       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0318439     0.042247      0.0537609     0.0416679    0.00712302       10     
reco:emtrkmichelid:EmTrackMichelId                   12.4116       16.7553       21.8042       16.0493        3.003         10     
[art]:TriggerResults:TriggerResultInserter         1.8056e-05    2.52369e-05   5.9293e-05    2.1221e-05    1.14813e-05      10     
end_path:out1:RootOutput                            7.135e-06    8.6504e-06     1.897e-05     7.394e-06    3.45943e-06      10     
end_path:out1:RootOutput(write)                      0.82879       1.11987       1.59931       1.02395      0.233823        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5950.95 MB
  Peak resident set size usage (VmHWM): 3788.72 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11888108
-rw-r--r-- 1 dunegli fnalgrid     231384 Dec 18 23:10 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        202 Dec 18 23:10 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 23:10 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 23:10 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 23:10 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 23:10 time.db
-rw-r--r-- 1 dunegli fnalgrid 6166578909 Dec 18 23:10 pdhd_prod_beam__267379_156_1_20251212T162420Z_gen_g4_IonScintPDExt.root_262726_162_1_20251215T213957Z_PDInt.root_269125_0_1_20251218T212428Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   37192780 Dec 18 23:09 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 22:46 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 22:46 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5969195204 Dec 18 22:46 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        123 Dec 18 21:24 all-input-dids.txt
justIN time: 2026-02-04 10:46:50 UTC       justIN version: 01.06.00