justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.23@dunegpschedd02.fnal.gov

Jobsub ID263305.23@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-18 22:41:17
From worker nodeHostnamedunegli-7950920-0-fnpc18009.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:51:15
Input filesusertests:pdhd_prod_beam__261612_74_1_20251212T160117Z_gen_g4_IonScintPDExt.root_268535_57_1_20251215T215305Z_PDInt.root
JobscriptExit code0
Real time1h (6504s)
CPU time1h (6053s = 93%)
Max RSS bytes6634110976 (6326 MiB)
Outputting started2025-12-18 22:39:39
Output filesusertests:pdhd_prod_beam__261612_74_1_20251212T160117Z_gen_g4_IonScintPDExt.root_268535_57_1_20251215T215305Z_PDInt.root_263305_23_1_20251218T205121Z_detsim_reco1.root
Finished2025-12-18 22:41:03
Saved logsjustin-logs:263305.23-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

: ident=179 time=0 tick=512 with 57698 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":27228 [0] "wiener0":30470 [30470] ] cmm:[ bad:0 lf_noisy:25 ] input 1: frame: ident=179 time=0 tick=512 with 40356 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":17749 [0] "wiener1":22607 [22607] ] cmm:[ bad:0 lf_noisy:1 ] input 2: frame: ident=179 time=0 tick=512 with 38689 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":16787 [0] "wiener2":21902 [21902] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 32055 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":13812 [0] "wiener3":18243 [18243] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 168798 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":27228 [0] "gauss1":17749 [0] "gauss2":16787 [0] "gauss3":13812 [0] "wiener0":30470 [30470] "wiener1":22607 [22607] "wiener2":21902 [21902] "wiener3":18243 [18243] ] cmm:[ bad:0 lf_noisy:26 ] 
Retagger: tagging trace set: wiener with 93222 traces, 93222 summary
Retagger: tagging trace set: gauss with 75576 traces, 0 summary
[22:38:45.542] D [  glue  ] sink frame: #179 @0 with 168798 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[22:38:45.542] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[22:38:45.542] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[22:38:45.542] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[22:38:45.542] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[22:38:45.542] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[22:38:45.542] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[22:38:45.542] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[22:38:45.542] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[22:38:45.542] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[22:38:45.542] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[22:38:45.542] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[22:38:45.542] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[22:38:45.542] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[22:38:45.542] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[22:38:45.542] D [  glue  ] frame sink sees EOS
[22:38:45.542] D [ pgraph ] <Pgrapher:> graph execution complete 
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.31 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.87 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.85 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.6 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.64 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.46 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.46 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.44 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[22:38:45.542] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[22:38:45.543] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:45.543] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[22:38:45.543] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:45.543] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:38:45.543] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[22:38:45.543] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[22:38:45.543] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[22:38:45.543] I [ timer  ] Timer: Total node execution : 54.670001305639744 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 75576 traces tagged "gauss"
FrameSaver: q=2.23473e+06 n=1211866 tag=gauss
wclsFrameSaver: saving 93222 traces tagged "wiener"
FrameSaver: q=2.79964e+06 n=1181604 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 43435 XUs and 44983 XVs -> 1753 XUVs
C:0 T:3 1 XUs and 4 XVs -> 1 XUVs
C:0 T:4 0 XUs and 3 XVs -> 0 XUVs
C:0 T:5 21595 XUs and 25934 XVs -> 1203 XUVs
C:0 T:6 41473 XUs and 61151 XVs -> 3049 XUVs
C:0 T:7 3 XUs and 7 XVs -> 0 XUVs
6006 XUVs total
1646 collection wire objects
6006 potential space points
Neighbour search...
575106 tests to find 323848 neighbours
Iterating with no regularization...
Begin: 1.6878e+07
0 1.56396e+07
1 1.5535e+07
2 1.55131e+07
3 1.55059e+07
Now with regularization...
Begin: 1.43311e+07
0 1.43256e+07
18-Dec-2025 22:39:38 UTC  Closed output file "pdhd_prod_beam__261612_74_1_20251212T160117Z_gen_g4_IonScintPDExt.root_268535_57_1_20251215T215305Z_PDInt.root_263305_23_1_20251218T205121Z_detsim_reco1.root"
18-Dec-2025 22:39:38 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           103.685       117.631       133.021       116.049       10.7788        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00143785     0.003359     0.00561764    0.00335139    0.00122619       10     
reco:wclsdatahd:WireCellToolkit                      55.3798       59.8861       70.5459       58.6107       4.51235        10     
reco:wirefilter:WireFilter                          0.0184773     0.0236938     0.0289285     0.0229894    0.00297274       10     
reco:gaushit:GausHitFinder                          0.405986      0.730162       1.18425       0.70202      0.208801        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000166294   0.000333407   0.000527195   0.000311858   0.00011129       10     
reco:reco3d:SpacePointSolver                        0.213277      0.754343       1.50388      0.622161      0.434941        10     
reco:hitpdune:DisambigFromSpacePoints               0.157739       0.44075       1.01623      0.402301      0.210906        10     
reco:pandora:StandardPandora                         3.82366       8.87865       13.2918       8.48168       2.89943        10     
reco:pandoraWriter:StandardPandora                  0.142073      0.237421      0.493687      0.218667      0.0896834       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.440707       1.03502       1.34137       1.0799       0.266305        10     
reco:pandoraShower:LArPandoraShowerCreation         0.328345       1.10792       1.78594       1.08213      0.486195        10     
reco:pandoracalo:Calorimetry                        0.228483       0.54272      0.798968      0.549083      0.158382        10     
reco:pandoracalonosce:Calorimetry                    0.25278      0.528784      0.683546      0.586161      0.140416        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.326979       1.0996        1.67694       1.09141      0.462845        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.309607       1.08062       1.73252       1.06619      0.472542        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0422039     0.0580837     0.0703122     0.0583884    0.00818644       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0339365     0.055719      0.0714034     0.056924      0.0103297       10     
reco:emtrkmichelid:EmTrackMichelId                   25.4438       39.2964       46.8961       39.5454       6.48866        10     
[art]:TriggerResults:TriggerResultInserter         3.2471e-05    5.04124e-05   0.000146717   3.94545e-05   3.25774e-05      10     
end_path:out1:RootOutput                            7.574e-06    1.26327e-05   4.3883e-05     9.142e-06    1.04558e-05      10     
end_path:out1:RootOutput(write)                     0.923279       1.81606       3.54292       1.52178      0.864973        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5849.24 MB
  Peak resident set size usage (VmHWM): 3689.8 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11098844
-rw-r--r-- 1 dunegli fnalgrid     231454 Dec 18 22:39 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 18 22:39 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 22:39 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 22:39 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 22:39 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 22:39 time.db
-rw-r--r-- 1 dunegli fnalgrid 5756725012 Dec 18 22:39 pdhd_prod_beam__261612_74_1_20251212T160117Z_gen_g4_IonScintPDExt.root_268535_57_1_20251215T215305Z_PDInt.root_263305_23_1_20251218T205121Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   33443650 Dec 18 22:38 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 22:15 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 22:14 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5574584323 Dec 18 22:14 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 18 20:51 all-input-dids.txt
justIN time: 2025-12-18 23:06:52 UTC       justIN version: 01.05.03