justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.15@dunegpschedd02.fnal.gov

Jobsub ID263305.15@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-18 22:13:23
From worker nodeHostnamedunegli-8108563-0-fnpc19140.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:31:03
Input filesusertests:pdhd_prod_beam__267378_8_1_20251212T160702Z_gen_g4_IonScintPDExt.root_262727_87_1_20251215T214937Z_PDInt.root
JobscriptExit code0
Real time1h (6072s)
CPU time1h (5961s = 98%)
Max RSS bytes7845601280 (7482 MiB)
Outputting started2025-12-18 22:12:16
Output filesusertests:pdhd_prod_beam__267378_8_1_20251212T160702Z_gen_g4_IonScintPDExt.root_262727_87_1_20251215T214937Z_PDInt.root_263305_15_1_20251218T203108Z_detsim_reco1.root
Finished2025-12-18 22:13:23
Saved logsjustin-logs:263305.15-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 49179 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":23080 [0] "wiener0":26099 [26099] ] cmm:[ bad:0 lf_noisy:289 ] input 1: frame: ident=179 time=0 tick=512 with 39123 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":17136 [0] "wiener1":21987 [21987] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 58089 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":25074 [0] "wiener2":33015 [33015] ] cmm:[ bad:0 lf_noisy:5 ] input 3: frame: ident=179 time=0 tick=512 with 38167 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16606 [0] "wiener3":21561 [21561] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 184558 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23080 [0] "gauss1":17136 [0] "gauss2":25074 [0] "gauss3":16606 [0] "wiener0":26099 [26099] "wiener1":21987 [21987] "wiener2":33015 [33015] "wiener3":21561 [21561] ] cmm:[ bad:0 lf_noisy:294 ] 
Retagger: tagging trace set: wiener with 102662 traces, 102662 summary
Retagger: tagging trace set: gauss with 81896 traces, 0 summary
[22:11:37.958] D [  glue  ] sink frame: #179 @0 with 184558 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[22:11:37.958] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[22:11:37.958] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[22:11:37.958] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[22:11:37.958] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[22:11:37.958] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[22:11:37.958] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[22:11:37.958] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[22:11:37.958] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[22:11:37.958] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[22:11:37.958] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[22:11:37.958] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[22:11:37.958] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[22:11:37.958] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[22:11:37.958] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[22:11:37.958] D [  glue  ] frame sink sees EOS
[22:11:37.958] D [ pgraph ] <Pgrapher:> graph execution complete 
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.97 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.16 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.62 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.61 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.46 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.45 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.41 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.38 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[22:11:37.958] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[22:11:37.958] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[22:11:37.958] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[22:11:37.958] I [ timer  ] Timer: Total node execution : 44.09000041894615 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 81896 traces tagged "gauss"
FrameSaver: q=2.43244e+06 n=1320422 tag=gauss
wclsFrameSaver: saving 102662 traces tagged "wiener"
FrameSaver: q=2.93745e+06 n=1257073 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 22634 XUs and 20231 XVs -> 533 XUVs
C:0 T:4 32 XUs and 20 XVs -> 0 XUVs
C:0 T:5 85068 XUs and 60679 XVs -> 1510 XUVs
C:0 T:6 26816 XUs and 76611 XVs -> 1014 XUVs
C:0 T:7 25 XUs and 8 XVs -> 0 XUVs
3057 XUVs total
1882 collection wire objects
3057 potential space points
Neighbour search...
56457 tests to find 31010 neighbours
Iterating with no regularization...
Begin: 3.51718e+07
0 2.98714e+07
1 2.96799e+07
2 2.96643e+07
Now with regularization...
Begin: 2.85888e+07
0 2.85836e+07
18-Dec-2025 22:12:14 UTC  Closed output file "pdhd_prod_beam__267378_8_1_20251212T160702Z_gen_g4_IonScintPDExt.root_262727_87_1_20251215T214937Z_PDInt.root_263305_15_1_20251218T203108Z_detsim_reco1.root"
18-Dec-2025 22:12:14 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           58.4194       72.0066       94.3269       68.1409       11.3919        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000762637   0.00296193    0.00552772    0.00288321    0.00145636       10     
reco:wclsdatahd:WireCellToolkit                      38.7875       41.9447       45.2713       42.1292       2.17568        10     
reco:wirefilter:WireFilter                          0.0107372     0.0150208     0.0216601     0.0138759    0.00372477       10     
reco:gaushit:GausHitFinder                          0.295807      0.462875      0.660659      0.464409      0.109119        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000120781   0.000224802   0.000361783   0.000225442   7.71913e-05      10     
reco:reco3d:SpacePointSolver                        0.158197       0.4263       0.869798      0.326404      0.259915        10     
reco:hitpdune:DisambigFromSpacePoints               0.143199      0.313284      0.619715      0.275241      0.160383        10     
reco:pandora:StandardPandora                         3.13942       6.89309       19.4954       6.12298       4.54766        10     
reco:pandoraWriter:StandardPandora                  0.104412      0.157294      0.235125      0.153258      0.0441493       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.266748      0.684199       1.28987      0.608281      0.342377        10     
reco:pandoraShower:LArPandoraShowerCreation         0.237087      0.746082       1.80364      0.587108       0.49072        10     
reco:pandoracalo:Calorimetry                         0.16044      0.364088       0.65764      0.340128       0.16031        10     
reco:pandoracalonosce:Calorimetry                   0.156322       0.34903      0.630354       0.32714       0.15375        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.215821      0.677255       1.61529      0.538966      0.424334        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.216262      0.669621       1.49164      0.560692      0.399691        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0279346     0.0406939     0.0600052     0.0407509    0.00980077       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0278235     0.0399457     0.0587761     0.0391163    0.00937547       10     
reco:emtrkmichelid:EmTrackMichelId                   11.4708       17.1717       24.6439       16.8478       4.27762        10     
[art]:TriggerResults:TriggerResultInserter          1.744e-05    3.08413e-05    7.095e-05     2.735e-05    1.39586e-05      10     
end_path:out1:RootOutput                            6.88e-06      9.465e-06     2.026e-05     8.46e-06     3.67202e-06      10     
end_path:out1:RootOutput(write)                     0.714273      0.998953       1.38674      0.972585      0.215328        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6145.48 MB
  Peak resident set size usage (VmHWM): 3995.71 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12103184
-rw-r--r-- 1 dunegli fnalgrid     231355 Dec 18 22:12 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        199 Dec 18 22:12 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 22:12 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 22:12 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 22:12 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 22:12 time.db
-rw-r--r-- 1 dunegli fnalgrid 6273933452 Dec 18 22:12 pdhd_prod_beam__267378_8_1_20251212T160702Z_gen_g4_IonScintPDExt.root_262727_87_1_20251215T214937Z_PDInt.root_263305_15_1_20251218T203108Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   34616470 Dec 18 22:11 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 21:58 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:58 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6084650186 Dec 18 21:58 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Dec 18 20:31 all-input-dids.txt
justIN time: 2025-12-18 23:06:54 UTC       justIN version: 01.05.03