justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.34@dunegpschedd02.fnal.gov

Jobsub ID263305.34@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-18 21:52:00
From worker nodeHostnamedunegli-7951944-0-fnpc23110.fnal.gov
cpuinfoAMD EPYC 9354 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 21:01:58
Input filesusertests:pdhd_prod_beam__267378_18_1_20251212T160754Z_gen_g4_IonScintPDExt.root_262727_84_1_20251215T214923Z_PDInt.root
JobscriptExit code0
Real time49m (2947s)
CPU time48m (2886s = 97%)
Max RSS bytes6429761536 (6131 MiB)
Outputting started2025-12-18 21:51:06
Output filesusertests:pdhd_prod_beam__267378_18_1_20251212T160754Z_gen_g4_IonScintPDExt.root_262727_84_1_20251215T214923Z_PDInt.root_263305_34_1_20251218T210201Z_detsim_reco1.root
Finished2025-12-18 21:51:52
Saved logsjustin-logs:263305.34-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 46297 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":21868 [0] "wiener0":24429 [24429] ] cmm:[ bad:0 lf_noisy:232 ] input 1: frame: ident=179 time=0 tick=512 with 33066 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":14622 [0] "wiener1":18444 [18444] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 29534 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":13384 [0] "wiener2":16150 [16150] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 45208 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":19401 [0] "wiener3":25807 [25807] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 154105 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":21868 [0] "gauss1":14622 [0] "gauss2":13384 [0] "gauss3":19401 [0] "wiener0":24429 [24429] "wiener1":18444 [18444] "wiener2":16150 [16150] "wiener3":25807 [25807] ] cmm:[ bad:0 lf_noisy:232 ] 
Retagger: tagging trace set: wiener with 84830 traces, 84830 summary
Retagger: tagging trace set: gauss with 69275 traces, 0 summary
[21:50:52.317] D [  glue  ] sink frame: #179 @0 with 154105 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[21:50:52.317] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[21:50:52.317] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[21:50:52.317] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[21:50:52.317] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[21:50:52.318] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[21:50:52.318] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[21:50:52.318] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[21:50:52.318] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[21:50:52.318] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[21:50:52.318] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[21:50:52.318] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[21:50:52.318] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[21:50:52.318] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[21:50:52.318] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[21:50:52.318] D [  glue  ] frame sink sees EOS
[21:50:52.318] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.6 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.61 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.46 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.43 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.82 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.8 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.8 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.79 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:50:52.318] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[21:50:52.318] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:50:52.318] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:50:52.318] I [ timer  ] Timer: Total node execution : 27.329999823123217 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 69275 traces tagged "gauss"
FrameSaver: q=1.89267e+06 n=1037611 tag=gauss
wclsFrameSaver: saving 84830 traces tagged "wiener"
FrameSaver: q=2.40018e+06 n=1008221 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 32826 XUs and 23319 XVs -> 544 XUVs
C:0 T:4 22 XUs and 25 XVs -> 2 XUVs
C:0 T:5 10867 XUs and 8840 XVs -> 352 XUVs
C:0 T:6 28768 XUs and 46841 XVs -> 2228 XUVs
C:0 T:7 16 XUs and 24 XVs -> 4 XUVs
3130 XUVs total
1092 collection wire objects
3130 potential space points
Neighbour search...
216034 tests to find 123972 neighbours
Iterating with no regularization...
Begin: 2.18796e+07
0 2.06734e+07
1 2.05915e+07
2 2.05714e+07
Now with regularization...
Begin: 1.9471e+07
0 1.94635e+07
18-Dec-2025 21:51:05 UTC  Closed output file "pdhd_prod_beam__267378_18_1_20251212T160754Z_gen_g4_IonScintPDExt.root_262727_84_1_20251215T214923Z_PDInt.root_263305_34_1_20251218T210201Z_detsim_reco1.root"
18-Dec-2025 21:51:05 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           37.6338       41.8939       47.3622       42.6213       2.65787        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000355762    0.0030935    0.00509798    0.00357383    0.00154015       10     
reco:wclsdatahd:WireCellToolkit                      27.7715       28.0628       28.4916       27.9814      0.240817        10     
reco:wirefilter:WireFilter                         0.00659593    0.00813575    0.00927183    0.00820929    0.000728781      10     
reco:gaushit:GausHitFinder                          0.202206      0.260998      0.343069       0.26352      0.0429673       10     
reco:nhitsfilter:NumberOfHitsFilter                6.9206e-05    9.94839e-05   0.000217973   8.84395e-05   4.09482e-05      10     
reco:reco3d:SpacePointSolver                        0.0824804      0.1901       0.392147      0.165115      0.0868862       10     
reco:hitpdune:DisambigFromSpacePoints               0.0762958     0.142311       0.26174      0.132902      0.0474535       10     
reco:pandora:StandardPandora                         1.94948       3.01341       4.44411       3.23834      0.751601        10     
reco:pandoraWriter:StandardPandora                  0.0551463     0.0781735     0.101008      0.0754752     0.0136387       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.200319      0.318486      0.493293      0.310931      0.089093        10     
reco:pandoraShower:LArPandoraShowerCreation         0.150967      0.340821      0.582795      0.344477      0.116478        10     
reco:pandoracalo:Calorimetry                        0.125059      0.200661      0.299458      0.202507      0.0539851       10     
reco:pandoracalonosce:Calorimetry                   0.120323      0.194652      0.288494      0.195331      0.0524149       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.147897      0.348076       0.58654      0.362153      0.122995        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.144248      0.344513      0.576515      0.362264      0.121653        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0143853     0.0197859     0.0270248     0.0203865    0.00329884       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0138654     0.0187426     0.0260351     0.0187926    0.00313732       10     
reco:emtrkmichelid:EmTrackMichelId                   5.9922        7.71656       10.1683       7.92941       1.18643        10     
[art]:TriggerResults:TriggerResultInserter          1.332e-05    2.19825e-05   7.0216e-05    1.6886e-05    1.62429e-05      10     
end_path:out1:RootOutput                            3.546e-06    7.3641e-06     1.991e-05    6.3545e-06    4.27175e-06      10     
end_path:out1:RootOutput(write)                     0.476277      0.613641      0.750623      0.600753      0.0810782       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6153.97 MB
  Peak resident set size usage (VmHWM): 3779.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10931232
-rw-r--r-- 1 dunegli fnalgrid     230973 Dec 18 21:51 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 18 21:51 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:51 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 21:51 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 21:51 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 21:51 time.db
-rw-r--r-- 1 dunegli fnalgrid 5666835241 Dec 18 21:51 pdhd_prod_beam__267378_18_1_20251212T160754Z_gen_g4_IonScintPDExt.root_262727_84_1_20251215T214923Z_PDInt.root_263305_34_1_20251218T210201Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   31225610 Dec 18 21:50 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 21:43 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:43 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5495062395 Dec 18 21:43 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 18 21:02 all-input-dids.txt
justIN time: 2025-12-18 23:06:47 UTC       justIN version: 01.05.03