justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263343.0@dunegpschedd02.fnal.gov

Jobsub ID263343.0@dunegpschedd02.fnal.gov
Workflow ID11235
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 23:10:17
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-19 01:20:53
From worker nodeHostnamedunegli-8107706-0-fnpc18002.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 23:10:47
Input filesusertests:pdhd_prod_beam__261613_17_1_20251212T161229Z_gen_g4_IonScintPDExt.root_262728_34_1_20251215T215901Z_PDInt.root
JobscriptExit code0
Real time2h (7718s)
CPU time2h (7324s = 94%)
Max RSS bytes6748348416 (6435 MiB)
Outputting started2025-12-19 01:19:25
Output filesusertests:pdhd_prod_beam__261613_17_1_20251212T161229Z_gen_g4_IonScintPDExt.root_262728_34_1_20251215T215901Z_PDInt.root_263343_0_1_20251218T231053Z_detsim_reco1.root
Finished2025-12-19 01:20:53
Saved logsjustin-logs:263343.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tick=512 with 46703 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":22534 [0] "wiener0":24169 [24169] ] cmm:[ bad:0 lf_noisy:122 ] input 1: frame: ident=199 time=0 tick=512 with 42874 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":18350 [0] "wiener1":24524 [24524] ] cmm:[ bad:0 lf_noisy:3 ] input 2: frame: ident=199 time=0 tick=512 with 36171 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15796 [0] "wiener2":20375 [20375] ] cmm:[ bad:0 ] input 3: frame: ident=199 time=0 tick=512 with 32498 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14668 [0] "wiener3":17830 [17830] ] cmm:[ bad:0 lf_noisy:4 ] output: frame: ident=199 time=0 tick=512 with 158246 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":22534 [0] "gauss1":18350 [0] "gauss2":15796 [0] "gauss3":14668 [0] "wiener0":24169 [24169] "wiener1":24524 [24524] "wiener2":20375 [20375] "wiener3":17830 [17830] ] cmm:[ bad:0 lf_noisy:129 ] 
Retagger: tagging trace set: wiener with 86898 traces, 86898 summary
Retagger: tagging trace set: gauss with 71348 traces, 0 summary
[01:18:34.027] D [  glue  ] sink frame: #199 @0 with 158246 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[01:18:34.028] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[01:18:34.028] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[01:18:34.028] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[01:18:34.028] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[01:18:34.028] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[01:18:34.028] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[01:18:34.028] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[01:18:34.028] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[01:18:34.028] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[01:18:34.028] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[01:18:34.028] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[01:18:34.028] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[01:18:34.028] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[01:18:34.028] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[01:18:34.028] D [  glue  ] frame sink sees EOS
[01:18:34.028] D [ pgraph ] <Pgrapher:> graph execution complete 
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.54 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.47 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.39 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.19 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.67 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.6 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.86 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.49 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[01:18:34.028] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[01:18:34.028] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[01:18:34.028] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[01:18:34.028] I [ timer  ] Timer: Total node execution : 64.26000003702939 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 71348 traces tagged "gauss"
FrameSaver: q=2.35781e+06 n=1136423 tag=gauss
wclsFrameSaver: saving 86898 traces tagged "wiener"
FrameSaver: q=3.07903e+06 n=1124386 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 24399 XUs and 31098 XVs -> 786 XUVs
C:0 T:3 19 XUs and 51 XVs -> 0 XUVs
C:0 T:4 11 XUs and 12 XVs -> 0 XUVs
C:0 T:5 19866 XUs and 16483 XVs -> 1010 XUVs
C:0 T:6 28678 XUs and 55250 XVs -> 2300 XUVs
C:0 T:7 50 XUs and 106 XVs -> 9 XUVs
4105 XUVs total
1526 collection wire objects
4105 potential space points
Neighbour search...
203485 tests to find 90598 neighbours
Iterating with no regularization...
Begin: 3.14608e+07
0 2.58564e+07
1 2.55207e+07
2 2.5493e+07
3 2.5484e+07
Now with regularization...
Begin: 2.40665e+07
0 2.40604e+07
19-Dec-2025 01:19:24 UTC  Closed output file "pdhd_prod_beam__261613_17_1_20251212T161229Z_gen_g4_IonScintPDExt.root_262728_34_1_20251215T215901Z_PDInt.root_263343_0_1_20251218T231053Z_detsim_reco1.root"
19-Dec-2025 01:19:24 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           100.178       111.642       128.636       110.985       8.13629        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00108144    0.00314727    0.00452069    0.00308435    0.00115335       10     
reco:wclsdatahd:WireCellToolkit                      55.9177       62.8443       69.8436       62.4353       4.59774        10     
reco:wirefilter:WireFilter                          0.0199124     0.0245342     0.0317153     0.023634     0.00370259       10     
reco:gaushit:GausHitFinder                          0.511352      0.696888       1.11607       0.68588      0.164396        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000235649   0.000327226   0.000524308   0.000296901   8.35345e-05      10     
reco:reco3d:SpacePointSolver                        0.190992      0.426799      0.837338      0.389041      0.167759        10     
reco:hitpdune:DisambigFromSpacePoints               0.155223      0.350348      0.686508      0.319046      0.139796        10     
reco:pandora:StandardPandora                         4.35408       7.73511       12.8956       7.50372       2.15395        10     
reco:pandoraWriter:StandardPandora                  0.154936      0.212289      0.312431      0.209738      0.0414239       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.542173      0.999119       1.40576       1.03076      0.258402        10     
reco:pandoraShower:LArPandoraShowerCreation          0.50867       1.02654       1.39951       1.02903      0.291178        10     
reco:pandoracalo:Calorimetry                        0.323613       0.50705      0.635718      0.510685      0.0992428       10     
reco:pandoracalonosce:Calorimetry                   0.308632      0.484842       0.64379      0.481507      0.0990795       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.468262      0.926071       1.42368      0.877493      0.298152        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.463084      0.935842       1.37755      0.849986      0.276296        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0451618     0.0575322     0.0900091     0.0532126     0.0134118       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0397739     0.0531686     0.0787379     0.0505021     0.012601        10     
reco:emtrkmichelid:EmTrackMichelId                   24.8866       32.8994       42.8497       33.0972       4.73495        10     
[art]:TriggerResults:TriggerResultInserter          2.696e-05    3.63391e-05   7.8782e-05    2.9823e-05    1.55879e-05      10     
end_path:out1:RootOutput                            6.286e-06    1.09025e-05   2.7657e-05     9.708e-06    5.79125e-06      10     
end_path:out1:RootOutput(write)                     0.981341       1.40732       1.83709       1.41138      0.216803        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5829.73 MB
  Peak resident set size usage (VmHWM): 3659.71 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11525624
-rw-r--r-- 1 dunegli fnalgrid     231165 Dec 19 01:19 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 19 01:19 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 19 01:19 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 19 01:19 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 19 01:19 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 19 01:19 time.db
-rw-r--r-- 1 dunegli fnalgrid 5971118219 Dec 19 01:19 pdhd_prod_beam__261613_17_1_20251212T161229Z_gen_g4_IonScintPDExt.root_262728_34_1_20251215T215901Z_PDInt.root_263343_0_1_20251218T231053Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   31304700 Dec 19 01:18 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 19 00:56 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 19 00:56 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5799350334 Dec 19 00:56 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 18 23:10 all-input-dids.txt
justIN time: 2026-02-07 09:23:00 UTC       justIN version: 01.06.00