justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.13@dunegpschedd02.fnal.gov

Jobsub ID263305.13@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-18 21:10:34
From worker nodeHostnamefc30555.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:30:02
Input filesusertests:pdhd_prod_beam__261612_62_1_20251212T155908Z_gen_g4_IonScintPDExt.root_268535_9_1_20251215T213715Z_PDInt.root
JobscriptExit code0
Real time39m (2384s)
CPU time35m (2151s = 90%)
Max RSS bytes7153852416 (6822 MiB)
Outputting started2025-12-18 21:09:47
Output filesusertests:pdhd_prod_beam__261612_62_1_20251212T155908Z_gen_g4_IonScintPDExt.root_268535_9_1_20251215T213715Z_PDInt.root_263305_13_1_20251218T203004Z_detsim_reco1.root
Finished2025-12-18 21:10:34
Saved logsjustin-logs:263305.13-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] "wiener0":23145 [23145] ] cmm:[ bad:0 lf_noisy:229 ] 
[13:09:33.396] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 43931 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":20786 [0] "wiener0":23145 [23145] ] cmm:[ bad:0 lf_noisy:229 ] input 1: frame: ident=179 time=0 tick=512 with 51325 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":21512 [0] "wiener1":29813 [29813] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 33572 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15520 [0] "wiener2":18052 [18052] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 40494 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":17439 [0] "wiener3":23055 [23055] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 169322 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":20786 [0] "gauss1":21512 [0] "gauss2":15520 [0] "gauss3":17439 [0] "wiener0":23145 [23145] "wiener1":29813 [29813] "wiener2":18052 [18052] "wiener3":23055 [23055] ] cmm:[ bad:0 lf_noisy:229 ] 
Retagger: tagging trace set: wiener with 94065 traces, 94065 summary
Retagger: tagging trace set: gauss with 75257 traces, 0 summary
[13:09:33.401] D [  glue  ] sink frame: #179 @0 with 169322 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[13:09:33.402] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[13:09:33.402] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[13:09:33.402] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[13:09:33.402] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[13:09:33.402] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[13:09:33.402] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[13:09:33.402] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[13:09:33.402] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[13:09:33.402] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[13:09:33.402] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[13:09:33.402] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[13:09:33.402] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[13:09:33.402] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[13:09:33.402] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[13:09:33.402] D [  glue  ] frame sink sees EOS
[13:09:33.402] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.92 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.24 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.97 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.96 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.71 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.66 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.62 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.61 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:09:33.402] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[13:09:33.402] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[13:09:33.402] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:09:33.402] I [ timer  ] Timer: Total node execution : 24.68999993801117 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 75257 traces tagged "gauss"
FrameSaver: q=2.31086e+06 n=1232422 tag=gauss
wclsFrameSaver: saving 94065 traces tagged "wiener"
FrameSaver: q=2.78212e+06 n=1187063 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 54175 XUs and 40376 XVs -> 1591 XUVs
C:0 T:3 31 XUs and 21 XVs -> 2 XUVs
C:0 T:4 11 XUs and 3 XVs -> 0 XUVs
C:0 T:5 11063 XUs and 9628 XVs -> 443 XUVs
C:0 T:6 42510 XUs and 37570 XVs -> 535 XUVs
2571 XUVs total
1531 collection wire objects
2571 potential space points
Neighbour search...
40723 tests to find 23982 neighbours
Iterating with no regularization...
Begin: 1.83824e+07
0 1.7612e+07
1 1.75733e+07
2 1.75684e+07
Now with regularization...
Begin: 1.60185e+07
0 1.60165e+07
18-Dec-2025 13:09:46 PST  Closed output file "pdhd_prod_beam__261612_62_1_20251212T155908Z_gen_g4_IonScintPDExt.root_268535_9_1_20251215T213715Z_PDInt.root_263305_13_1_20251218T203004Z_detsim_reco1.root"
18-Dec-2025 13:09:46 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           35.3161       38.7375       42.4492       38.4154       2.40819        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000503779    0.0040681    0.00701747    0.00491076    0.00210801       10     
reco:wclsdatahd:WireCellToolkit                      24.9676       25.3116       25.8107       25.253       0.275012        10     
reco:wirefilter:WireFilter                         0.00910914     0.0108917     0.0137643     0.0106434    0.00134365       10     
reco:gaushit:GausHitFinder                          0.217034      0.291093      0.376369      0.289047      0.0426807       10     
reco:nhitsfilter:NumberOfHitsFilter                0.00011585    0.000157155   0.000325359   0.000122555   6.35348e-05      10     
reco:reco3d:SpacePointSolver                        0.123796      0.288766      0.555373      0.268776      0.128726        10     
reco:hitpdune:DisambigFromSpacePoints               0.105778      0.180126      0.255785      0.168642      0.0412201       10     
reco:pandora:StandardPandora                         2.63669       4.16842       5.79243       4.19285       1.07693        10     
reco:pandoraWriter:StandardPandora                  0.0913844     0.130102      0.159977      0.132572      0.0203072       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.338321      0.493631      0.592523      0.497061      0.0762942       10     
reco:pandoraShower:LArPandoraShowerCreation         0.345434      0.502845      0.804209      0.481536      0.128567        10     
reco:pandoracalo:Calorimetry                        0.183111      0.255479       0.32621      0.251292      0.0415031       10     
reco:pandoracalonosce:Calorimetry                   0.168405      0.242377      0.307887      0.233687      0.0410491       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.329266      0.472532      0.746383      0.469975      0.116783        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.324063      0.467689       0.74576      0.453905      0.116119        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0182717     0.0234778     0.0283571     0.0235252    0.00301104       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0197805     0.0243499     0.0283808     0.0248874    0.00256635       10     
reco:emtrkmichelid:EmTrackMichelId                   4.43497       5.18476       6.05169       4.99646      0.574229        10     
[art]:TriggerResults:TriggerResultInserter          1.403e-05    2.2045e-05     6.376e-05    1.6825e-05    1.41128e-05      10     
end_path:out1:RootOutput                            3.57e-06      6.97e-06      1.982e-05     5.865e-06    4.40026e-06      10     
end_path:out1:RootOutput(write)                     0.534557      0.626238      0.750305      0.615455      0.0693879       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6495.51 MB
  Peak resident set size usage (VmHWM): 4133.42 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12609088
-rw-r--r-- 1 nilic nilic     231587 Dec 18 13:09 jobscript.log
-rw-r--r-- 1 nilic nilic        199 Dec 18 13:09 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:09 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 18 13:09 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 18 13:09 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 18 13:09 time.db
-rw-r--r-- 1 nilic nilic 6536147662 Dec 18 13:09 pdhd_prod_beam__261612_62_1_20251212T155908Z_gen_g4_IonScintPDExt.root_268535_9_1_20251215T213715Z_PDInt.root_263305_13_1_20251218T203004Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   37311470 Dec 18 13:09 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 18 13:02 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:02 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 6337787784 Dec 18 13:02 temp_detsim.root
-rw-r--r-- 1 nilic nilic        120 Dec 18 12:30 all-input-dids.txt
justIN time: 2025-12-18 23:05:59 UTC       justIN version: 01.05.03