justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263382.0@dunegpschedd02.fnal.gov

Jobsub ID263382.0@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-19 07:30:40
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-19 08:05:29
From worker nodeHostnamefc30554.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-19 07:31:30
Input filesusertests:pdhd_prod_beam__261612_20_1_20251212T155517Z_gen_g4_IonScintPDExt.root_268535_24_1_20251215T214602Z_PDInt.root
JobscriptExit code0
Real time33m (1996s)
CPU time28m (1728s = 86%)
Max RSS bytes6715768832 (6404 MiB)
Outputting started2025-12-19 08:04:47
Output filesusertests:pdhd_prod_beam__261612_20_1_20251212T155517Z_gen_g4_IonScintPDExt.root_268535_24_1_20251215T214602Z_PDInt.root_263382_0_1_20251219T073132Z_detsim_reco1.root
Finished2025-12-19 08:05:29
Saved logsjustin-logs:263382.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

all=18 input 0: frame: ident=179 time=0 tick=512 with 65692 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":30926 [0] "wiener0":34766 [34766] ] cmm:[ bad:0 lf_noisy:39 ] input 1: frame: ident=179 time=0 tick=512 with 37173 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":15873 [0] "wiener1":21300 [21300] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 55910 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":23031 [0] "wiener2":32879 [32879] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 34138 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15042 [0] "wiener3":19096 [19096] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 192913 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":30926 [0] "gauss1":15873 [0] "gauss2":23031 [0] "gauss3":15042 [0] "wiener0":34766 [34766] "wiener1":21300 [21300] "wiener2":32879 [32879] "wiener3":19096 [19096] ] cmm:[ bad:0 lf_noisy:39 ] 
Retagger: tagging trace set: wiener with 108041 traces, 108041 summary
Retagger: tagging trace set: gauss with 84872 traces, 0 summary
[00:04:36.003] D [  glue  ] sink frame: #179 @0 with 192913 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[00:04:36.003] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[00:04:36.003] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[00:04:36.003] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[00:04:36.003] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[00:04:36.003] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[00:04:36.003] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[00:04:36.003] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[00:04:36.003] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[00:04:36.003] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[00:04:36.003] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[00:04:36.003] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[00:04:36.003] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[00:04:36.003] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[00:04:36.003] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[00:04:36.003] D [  glue  ] frame sink sees EOS
[00:04:36.003] D [ pgraph ] <Pgrapher:> graph execution complete 
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.28 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.55 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.37 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.31 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.42 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.41 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.41 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.4 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[00:04:36.003] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[00:04:36.003] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[00:04:36.003] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[00:04:36.003] I [ timer  ] Timer: Total node execution : 21.169999856501818 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 84872 traces tagged "gauss"
FrameSaver: q=2.63639e+06 n=1271772 tag=gauss
wclsFrameSaver: saving 108041 traces tagged "wiener"
FrameSaver: q=3.58856e+06 n=1250166 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 10673 XUs and 29950 XVs -> 1405 XUVs
C:0 T:3 5 XUs and 4 XVs -> 0 XUVs
C:0 T:4 81 XUs and 313 XVs -> 0 XUVs
C:0 T:5 80119 XUs and 118131 XVs -> 8828 XUVs
C:0 T:6 10089 XUs and 18978 XVs -> 546 XUVs
10779 XUVs total
2053 collection wire objects
10779 potential space points
Neighbour search...
2169341 tests to find 932202 neighbours
Iterating with no regularization...
Begin: 2.74282e+07
0 2.23304e+07
1 2.20388e+07
2 2.19871e+07
3 2.19707e+07
Now with regularization...
Begin: 2.06653e+07
0 2.06523e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
19-Dec-2025 00:04:46 PST  Closed output file "pdhd_prod_beam__261612_20_1_20251212T155517Z_gen_g4_IonScintPDExt.root_268535_24_1_20251215T214602Z_PDInt.root_263382_0_1_20251219T073132Z_detsim_reco1.root"
19-Dec-2025 00:04:46 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           29.0127       32.8116       36.9686       32.964        2.2236         10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000440816   0.00333703    0.00535251     0.0037421    0.00171862       10     
reco:wclsdatahd:WireCellToolkit                      22.1726       22.4965       22.9788       22.476       0.223434        10     
reco:wirefilter:WireFilter                         0.00665393    0.00825934    0.00999962    0.00832893    0.00103666       10     
reco:gaushit:GausHitFinder                          0.153035      0.229418      0.281475      0.227703      0.0417918       10     
reco:nhitsfilter:NumberOfHitsFilter                 8.181e-05    0.000131829   0.000226738   0.000131219   3.93634e-05      10     
reco:reco3d:SpacePointSolver                        0.0756607      0.27025      0.711016      0.241788      0.167363        10     
reco:hitpdune:DisambigFromSpacePoints               0.0601126     0.145393      0.199185      0.147784      0.0402666       10     
reco:pandora:StandardPandora                         1.35706       3.05948       4.37156       3.28017      0.831753        10     
reco:pandoraWriter:StandardPandora                  0.0576828     0.0820784     0.108899      0.0770391     0.0184289       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.154419       0.3307       0.581356       0.30901      0.117562        10     
reco:pandoraShower:LArPandoraShowerCreation         0.166524      0.348001      0.600005      0.348418      0.134331        10     
reco:pandoracalo:Calorimetry                        0.100779      0.181032      0.285094      0.173332      0.0534895       10     
reco:pandoracalonosce:Calorimetry                   0.0939301     0.175486      0.277224      0.167527      0.0519605       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.136493      0.335984      0.583226      0.344882      0.131388        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.133767      0.329167      0.545422      0.337138      0.126159        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.012574      0.0174405     0.0225233     0.0175039    0.00325155       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0130395     0.0168624     0.0225068     0.0168907    0.00323388       10     
reco:emtrkmichelid:EmTrackMichelId                   3.27582       4.23102       5.19848       4.20498      0.658473        10     
[art]:TriggerResults:TriggerResultInserter          1.335e-05    2.14489e-05   5.7169e-05     1.764e-05    1.22616e-05      10     
end_path:out1:RootOutput                            4.04e-06      7.618e-06     1.747e-05     5.96e-06     3.7366e-06       10     
end_path:out1:RootOutput(write)                      0.43624      0.531762      0.638469      0.519088      0.0762575       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6402.2 MB
  Peak resident set size usage (VmHWM): 4048.03 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11773308
-rw-r--r-- 1 nilic nilic     231289 Dec 19 00:04 jobscript.log
-rw-r--r-- 1 nilic nilic        200 Dec 19 00:04 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 19 00:04 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 19 00:04 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 19 00:04 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 19 00:04 time.db
-rw-r--r-- 1 nilic nilic 6103533823 Dec 19 00:04 pdhd_prod_beam__261612_20_1_20251212T155517Z_gen_g4_IonScintPDExt.root_268535_24_1_20251215T214602Z_PDInt.root_263382_0_1_20251219T073132Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   34214970 Dec 19 00:04 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 18 23:58 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 18 23:58 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 5917659395 Dec 18 23:58 temp_detsim.root
-rw-r--r-- 1 nilic nilic        121 Dec 18 23:31 all-input-dids.txt
justIN time: 2025-12-19 09:27:54 UTC       justIN version: 01.05.03