justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.20@dunegpschedd02.fnal.gov

Jobsub ID263305.20@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-18 21:20:39
From worker nodeHostnamefc30563.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:47:57
Input filesusertests:pdhd_prod_beam__261612_100_1_20251212T160347Z_gen_g4_IonScintPDExt.root_268535_47_1_20251215T215135Z_PDInt.root
JobscriptExit code0
Real time32m (1921s)
CPU time28m (1720s = 89%)
Max RSS bytes7210524672 (6876 MiB)
Outputting started2025-12-18 21:19:59
Output filesusertests:pdhd_prod_beam__261612_100_1_20251212T160347Z_gen_g4_IonScintPDExt.root_268535_47_1_20251215T215135Z_PDInt.root_263305_20_1_20251218T204759Z_detsim_reco1.root
Finished2025-12-18 21:20:39
Saved logsjustin-logs:263305.20-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oisy:227 ] 
[13:19:52.898] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 46137 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":21706 [0] "wiener0":24431 [24431] ] cmm:[ bad:0 lf_noisy:227 ] input 1: frame: ident=179 time=0 tick=512 with 34770 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":15013 [0] "wiener1":19757 [19757] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 37652 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":16938 [0] "wiener2":20714 [20714] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 20713 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":9957 [0] "wiener3":10756 [10756] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 139272 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":21706 [0] "gauss1":15013 [0] "gauss2":16938 [0] "gauss3":9957 [0] "wiener0":24431 [24431] "wiener1":19757 [19757] "wiener2":20714 [20714] "wiener3":10756 [10756] ] cmm:[ bad:0 lf_noisy:227 ] 
Retagger: tagging trace set: wiener with 75658 traces, 75658 summary
Retagger: tagging trace set: gauss with 63614 traces, 0 summary
[13:19:52.902] D [  glue  ] sink frame: #179 @0 with 139272 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[13:19:52.902] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[13:19:52.902] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[13:19:52.902] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[13:19:52.902] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[13:19:52.902] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[13:19:52.902] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[13:19:52.902] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[13:19:52.902] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[13:19:52.902] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[13:19:52.902] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[13:19:52.902] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[13:19:52.902] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[13:19:52.902] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[13:19:52.902] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[13:19:52.902] D [  glue  ] frame sink sees EOS
[13:19:52.902] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.89 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.24 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.22 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.12 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.42 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.42 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.42 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.4 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[13:19:52.902] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:19:52.902] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:19:52.902] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:19:52.902] I [ timer  ] Timer: Total node execution : 20.139999637380242 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 63614 traces tagged "gauss"
FrameSaver: q=1.67275e+06 n=924364 tag=gauss
wclsFrameSaver: saving 75658 traces tagged "wiener"
FrameSaver: q=2.18015e+06 n=918081 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 12600 XUs and 17213 XVs -> 995 XUVs
C:0 T:3 4 XUs and 11 XVs -> 0 XUVs
C:0 T:5 42558 XUs and 17095 XVs -> 765 XUVs
C:0 T:6 1822 XUs and 8356 XVs -> 44 XUVs
1804 XUVs total
1018 collection wire objects
1804 potential space points
Neighbour search...
40024 tests to find 21790 neighbours
Iterating with no regularization...
Begin: 1.73856e+07
0 1.61058e+07
1 1.60205e+07
2 1.60108e+07
Now with regularization...
Begin: 1.49462e+07
0 1.49428e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Dec-2025 13:19:58 PST  Closed output file "pdhd_prod_beam__261612_100_1_20251212T160347Z_gen_g4_IonScintPDExt.root_268535_47_1_20251215T215135Z_PDInt.root_263305_20_1_20251218T204759Z_detsim_reco1.root"
18-Dec-2025 13:19:58 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           26.0339       30.7462       40.555        30.2686       3.66604        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000242988   0.00318659    0.00548566    0.00344935    0.00172693       10     
reco:wclsdatahd:WireCellToolkit                      20.6119       21.2409       22.5355       21.0609      0.556691        10     
reco:wirefilter:WireFilter                         0.00590669    0.00722202    0.00921841    0.00710123    0.000944209      10     
reco:gaushit:GausHitFinder                           0.15142      0.216181      0.302345      0.206088      0.0426323       10     
reco:nhitsfilter:NumberOfHitsFilter                 5.592e-05    9.54809e-05   0.00017779     8.65e-05     3.38828e-05      10     
reco:reco3d:SpacePointSolver                        0.0984553     0.184585      0.451807      0.144506      0.108311        10     
reco:hitpdune:DisambigFromSpacePoints               0.0696779     0.123594      0.289432      0.104877      0.0599263       10     
reco:pandora:StandardPandora                         1.40618       3.00113       7.97928       2.40839       1.7278         10     
reco:pandoraWriter:StandardPandora                  0.0449067     0.0708822     0.110016      0.0698901     0.0164655       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.164438      0.313229      0.567567      0.296847      0.125545        10     
reco:pandoraShower:LArPandoraShowerCreation         0.138196      0.332913      0.907362      0.286934      0.208421        10     
reco:pandoracalo:Calorimetry                        0.0824623     0.162491      0.281445      0.161086      0.0550022       10     
reco:pandoracalonosce:Calorimetry                   0.0786893     0.157747      0.276167      0.155872      0.0537042       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.112881      0.310247      0.787803      0.287678      0.181324        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.110322      0.307473      0.779843      0.287201      0.179603        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry         0.00997268     0.0152383     0.0226875     0.0152204    0.00340518       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry    0.00989263     0.0146064      0.02124      0.0147497    0.00310165       10     
reco:emtrkmichelid:EmTrackMichelId                   2.55545       3.79369       5.90596       3.72012      0.830191        10     
[art]:TriggerResults:TriggerResultInserter          1.256e-05    2.03169e-05    5.603e-05    1.57595e-05   1.21312e-05      10     
end_path:out1:RootOutput                            4.01e-06     6.9119e-06    1.8629e-05     5.92e-06     3.94764e-06      10     
end_path:out1:RootOutput(write)                     0.352344      0.477209       0.67772      0.474892      0.0833238       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6270.02 MB
  Peak resident set size usage (VmHWM): 3881.41 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11626820
-rw-r--r-- 1 nilic nilic     230984 Dec 18 13:19 jobscript.log
-rw-r--r-- 1 nilic nilic        201 Dec 18 13:19 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:19 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 18 13:19 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 18 13:19 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 18 13:19 time.db
-rw-r--r-- 1 nilic nilic 6023928301 Dec 18 13:19 pdhd_prod_beam__261612_100_1_20251212T160347Z_gen_g4_IonScintPDExt.root_268535_47_1_20251215T215135Z_PDInt.root_263305_20_1_20251218T204759Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   32097800 Dec 18 13:19 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 18 13:14 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:14 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 5849381378 Dec 18 13:14 temp_detsim.root
-rw-r--r-- 1 nilic nilic        122 Dec 18 12:47 all-input-dids.txt
justIN time: 2025-12-18 23:05:59 UTC       justIN version: 01.05.03