justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.29@dunegpschedd02.fnal.gov

Jobsub ID263305.29@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-18 21:41:08
From worker nodeHostnamefc30554.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 21:01:24
Input filesusertests:pdhd_prod_beam__261612_21_1_20251212T155537Z_gen_g4_IonScintPDExt.root_268535_2_1_20251215T213100Z_PDInt.root
JobscriptExit code0
Real time38m (2333s)
CPU time35m (2128s = 91%)
Max RSS bytes6717759488 (6406 MiB)
Outputting started2025-12-18 21:40:18
Output filesusertests:pdhd_prod_beam__261612_21_1_20251212T155537Z_gen_g4_IonScintPDExt.root_268535_2_1_20251215T213100Z_PDInt.root_263305_29_1_20251218T210127Z_detsim_reco1.root
Finished2025-12-18 21:41:08
Saved logsjustin-logs:263305.29-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

:[ bad:0 lf_noisy:279 ] 
[13:39:53.813] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 71410 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":33372 [0] "wiener0":38038 [38038] ] cmm:[ bad:0 lf_noisy:279 ] input 1: frame: ident=179 time=0 tick=512 with 53335 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":22533 [0] "wiener1":30802 [30802] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 43334 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":18628 [0] "wiener2":24706 [24706] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 53002 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":22581 [0] "wiener3":30421 [30421] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 221081 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":33372 [0] "gauss1":22533 [0] "gauss2":18628 [0] "gauss3":22581 [0] "wiener0":38038 [38038] "wiener1":30802 [30802] "wiener2":24706 [24706] "wiener3":30421 [30421] ] cmm:[ bad:0 lf_noisy:279 ] 
Retagger: tagging trace set: wiener with 123967 traces, 123967 summary
Retagger: tagging trace set: gauss with 97114 traces, 0 summary
[13:39:53.834] D [  glue  ] sink frame: #179 @0 with 221081 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[13:39:53.834] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[13:39:53.834] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[13:39:53.834] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[13:39:53.834] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[13:39:53.834] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[13:39:53.834] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[13:39:53.834] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[13:39:53.834] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[13:39:53.834] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[13:39:53.834] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[13:39:53.834] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[13:39:53.834] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[13:39:53.834] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[13:39:53.834] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[13:39:53.834] D [  glue  ] frame sink sees EOS
[13:39:53.834] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.98 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.55 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.01 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.79 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.87 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.76 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.71 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.66 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:39:53.834] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:39:53.834] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:39:53.834] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:39:53.834] I [ timer  ] Timer: Total node execution : 30.380000399425626 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 97114 traces tagged "gauss"
FrameSaver: q=3.14917e+06 n=1564739 tag=gauss
wclsFrameSaver: saving 123967 traces tagged "wiener"
FrameSaver: q=4.09933e+06 n=1504017 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 57878 XUs and 72342 XVs -> 4239 XUVs
C:0 T:3 6 XUs and 7 XVs -> 3 XUVs
C:0 T:5 29225 XUs and 51827 XVs -> 526 XUVs
C:0 T:6 54691 XUs and 77905 XVs -> 6213 XUVs
C:0 T:7 5 XUs and 10 XVs -> 0 XUVs
10981 XUVs total
2491 collection wire objects
10981 potential space points
Neighbour search...
1893683 tests to find 879632 neighbours
Iterating with no regularization...
Begin: 3.52978e+07
0 3.11219e+07
1 3.0816e+07
2 3.07561e+07
3 3.07347e+07
Now with regularization...
Begin: 2.82963e+07
0 2.82804e+07
18-Dec-2025 13:40:17 PST  Closed output file "pdhd_prod_beam__261612_21_1_20251212T155537Z_gen_g4_IonScintPDExt.root_268535_2_1_20251215T213100Z_PDInt.root_263305_29_1_20251218T210127Z_detsim_reco1.root"
18-Dec-2025 13:40:17 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           36.6907       43.517        53.4898       44.0653       4.22396        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00105669    0.00468702     0.0090664    0.00406225    0.00273989       10     
reco:wclsdatahd:WireCellToolkit                      29.3948       30.5253       31.608        30.6258      0.654907        10     
reco:wirefilter:WireFilter                          0.0097119     0.0213899      0.03571      0.0218625    0.00765539       10     
reco:gaushit:GausHitFinder                          0.156954       0.26273      0.387127      0.260204      0.0541398       10     
reco:nhitsfilter:NumberOfHitsFilter                 9.308e-05    0.000489046   0.00102691    0.000466662   0.000327279      10     
reco:reco3d:SpacePointSolver                        0.149405      0.339091       1.00363       0.22731      0.252129        10     
reco:hitpdune:DisambigFromSpacePoints               0.048675       0.1557       0.332735      0.147166      0.0717789       10     
reco:pandora:StandardPandora                         1.52764       3.83195       7.98672       3.5149        1.66471        10     
reco:pandoraWriter:StandardPandora                  0.0832454     0.151119      0.196875      0.165909      0.0378274       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.283593      0.646963       1.03634      0.667185      0.250161        10     
reco:pandoraShower:LArPandoraShowerCreation         0.112152       0.48627       0.69119      0.482597      0.164064        10     
reco:pandoracalo:Calorimetry                        0.106155      0.261481       0.41894      0.254349      0.097028        10     
reco:pandoracalonosce:Calorimetry                   0.108192      0.258432      0.396214       0.26127      0.0852434       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.0983048     0.391976      0.654155       0.40561      0.136848        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.0937619     0.413291      0.823381      0.411267      0.175973        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0124132     0.0272892     0.0417309     0.027307     0.00723038       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0134524     0.0273383     0.0385173     0.0258686    0.00691366       10     
reco:emtrkmichelid:EmTrackMichelId                   2.78597       5.00151       7.44548       4.91918       1.14984        10     
[art]:TriggerResults:TriggerResultInserter          1.567e-05    2.62278e-05    7.179e-05    2.20795e-05   1.5527e-05       10     
end_path:out1:RootOutput                            4.89e-06      7.812e-06     2.112e-05     5.905e-06    4.7725e-06       10     
end_path:out1:RootOutput(write)                     0.369087      0.628537      0.964089      0.627185      0.147493        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6340.19 MB
  Peak resident set size usage (VmHWM): 3967.8 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11388392
-rw-r--r-- 1 nilic nilic     231159 Dec 18 13:40 jobscript.log
-rw-r--r-- 1 nilic nilic        199 Dec 18 13:40 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:40 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 18 13:40 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 18 13:40 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 18 13:40 time.db
-rw-r--r-- 1 nilic nilic 5902456127 Dec 18 13:40 pdhd_prod_beam__261612_21_1_20251212T155537Z_gen_g4_IonScintPDExt.root_268535_2_1_20251215T213100Z_PDInt.root_263305_29_1_20251218T210127Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   31894190 Dec 18 13:40 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 18 13:32 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:32 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 5726905102 Dec 18 13:32 temp_detsim.root
-rw-r--r-- 1 nilic nilic        120 Dec 18 13:01 all-input-dids.txt
justIN time: 2025-12-18 23:06:47 UTC       justIN version: 01.05.03