justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268971.0@dunegpschedd01.fnal.gov

Jobsub ID268971.0@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 01:38:13
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-18 02:24:34
From worker nodeHostnamefc30554.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 01:39:39
Input filesusertests:pdhd_prod_beam__267379_25_1_20251212T160804Z_gen_g4_IonScintPDExt.root_262726_147_1_20251215T213644Z_PDInt.root
JobscriptExit code0
Real time43m (2635s)
CPU time36m (2171s = 82%)
Max RSS bytes8782389248 (8375 MiB)
Outputting started2025-12-18 02:23:35
Output filesusertests:pdhd_prod_beam__267379_25_1_20251212T160804Z_gen_g4_IonScintPDExt.root_262726_147_1_20251215T213644Z_PDInt.root_268971_0_1_20251218T013942Z_detsim_reco1.root
Finished2025-12-18 02:24:34
Saved logsjustin-logs:268971.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gauss0":19258 [0] "wiener0":20800 [20800] ] cmm:[ bad:0 lf_noisy:239 ] input 1: frame: ident=189 time=0 tick=512 with 56844 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":23659 [0] "wiener1":33185 [33185] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 34656 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15000 [0] "wiener2":19656 [19656] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 46494 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":18825 [0] "wiener3":27669 [27669] ] cmm:[ bad:0 ] output: frame: ident=189 time=0 tick=512 with 178052 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":19258 [0] "gauss1":23659 [0] "gauss2":15000 [0] "gauss3":18825 [0] "wiener0":20800 [20800] "wiener1":33185 [33185] "wiener2":19656 [19656] "wiener3":27669 [27669] ] cmm:[ bad:0 lf_noisy:239 ] 
Retagger: tagging trace set: wiener with 101310 traces, 101310 summary
Retagger: tagging trace set: gauss with 76742 traces, 0 summary
[18:23:18.741] D [  glue  ] sink frame: #189 @0 with 178052 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[18:23:18.741] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[18:23:18.741] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[18:23:18.741] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[18:23:18.741] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[18:23:18.741] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[18:23:18.741] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[18:23:18.741] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[18:23:18.741] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[18:23:18.741] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[18:23:18.741] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[18:23:18.741] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[18:23:18.741] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[18:23:18.741] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[18:23:18.741] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[18:23:18.741] D [  glue  ] frame sink sees EOS
[18:23:18.741] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.83 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.95 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.69 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.63 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.6 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.54 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.49 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.47 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:23:18.741] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[18:23:18.741] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:23:18.741] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:23:18.741] I [ timer  ] Timer: Total node execution : 23.210000166669488 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 76742 traces tagged "gauss"
FrameSaver: q=2.57656e+06 n=1230900 tag=gauss
wclsFrameSaver: saving 101310 traces tagged "wiener"
FrameSaver: q=3.13026e+06 n=1173441 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 43102 XUs and 40927 XVs -> 1024 XUVs
C:0 T:3 29 XUs and 28 XVs -> 0 XUVs
C:0 T:4 4 XUs and 12 XVs -> 0 XUVs
C:0 T:5 60862 XUs and 52274 XVs -> 5202 XUVs
C:0 T:6 38806 XUs and 32798 XVs -> 4055 XUVs
C:0 T:7 35 XUs and 45 XVs -> 4 XUVs
10285 XUVs total
1787 collection wire objects
10285 potential space points
Neighbour search...
2007811 tests to find 872574 neighbours
Iterating with no regularization...
Begin: 5.15901e+07
0 4.46482e+07
1 4.42774e+07
2 4.42129e+07
3 4.41929e+07
Now with regularization...
Begin: 4.23474e+07
0 4.23304e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Dec-2025 18:23:34 PST  Closed output file "pdhd_prod_beam__267379_25_1_20251212T160804Z_gen_g4_IonScintPDExt.root_262726_147_1_20251215T213644Z_PDInt.root_268971_0_1_20251218T013942Z_detsim_reco1.root"
17-Dec-2025 18:23:34 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           27.893        35.7299       45.1639       33.7425       5.08016        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000659166   0.00351672    0.00706245    0.00368657     0.0018227       10     
reco:wclsdatahd:WireCellToolkit                      22.6122       23.3439       24.3414       23.3509       0.5571         10     
reco:wirefilter:WireFilter                          0.0078901     0.0107833     0.013986      0.0105915    0.00184341       10     
reco:gaushit:GausHitFinder                          0.162173      0.259533      0.368514      0.247034      0.0648319       10     
reco:nhitsfilter:NumberOfHitsFilter                9.6269e-05    0.000147705   0.000236549   0.000133135   4.45323e-05      10     
reco:reco3d:SpacePointSolver                        0.0589187      0.32755      0.807577      0.228401      0.251633        10     
reco:hitpdune:DisambigFromSpacePoints               0.0612033     0.180465      0.430035      0.145827      0.100875        10     
reco:pandora:StandardPandora                         1.20474       3.99845       6.70723       3.55655       1.83104        10     
reco:pandoraWriter:StandardPandora                  0.0520943     0.102954      0.150758      0.0957091     0.031104        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.163083      0.444888        1.305       0.357634      0.307183        10     
reco:pandoraShower:LArPandoraShowerCreation         0.149689       0.46303       1.07658      0.368972       0.26894        10     
reco:pandoracalo:Calorimetry                        0.0833458     0.219499      0.517303      0.182935      0.115458        10     
reco:pandoracalonosce:Calorimetry                   0.0807442     0.215666      0.506053      0.183351      0.112261        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.132365      0.440117      0.930758      0.362781      0.245362        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.130575      0.431153      0.909938      0.363976      0.235694        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0124364     0.0204093     0.0302702     0.0189915    0.00507078       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0129031     0.0212197     0.0324498     0.0205386    0.00534302       10     
reco:emtrkmichelid:EmTrackMichelId                   2.56721       4.63446       6.59532       4.17639       1.16254        10     
[art]:TriggerResults:TriggerResultInserter         1.4689e-05    1.82208e-05    4.526e-05     1.519e-05    9.0222e-06       10     
end_path:out1:RootOutput                            3.39e-06      5.377e-06     1.542e-05     3.77e-06     3.51001e-06      10     
end_path:out1:RootOutput(write)                     0.374592      0.566874      0.778749      0.517425       0.12086        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6883.38 MB
  Peak resident set size usage (VmHWM): 4515.72 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12732732
-rw-r--r-- 1 nilic nilic     231446 Dec 17 18:23 jobscript.log
-rw-r--r-- 1 nilic nilic        201 Dec 17 18:23 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 17 18:23 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 17 18:23 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 17 18:23 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 17 18:23 time.db
-rw-r--r-- 1 nilic nilic 6598101791 Dec 17 18:23 pdhd_prod_beam__267379_25_1_20251212T160804Z_gen_g4_IonScintPDExt.root_262726_147_1_20251215T213644Z_PDInt.root_268971_0_1_20251218T013942Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   36673360 Dec 17 18:23 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 17 18:17 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 17 18:16 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 6403084379 Dec 17 18:16 temp_detsim.root
-rw-r--r-- 1 nilic nilic        122 Dec 17 17:39 all-input-dids.txt
justIN time: 2025-12-18 20:22:37 UTC       justIN version: 01.05.03