justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.93@dunegpschedd01.fnal.gov

Jobsub ID269119.93@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_condce_mcore
Last heartbeat2025-12-18 22:23:33
From worker nodeHostnameuct2-c533.mwt2.org
cpuinfoAMD EPYC 7402 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:36:04
Input filesusertests:pdhd_prod_beam__261612_19_1_20251212T155518Z_gen_g4_IonScintPDExt.root_268535_22_1_20251215T214628Z_PDInt.root
JobscriptExit code0
Real time1h (6341s)
CPU time1h (6144s = 96%)
Max RSS bytes6859878400 (6542 MiB)
Outputting started2025-12-18 22:21:45
Output filesusertests:pdhd_prod_beam__261612_19_1_20251212T155518Z_gen_g4_IonScintPDExt.root_268535_22_1_20251215T214628Z_PDInt.root_269119_93_1_20251218T203612Z_detsim_reco1.root
Finished2025-12-18 22:23:33
Saved logsjustin-logs:269119.93-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 55131 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":26317 [0] "wiener0":28814 [28814] ] cmm:[ bad:0 lf_noisy:228 ] input 1: frame: ident=179 time=0 tick=512 with 29848 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13536 [0] "wiener1":16312 [16312] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 34161 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15040 [0] "wiener2":19121 [19121] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 31292 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":13116 [0] "wiener3":18176 [18176] ] cmm:[ bad:0 lf_noisy:1 ] output: frame: ident=179 time=0 tick=512 with 150432 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":26317 [0] "gauss1":13536 [0] "gauss2":15040 [0] "gauss3":13116 [0] "wiener0":28814 [28814] "wiener1":16312 [16312] "wiener2":19121 [19121] "wiener3":18176 [18176] ] cmm:[ bad:0 lf_noisy:229 ] 
Retagger: tagging trace set: wiener with 82423 traces, 82423 summary
Retagger: tagging trace set: gauss with 68009 traces, 0 summary
[16:21:12.889] D [  glue  ] sink frame: #179 @0 with 150432 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[16:21:12.890] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[16:21:12.890] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[16:21:12.890] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[16:21:12.890] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[16:21:12.890] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[16:21:12.890] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[16:21:12.890] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[16:21:12.890] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[16:21:12.890] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[16:21:12.890] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[16:21:12.890] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[16:21:12.890] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[16:21:12.890] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[16:21:12.890] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[16:21:12.890] D [  glue  ] frame sink sees EOS
[16:21:12.890] D [ pgraph ] <Pgrapher:> graph execution complete 
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.77 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.41 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.29 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.85 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.28 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.24 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.2 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.78 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:21:12.890] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[16:21:12.890] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[16:21:12.890] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:21:12.890] I [ timer  ] Timer: Total node execution : 58.88000064715743 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 68009 traces tagged "gauss"
FrameSaver: q=1.79743e+06 n=962244 tag=gauss
wclsFrameSaver: saving 82423 traces tagged "wiener"
FrameSaver: q=2.59055e+06 n=964058 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 9511 XUs and 17353 XVs -> 279 XUVs
C:0 T:3 12 XUs and 26 XVs -> 0 XUVs
C:0 T:4 3 XUs and 13 XVs -> 0 XUVs
C:0 T:5 11084 XUs and 10689 XVs -> 496 XUVs
C:0 T:6 17123 XUs and 17966 XVs -> 407 XUVs
1182 XUVs total
761 collection wire objects
1182 potential space points
Neighbour search...
13664 tests to find 8212 neighbours
Iterating with no regularization...
Begin: 1.7203e+07
0 1.66452e+07
1 1.65979e+07
2 1.65927e+07
Now with regularization...
Begin: 1.54326e+07
0 1.54309e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Dec-2025 16:21:44 CST  Closed output file "pdhd_prod_beam__261612_19_1_20251212T155518Z_gen_g4_IonScintPDExt.root_268535_22_1_20251215T214628Z_PDInt.root_269119_93_1_20251218T203612Z_detsim_reco1.root"
18-Dec-2025 16:21:44 CST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           88.2059       113.273       141.608       112.925       16.7812        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00138576    0.00480788    0.00735887    0.00486579    0.00205839       10     
reco:wclsdatahd:WireCellToolkit                      55.6592       64.4045       72.0257       63.907        4.61884        10     
reco:wirefilter:WireFilter                          0.0199214     0.0325693     0.057618      0.029392      0.0111037       10     
reco:gaushit:GausHitFinder                          0.427231      0.666134       0.84672       0.65287      0.145484        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000195117   0.000452945   0.000823227   0.000376427   0.000187845      10     
reco:reco3d:SpacePointSolver                        0.238648      0.918291       2.92015      0.466721      0.870436        10     
reco:hitpdune:DisambigFromSpacePoints               0.165766      0.490622       1.00164       0.54638      0.264351        10     
reco:pandora:StandardPandora                         3.88084       9.57085       16.0088       10.2556       4.21008        10     
reco:pandoraWriter:StandardPandora                  0.152134      0.260483      0.401149      0.261647      0.0741049       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.366266       1.10281       1.9341        1.05035      0.471506        10     
reco:pandoraShower:LArPandoraShowerCreation         0.356072       1.08279        2.099        1.17948      0.535262        10     
reco:pandoracalo:Calorimetry                        0.261766      0.610941       1.0648       0.613932      0.218865        10     
reco:pandoracalonosce:Calorimetry                   0.256759      0.644316       1.03338      0.653854      0.244576        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.354255       1.09274       1.64755       1.07594      0.448302        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.337047       1.07603       2.02328       1.05303      0.533293        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.044611      0.0667711     0.0924243     0.0695872     0.0174534       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0442475     0.0649688     0.0860998     0.0698246     0.0150896       10     
reco:emtrkmichelid:EmTrackMichelId                   20.7377       29.6215       40.2032       30.7514       6.5052         10     
[art]:TriggerResults:TriggerResultInserter         3.0748e-05    7.57915e-05   0.00023421    6.0504e-05    5.56494e-05      10     
end_path:out1:RootOutput                            4.328e-06    1.04525e-05   4.8401e-05     5.15e-06     1.28538e-05      10     
end_path:out1:RootOutput(write)                      1.01601       1.47969       1.79736       1.54299      0.282182        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5883.7 MB
  Peak resident set size usage (VmHWM): 3693.17 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11938644
-rw-r--r--. 1 dune osgvo     231280 Dec 18 16:21 jobscript.log
-rw-r--r--. 1 dune osgvo        200 Dec 18 16:21 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Dec 18 16:21 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Dec 18 16:21 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     139264 Dec 18 16:21 mem.db
-rw-r--r--. 1 dune osgvo      28672 Dec 18 16:21 time.db
-rw-r--r--. 1 dune osgvo 6189076072 Dec 18 16:21 pdhd_prod_beam__261612_19_1_20251212T155518Z_gen_g4_IonScintPDExt.root_268535_22_1_20251215T214628Z_PDInt.root_269119_93_1_20251218T203612Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   34280090 Dec 18 16:21 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Dec 18 16:00 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Dec 18 16:00 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 6001367696 Dec 18 16:00 temp_detsim.root
-rw-r--r--. 1 dune osgvo        121 Dec 18 14:36 all-input-dids.txt
justIN time: 2025-12-19 07:35:57 UTC       justIN version: 01.05.03