justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263113.1@dunegpschedd02.fnal.gov

Jobsub ID263113.1@dunegpschedd02.fnal.gov
Workflow ID11225
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 20:03:57
SiteUS_UChicago
EntryEngage_US_MWT2_uiuc_condce_mcore
Last heartbeat2025-12-17 22:34:54
From worker nodeHostnamemwt2-c164.campuscluster.illinois.edu
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-17 20:05:11
Input filesusertests:pdhd_prod_beam__265003_29_1_20251208T175516Z_gen_g4_IonScintPDExt.root_266014_12_1_20251210T115905Z_PDInt.root
JobscriptExit code0
Real time2h (8858s)
CPU time2h (8612s = 97%)
Max RSS bytes7526916096 (7178 MiB)
Outputting started2025-12-17 22:32:50
Output filesusertests:pdhd_prod_beam__265003_29_1_20251208T175516Z_gen_g4_IonScintPDExt.root_266014_12_1_20251210T115905Z_PDInt.root_263113_1_1_20251217T200524Z_detsim_reco1.root
Finished2025-12-17 22:34:54
Saved logsjustin-logs:263113.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ner0":26149 [26149] ] cmm:[ bad:0 lf_noisy:185 ] 
[16:31:58.950] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=129 time=0 tick=512 with 50410 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":24261 [0] "wiener0":26149 [26149] ] cmm:[ bad:0 lf_noisy:185 ] input 1: frame: ident=129 time=0 tick=512 with 35294 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":14979 [0] "wiener1":20315 [20315] ] cmm:[ bad:0 ] input 2: frame: ident=129 time=0 tick=512 with 39689 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":17283 [0] "wiener2":22406 [22406] ] cmm:[ bad:0 ] input 3: frame: ident=129 time=0 tick=512 with 33362 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14435 [0] "wiener3":18927 [18927] ] cmm:[ bad:0 ] output: frame: ident=129 time=0 tick=512 with 158755 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":24261 [0] "gauss1":14979 [0] "gauss2":17283 [0] "gauss3":14435 [0] "wiener0":26149 [26149] "wiener1":20315 [20315] "wiener2":22406 [22406] "wiener3":18927 [18927] ] cmm:[ bad:0 lf_noisy:185 ] 
Retagger: tagging trace set: wiener with 87797 traces, 87797 summary
Retagger: tagging trace set: gauss with 70958 traces, 0 summary
[16:31:58.985] D [  glue  ] sink frame: #129 @0 with 158755 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[16:31:58.985] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[16:31:58.985] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[16:31:58.985] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[16:31:58.985] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[16:31:58.986] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[16:31:58.986] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[16:31:58.986] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[16:31:58.986] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[16:31:58.986] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[16:31:58.986] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[16:31:58.986] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[16:31:58.986] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[16:31:58.986] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[16:31:58.986] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[16:31:58.986] D [  glue  ] frame sink sees EOS
[16:31:58.986] D [ pgraph ] <Pgrapher:> graph execution complete 
[16:31:58.986] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 19.21 sec
[16:31:58.986] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.68 sec
[16:31:58.986] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.62 sec
[16:31:58.986] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.91 sec
[16:31:58.986] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.12 sec
[16:31:58.986] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.96 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.91 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.8 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.03 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:31:58.987] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[16:31:58.987] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[16:31:58.987] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:31:58.987] I [ timer  ] Timer: Total node execution : 71.30999908223748 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 70958 traces tagged "gauss"
FrameSaver: q=2.2297e+06 n=1106985 tag=gauss
wclsFrameSaver: saving 87797 traces tagged "wiener"
FrameSaver: q=2.77457e+06 n=1075113 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 21732 XUs and 17813 XVs -> 748 XUVs
C:0 T:3 7 XUs and 45 XVs -> 0 XUVs
C:0 T:4 37 XUs and 23 XVs -> 0 XUVs
C:0 T:5 20371 XUs and 18510 XVs -> 808 XUVs
C:0 T:6 11885 XUs and 19520 XVs -> 482 XUVs
2038 XUVs total
1345 collection wire objects
2038 potential space points
Neighbour search...
28646 tests to find 16152 neighbours
Iterating with no regularization...
Begin: 3.12883e+07
0 2.97218e+07
1 2.96684e+07
2 2.96644e+07
Now with regularization...
Begin: 2.79725e+07
0 2.79705e+07
17-Dec-2025 16:32:47 CST  Closed output file "pdhd_prod_beam__265003_29_1_20251208T175516Z_gen_g4_IonScintPDExt.root_266014_12_1_20251210T115905Z_PDInt.root_263113_1_1_20251217T200524Z_detsim_reco1.root"
17-Dec-2025 16:32:47 CST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           109.316       144.005       172.304       147.997       19.8441        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00153227     0.0048107    0.00680687    0.00493736    0.00162552       10     
reco:wclsdatahd:WireCellToolkit                      73.6557       81.932        94.0459       80.3825       7.0038         10     
reco:wirefilter:WireFilter                          0.0354329     0.0546545     0.0808567     0.0544326     0.0124985       10     
reco:gaushit:GausHitFinder                          0.529442       0.89204       1.20692      0.955449      0.222745        10     
reco:nhitsfilter:NumberOfHitsFilter                0.00032984    0.000545963   0.000693057   0.000581111   0.000122564      10     
reco:reco3d:SpacePointSolver                         0.46813      0.878858       1.83265      0.706809      0.427083        10     
reco:hitpdune:DisambigFromSpacePoints               0.408539      0.817464       1.32887       0.77186      0.312318        10     
reco:pandora:StandardPandora                         5.75114       11.4714       16.8604       11.6124       3.53673        10     
reco:pandoraWriter:StandardPandora                   0.23127       0.34042      0.460644      0.342932      0.0764499       10     
reco:pandoraTrack:LArPandoraTrackCreation            1.02627       2.26509       3.79548        2.078       0.867998        10     
reco:pandoraShower:LArPandoraShowerCreation          1.13244       2.37119       3.88429       2.1601       0.892899        10     
reco:pandoracalo:Calorimetry                        0.677873       1.11762       1.65788       1.05031      0.376609        10     
reco:pandoracalonosce:Calorimetry                   0.538877       1.10601       1.70251       1.08349      0.354306        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.802688       2.0587        3.26936       2.0613       0.873272        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.778537       2.24116       4.65858       1.89056       1.19401        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0445398     0.0768281     0.125898      0.0731852     0.0289265       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0451497     0.0731225     0.112207      0.0678773     0.0248774       10     
reco:emtrkmichelid:EmTrackMichelId                   18.6442       34.4293       51.2195       31.5406       10.3907        10     
[art]:TriggerResults:TriggerResultInserter         4.3498e-05    8.76137e-05   0.000209562   8.27025e-05   4.32343e-05      10     
end_path:out1:RootOutput                           1.0364e-05    1.71034e-05   4.8658e-05    1.31535e-05   1.0802e-05       10     
end_path:out1:RootOutput(write)                      1.2483        1.77514       2.38844       1.71924      0.340676        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6291.67 MB
  Peak resident set size usage (VmHWM): 4099.14 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12220964
-rw-r--r-- 1 dune osgvo     231430 Dec 17 16:32 jobscript.log
-rw-r--r-- 1 dune osgvo        200 Dec 17 16:32 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Dec 17 16:32 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Dec 17 16:32 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     139264 Dec 17 16:32 mem.db
-rw-r--r-- 1 dune osgvo      28672 Dec 17 16:32 time.db
-rw-r--r-- 1 dune osgvo 6337372617 Dec 17 16:32 pdhd_prod_beam__265003_29_1_20251208T175516Z_gen_g4_IonScintPDExt.root_266014_12_1_20251210T115905Z_PDInt.root_263113_1_1_20251217T200524Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   36390220 Dec 17 16:32 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Dec 17 16:05 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Dec 17 16:05 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 6140055384 Dec 17 16:05 temp_detsim.root
-rw-r--r-- 1 dune osgvo        121 Dec 17 14:05 all-input-dids.txt
justIN time: 2025-12-18 20:20:55 UTC       justIN version: 01.05.03