justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268771.1@dunegpschedd01.fnal.gov

Jobsub ID268771.1@dunegpschedd01.fnal.gov
Workflow ID11235
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 02:37:02
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2025-12-17 04:38:38
From worker nodeHostnamecompute-1-1.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-17 02:39:01
Input filesusertests:pdhd_prod_beam__261613_103_1_20251212T161700Z_gen_g4_IonScintPDExt.root_262728_76_1_20251215T220101Z_PDInt.root
JobscriptExit code0
Real time1h (7011s)
CPU time1h (6697s = 95%)
Max RSS bytes6730158080 (6418 MiB)
Outputting started2025-12-17 04:35:53
Output filesusertests:pdhd_prod_beam__261613_103_1_20251212T161700Z_gen_g4_IonScintPDExt.root_262728_76_1_20251215T220101Z_PDInt.root_268771_1_1_20251217T023906Z_detsim_reco1.root
Finished2025-12-17 04:38:38
Saved logsjustin-logs:268771.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2 [0] "wiener0":31147 [31147] ] cmm:[ bad:0 lf_noisy:173 ] input 1: frame: ident=199 time=0 tick=512 with 54043 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":22897 [0] "wiener1":31146 [31146] ] cmm:[ bad:0 lf_noisy:1 ] input 2: frame: ident=199 time=0 tick=512 with 33377 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":14568 [0] "wiener2":18809 [18809] ] cmm:[ bad:0 ] input 3: frame: ident=199 time=0 tick=512 with 55333 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":23196 [0] "wiener3":32137 [32137] ] cmm:[ bad:0 ] output: frame: ident=199 time=0 tick=512 with 201622 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":27722 [0] "gauss1":22897 [0] "gauss2":14568 [0] "gauss3":23196 [0] "wiener0":31147 [31147] "wiener1":31146 [31146] "wiener2":18809 [18809] "wiener3":32137 [32137] ] cmm:[ bad:0 lf_noisy:174 ] 
Retagger: tagging trace set: wiener with 113239 traces, 113239 summary
Retagger: tagging trace set: gauss with 88383 traces, 0 summary
[00:33:57.285] D [  glue  ] sink frame: #199 @0 with 201622 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[00:33:57.285] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[00:33:57.285] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[00:33:57.285] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[00:33:57.285] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[00:33:57.285] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[00:33:57.285] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[00:33:57.285] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[00:33:57.285] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[00:33:57.285] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[00:33:57.285] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[00:33:57.285] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[00:33:57.285] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[00:33:57.285] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[00:33:57.285] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[00:33:57.285] D [  glue  ] frame sink sees EOS
[00:33:57.285] D [ pgraph ] <Pgrapher:> graph execution complete 
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 21.94 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.9 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.38 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.07 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.01 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.98 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.9 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:33:57.286] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[00:33:57.286] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[00:33:57.286] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[00:33:57.286] I [ timer  ] Timer: Total node execution : 82.24000078067183 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 88383 traces tagged "gauss"
FrameSaver: q=2.74289e+06 n=1398713 tag=gauss
wclsFrameSaver: saving 113239 traces tagged "wiener"
FrameSaver: q=3.51105e+06 n=1335958 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 55416 XUs and 56670 XVs -> 1348 XUVs
C:0 T:3 9 XUs and 12 XVs -> 0 XUVs
C:0 T:4 14 XUs and 12 XVs -> 7 XUVs
C:0 T:5 10672 XUs and 7981 XVs -> 326 XUVs
C:0 T:6 70574 XUs and 53457 XVs -> 1433 XUVs
C:0 T:7 45 XUs and 25 XVs -> 2 XUVs
3116 XUVs total
1703 collection wire objects
3116 potential space points
Neighbour search...
84518 tests to find 49010 neighbours
Iterating with no regularization...
Begin: 1.71859e+07
0 1.61647e+07
1 1.60995e+07
2 1.60907e+07
Now with regularization...
Begin: 1.49657e+07
0 1.49618e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Dec-2025 00:35:51 AST  Closed output file "pdhd_prod_beam__261613_103_1_20251212T161700Z_gen_g4_IonScintPDExt.root_262728_76_1_20251215T220101Z_PDInt.root_268771_1_1_20251217T023906Z_detsim_reco1.root"
17-Dec-2025 00:35:51 AST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           99.6699       155.97        196.071       154.766       30.9225        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000711681   0.00338784    0.00555366    0.00364598    0.00151545       10     
reco:wclsdatahd:WireCellToolkit                      62.123        74.4106       83.284        75.3841       6.52645        10     
reco:wirefilter:WireFilter                          0.0154044     0.0218674     0.0267296     0.022313      0.0039966       10     
reco:gaushit:GausHitFinder                          0.400067      0.853556       1.39954      0.831991      0.281029        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000129064   0.000326767   0.000526516   0.000346927   0.000119629      10     
reco:reco3d:SpacePointSolver                        0.0893099     0.518761      0.927998      0.561269      0.260553        10     
reco:hitpdune:DisambigFromSpacePoints               0.178681      0.678921       1.24325      0.817388      0.344799        10     
reco:pandora:StandardPandora                         2.67706       11.3041       22.0904       9.84291       6.69502        10     
reco:pandoraWriter:StandardPandora                   0.11071      0.191698      0.268683      0.203951      0.0528953       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.291134       1.20003       1.79467       1.36311      0.516431        10     
reco:pandoraShower:LArPandoraShowerCreation         0.209248       1.12151       2.91492       1.08956       0.73592        10     
reco:pandoracalo:Calorimetry                        0.201195      0.609662      0.898006      0.682807      0.235439        10     
reco:pandoracalonosce:Calorimetry                   0.198956      0.594398      0.879674      0.667107      0.229213        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.205857       1.08483       2.67472       1.10263      0.674101        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.202558       1.07297       2.63042       1.09581      0.663936        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0311582     0.0529849     0.0708065     0.0573897     0.0140791       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0298977     0.0526523     0.0711043     0.0581957     0.014324        10     
reco:emtrkmichelid:EmTrackMichelId                   31.9324       60.9383       81.0128       63.7466       17.5213        10     
[art]:TriggerResults:TriggerResultInserter         1.9823e-05    2.46905e-05   6.0295e-05    2.1067e-05    1.18769e-05      10     
end_path:out1:RootOutput                            4.015e-06    5.4791e-06    1.7063e-05     4.233e-06    3.86345e-06      10     
end_path:out1:RootOutput(write)                     0.751211       1.22091       1.66461       1.31264      0.280822        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5823.34 MB
  Peak resident set size usage (VmHWM): 3437.88 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11264040
-rw-r--r-- 1 dunepilot gridusers     231211 Dec 17 00:35 jobscript.log
-rw-r--r-- 1 dunepilot gridusers        201 Dec 17 00:35 justin-processed-pfns.txt
-rw-r--r-- 1 dunepilot gridusers        519 Dec 17 00:35 reco_protodunehd_hist.root
-rw-r--r-- 1 dunepilot gridusers       3574 Dec 17 00:35 Pandora_Geometry.xml
-rw-r--r-- 1 dunepilot gridusers     139264 Dec 17 00:35 mem.db
-rw-r--r-- 1 dunepilot gridusers      28672 Dec 17 00:35 time.db
-rw-r--r-- 1 dunepilot gridusers 5843798719 Dec 17 00:35 pdhd_prod_beam__261613_103_1_20251212T161700Z_gen_g4_IonScintPDExt.root_262728_76_1_20251215T220101Z_PDInt.root_268771_1_1_20251217T023906Z_detsim_reco1.root
-rw-r--r-- 1 dunepilot gridusers   35376900 Dec 17 00:34 Pandora_Events.pndr
-rw-r--r-- 1 dunepilot gridusers          0 Dec 17 00:07 debugprod.log
-rw-r--r-- 1 dunepilot gridusers        519 Dec 17 00:07 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunepilot gridusers 5654756046 Dec 17 00:07 temp_detsim.root
-rw-r--r-- 1 dunepilot gridusers        122 Dec 16 22:39 all-input-dids.txt
justIN time: 2025-12-19 21:58:36 UTC       justIN version: 01.05.03