justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.19@dunegpschedd01.fnal.gov

Jobsub ID269119.19@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2025-12-18 22:15:54
From worker nodeHostnamemh-7763-13.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 19:31:15
Input filesusertests:pdhd_prod_beam__261612_28_1_20251212T155604Z_gen_g4_IonScintPDExt.root_262727_96_1_20251215T215051Z_PDInt.root
JobscriptExit code0
Real time2h (9672s)
CPU time1h (5364s = 55%)
Max RSS bytes6186024960 (5899 MiB)
Outputting started2025-12-18 22:12:28
Output filesusertests:pdhd_prod_beam__261612_28_1_20251212T155604Z_gen_g4_IonScintPDExt.root_262727_96_1_20251215T215051Z_PDInt.root_269119_19_1_20251218T193121Z_detsim_reco1.root
Finished2025-12-18 22:15:54
Saved logsjustin-logs:269119.19-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] 
[14:11:53.762] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 55800 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":26230 [0] "wiener0":29570 [29570] ] cmm:[ bad:0 lf_noisy:92 ] input 1: frame: ident=179 time=0 tick=512 with 38700 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16735 [0] "wiener1":21965 [21965] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 36753 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15686 [0] "wiener2":21067 [21067] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 32768 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14363 [0] "wiener3":18405 [18405] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 164021 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":26230 [0] "gauss1":16735 [0] "gauss2":15686 [0] "gauss3":14363 [0] "wiener0":29570 [29570] "wiener1":21965 [21965] "wiener2":21067 [21067] "wiener3":18405 [18405] ] cmm:[ bad:0 lf_noisy:92 ] 
Retagger: tagging trace set: wiener with 91007 traces, 91007 summary
Retagger: tagging trace set: gauss with 73014 traces, 0 summary
[14:11:53.774] D [  glue  ] sink frame: #179 @0 with 164021 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[14:11:53.774] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[14:11:53.774] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[14:11:53.774] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[14:11:53.774] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[14:11:53.774] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[14:11:53.774] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[14:11:53.774] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[14:11:53.774] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[14:11:53.774] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[14:11:53.774] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[14:11:53.774] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[14:11:53.775] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[14:11:53.775] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[14:11:53.775] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[14:11:53.775] D [  glue  ] frame sink sees EOS
[14:11:53.775] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.92 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.83 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.67 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.25 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.88 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.79 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.64 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.63 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:11:53.775] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:11:53.775] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:11:53.775] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:11:53.775] I [ timer  ] Timer: Total node execution : 45.63000037148595 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 73014 traces tagged "gauss"
FrameSaver: q=2.27139e+06 n=1138422 tag=gauss
wclsFrameSaver: saving 91007 traces tagged "wiener"
FrameSaver: q=2.90817e+06 n=1106366 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 17905 XUs and 25567 XVs -> 560 XUVs
C:0 T:3 2 XUs and 0 XVs -> 0 XUVs
C:0 T:4 27 XUs and 27 XVs -> 2 XUVs
C:0 T:5 26857 XUs and 25311 XVs -> 906 XUVs
C:0 T:6 9426 XUs and 9023 XVs -> 297 XUVs
C:0 T:7 10 XUs and 6 XVs -> 0 XUVs
1765 XUVs total
1107 collection wire objects
1765 potential space points
Neighbour search...
32811 tests to find 19792 neighbours
Iterating with no regularization...
Begin: 1.31936e+07
0 1.24386e+07
1 1.23496e+07
2 1.23392e+07
Now with regularization...
Begin: 1.14369e+07
0 1.14342e+07
18-Dec-2025 14:12:26 PST  Closed output file "pdhd_prod_beam__261612_28_1_20251212T155604Z_gen_g4_IonScintPDExt.root_262727_96_1_20251215T215051Z_PDInt.root_269119_19_1_20251218T193121Z_detsim_reco1.root"
18-Dec-2025 14:12:26 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           67.6636       76.9504       91.4627       75.5437       6.77056        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000900927   0.00290677    0.00513099    0.00310135    0.00128781       10     
reco:wclsdatahd:WireCellToolkit                      43.5767       46.2278       50.0382       46.4021       1.70855        10     
reco:wirefilter:WireFilter                          0.0118045     0.013668      0.0197052     0.0128261     0.0023133       10     
reco:gaushit:GausHitFinder                          0.315264      0.450077      0.572787      0.432917      0.0833435       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000155688   0.000234964   0.000367904   0.000222692   5.53497e-05      10     
reco:reco3d:SpacePointSolver                        0.134465      0.413804       1.02725      0.296102      0.251812        10     
reco:hitpdune:DisambigFromSpacePoints               0.113879      0.295412      0.706916       0.22118      0.162724        10     
reco:pandora:StandardPandora                         2.65745       5.27021       9.33943       4.72054       1.82306        10     
reco:pandoraWriter:StandardPandora                  0.0846799     0.135143      0.185598      0.134177      0.0295072       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.190314      0.549677       1.03597      0.501161      0.223194        10     
reco:pandoraShower:LArPandoraShowerCreation          0.24723      0.557149       1.09723      0.493159      0.254835        10     
reco:pandoracalo:Calorimetry                        0.134479      0.368547      0.545927      0.346375      0.117453        10     
reco:pandoracalonosce:Calorimetry                   0.131009      0.351598      0.601809       0.32645      0.124874        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.249036      0.584679       1.18265      0.531877      0.263312        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.219649      0.638159       1.52235      0.553465      0.374709        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0224591     0.0449402     0.0913407     0.0379187     0.0186562       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0227062     0.0402021     0.054479      0.0378266     0.0103064       10     
reco:emtrkmichelid:EmTrackMichelId                   13.6354       20.0302       26.0392       20.8772       3.51256        10     
[art]:TriggerResults:TriggerResultInserter          1.685e-05    3.80074e-05   7.7869e-05    3.5699e-05    1.61612e-05      10     
end_path:out1:RootOutput                            7.82e-06     1.24329e-05    2.778e-05    1.1575e-05    5.54776e-06      10     
end_path:out1:RootOutput(write)                      0.64639      0.937638       1.30235      0.872606      0.188614        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5882.02 MB
  Peak resident set size usage (VmHWM): 3635.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11046164
-rw-r--r--. 1 cuser cuser     230854 Dec 18 14:12 jobscript.log
-rw-r--r--. 1 cuser cuser        200 Dec 18 14:12 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser        519 Dec 18 14:12 reco_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser       3574 Dec 18 14:12 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser     139264 Dec 18 14:12 mem.db
-rw-r--r--. 1 cuser cuser      28672 Dec 18 14:12 time.db
-rw-r--r--. 1 cuser cuser 5725675410 Dec 18 14:12 pdhd_prod_beam__261612_28_1_20251212T155604Z_gen_g4_IonScintPDExt.root_262727_96_1_20251215T215051Z_PDInt.root_269119_19_1_20251218T193121Z_detsim_reco1.root
-rw-r--r--. 1 cuser cuser   31920040 Dec 18 14:12 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser          0 Dec 18 13:58 debugprod.log
-rw-r--r--. 1 cuser cuser        519 Dec 18 13:58 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser 5553217042 Dec 18 13:58 temp_detsim.root
-rw-r--r--. 1 cuser cuser        121 Dec 18 11:31 all-input-dids.txt
justIN time: 2025-12-18 22:58:08 UTC       justIN version: 01.05.03