justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.16@dunegpschedd02.fnal.gov

Jobsub ID263305.16@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-18 22:22:56
From worker nodeHostnamedunegli-8108563-0-fnpc19140.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:31:36
Input filesusertests:pdhd_prod_beam__261612_112_1_20251212T160644Z_gen_g4_IonScintPDExt.root_262727_23_1_20251215T214400Z_PDInt.root
JobscriptExit code0
Real time1h (6611s)
CPU time1h (6527s = 98%)
Max RSS bytes7794831360 (7433 MiB)
Outputting started2025-12-18 22:21:47
Output filesusertests:pdhd_prod_beam__261612_112_1_20251212T160644Z_gen_g4_IonScintPDExt.root_262727_23_1_20251215T214400Z_PDInt.root_263305_16_1_20251218T203141Z_detsim_reco1.root
Finished2025-12-18 22:22:56
Saved logsjustin-logs:263305.16-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

bad:0 lf_noisy:83 ] 
[22:21:22.307] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 41248 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":19602 [0] "wiener0":21646 [21646] ] cmm:[ bad:0 lf_noisy:83 ] input 1: frame: ident=179 time=0 tick=512 with 40767 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16881 [0] "wiener1":23886 [23886] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 32979 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":14643 [0] "wiener2":18336 [18336] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 35215 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15108 [0] "wiener3":20107 [20107] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 150209 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":19602 [0] "gauss1":16881 [0] "gauss2":14643 [0] "gauss3":15108 [0] "wiener0":21646 [21646] "wiener1":23886 [23886] "wiener2":18336 [18336] "wiener3":20107 [20107] ] cmm:[ bad:0 lf_noisy:83 ] 
Retagger: tagging trace set: wiener with 83975 traces, 83975 summary
Retagger: tagging trace set: gauss with 66234 traces, 0 summary
[22:21:22.316] D [  glue  ] sink frame: #179 @0 with 150209 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[22:21:22.316] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[22:21:22.316] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[22:21:22.316] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[22:21:22.316] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[22:21:22.316] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[22:21:22.316] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[22:21:22.316] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[22:21:22.316] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[22:21:22.316] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[22:21:22.316] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[22:21:22.316] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[22:21:22.316] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[22:21:22.316] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[22:21:22.316] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[22:21:22.317] D [  glue  ] frame sink sees EOS
[22:21:22.317] D [ pgraph ] <Pgrapher:> graph execution complete 
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.61 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.74 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.59 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.28 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.49 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.47 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.47 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.37 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:21:22.317] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[22:21:22.317] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[22:21:22.317] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[22:21:22.317] I [ timer  ] Timer: Total node execution : 39.04999974183738 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 66234 traces tagged "gauss"
FrameSaver: q=1.96698e+06 n=1002637 tag=gauss
wclsFrameSaver: saving 83975 traces tagged "wiener"
FrameSaver: q=2.50677e+06 n=973935 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 24610 XUs and 36277 XVs -> 771 XUVs
C:0 T:4 37 XUs and 11 XVs -> 1 XUVs
C:0 T:5 17492 XUs and 8543 XVs -> 307 XUVs
C:0 T:6 19981 XUs and 24324 XVs -> 623 XUVs
1702 XUVs total
1107 collection wire objects
1702 potential space points
Neighbour search...
19558 tests to find 11708 neighbours
Iterating with no regularization...
Begin: 2.14014e+07
0 2.08604e+07
1 2.08289e+07
2 2.0827e+07
Now with regularization...
Begin: 1.95669e+07
0 1.95665e+07
18-Dec-2025 22:21:46 UTC  Closed output file "pdhd_prod_beam__261612_112_1_20251212T160644Z_gen_g4_IonScintPDExt.root_262727_23_1_20251215T214400Z_PDInt.root_263305_16_1_20251218T203141Z_detsim_reco1.root"
18-Dec-2025 22:21:46 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           63.2605       85.3939       129.422       75.4692       20.3875        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.0012713    0.00354876    0.00556599    0.00380729    0.00130436       10     
reco:wclsdatahd:WireCellToolkit                       40.08        48.0571       60.0511       45.0209       7.46388        10     
reco:wirefilter:WireFilter                          0.0130852     0.0216954     0.0327193     0.0201715    0.00721905       10     
reco:gaushit:GausHitFinder                          0.431578      0.661171       1.15554      0.523695      0.261363        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000178402    0.000305     0.000586755   0.000247292   0.000126502      10     
reco:reco3d:SpacePointSolver                        0.204674      0.513259       1.54683      0.320165      0.439604        10     
reco:hitpdune:DisambigFromSpacePoints               0.203604      0.441322       1.18149      0.288409      0.309594        10     
reco:pandora:StandardPandora                          3.87         9.73129       25.5186       6.82814       6.5028         10     
reco:pandoraWriter:StandardPandora                  0.129386      0.209667      0.402103      0.164365      0.0858705       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.458084      0.984188       2.15814      0.832518      0.543041        10     
reco:pandoraShower:LArPandoraShowerCreation         0.378161       1.03741       2.35316      0.714315      0.665698        10     
reco:pandoracalo:Calorimetry                        0.246674      0.502975       1.04186      0.403419      0.264118        10     
reco:pandoracalonosce:Calorimetry                   0.231877      0.508097       1.07092      0.387788      0.281945        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.385659       1.0326        2.70664       0.72305      0.734619        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.366508       1.0253        2.6571       0.709928      0.727682        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0308184     0.0548658     0.101551      0.0420163     0.0260467       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0308261     0.0534774     0.101047      0.0388679     0.0258844       10     
reco:emtrkmichelid:EmTrackMichelId                   13.5709       19.2154       29.014        17.5487       4.49161        10     
[art]:TriggerResults:TriggerResultInserter          2.47e-05     3.70074e-05   9.5331e-05    3.0445e-05    1.99001e-05      10     
end_path:out1:RootOutput                            7.49e-06     1.08941e-05    2.979e-05      8.4e-06     6.48494e-06      10     
end_path:out1:RootOutput(write)                     0.865444       1.2841        2.14237       1.07845      0.423002        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5790.15 MB
  Peak resident set size usage (VmHWM): 3623.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12128752
-rw-r--r-- 1 dunegli fnalgrid     231307 Dec 18 22:21 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Dec 18 22:21 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 22:21 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 22:21 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 22:21 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 22:21 time.db
-rw-r--r-- 1 dunegli fnalgrid 6291247344 Dec 18 22:21 pdhd_prod_beam__261612_112_1_20251212T160644Z_gen_g4_IonScintPDExt.root_262727_23_1_20251215T214400Z_PDInt.root_263305_16_1_20251218T203141Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   37325440 Dec 18 22:21 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 22:05 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 22:05 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6090814015 Dec 18 22:05 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Dec 18 20:31 all-input-dids.txt
justIN time: 2025-12-18 23:05:58 UTC       justIN version: 01.05.03