justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258266.0@dunegpschedd02.fnal.gov

Jobsub ID258266.0@dunegpschedd02.fnal.gov
Workflow ID11026
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 23:00:12
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:07:50
From worker nodeHostnamedunegli-7989276-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:44
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T171704Z_gen_001009_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (30090s)
CPU time7h (28709s = 95%)
Max RSS bytes3069992960 (2927 MiB)
Outputting started2025-12-06 10:00:14
Output files
Finished2025-12-06 10:07:50
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ier summary: 
Output 0: 0.139178, 
Output 1: 0.0681559, 0.0572544, 0.137458, 0.737132, 
Output 2: 0.0758002, 0.253913, 0.63654, 0.0337469, 
Output 3: 0.965665, 0.0250305, 0.00304446, 0.00625956, 
Output 4: 0.265853, 0.662295, 0.0674992, 0.00435256, 
Output 5: 0.492838, 0.421737, 0.0726745, 0.0127504, 
Output 6: 0.0250511, 0.0635571, 0.0781076, 0.833284, 

Begin processing the 10th record. run: 5026 subRun: 0 event: 10090 at 06-Dec-2025 09:08:22 UTC
Boundary wire vector sizes: 6813, 6751, 6661
minwire 0: 51
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.158373, 
Output 1: 0.0798043, 0.0632208, 0.157361, 0.699614, 
Output 2: 0.0537353, 0.215455, 0.699521, 0.0312884, 
Output 3: 0.962458, 0.0200997, 0.00460925, 0.0128331, 
Output 4: 0.314984, 0.582846, 0.0925468, 0.00962231, 
Output 5: 0.471777, 0.40787, 0.0999117, 0.020441, 
Output 6: 0.0227211, 0.0531115, 0.0709918, 0.853176, 

06-Dec-2025 09:58:22 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 09:58:22 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/a4/95/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T171704Z_gen_001009_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2633.73       2974.98       3511.19       2879.11       292.041        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00323562     0.0120061     0.0446183    0.00683418     0.0124627       10     
reco:pandora:StandardPandora                            2629.36       2969.76       3505.95       2875.14       291.142        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00270681     0.0442258     0.215901      0.0278785     0.0616905       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000716762    0.0100208     0.0576138     0.0025029     0.0170583       10     
reco:pandoracalo:Calorimetry                          0.000523186   0.00826857     0.0444894    0.00101956     0.0129638       10     
reco:pandorapid:Chi2ParticleID                        0.00013797    0.000845429   0.00614307    0.000192583   0.00177167       10     
reco:cvnmap:CVNMapper                                  0.0477322     0.072101      0.153071      0.0612881     0.0309398       10     
reco:cvneva:CVNEvaluator                                3.12948       4.1421        8.51461       3.55187       1.52778        10     
reco:energyrecnumu:EnergyReco                          0.0068274     0.102721      0.204327      0.123892      0.0641902       10     
reco:energyrecnue:EnergyReco                          0.00249604    0.00780796     0.0167079     0.0043922    0.00589966       10     
reco:energyrecnc:EnergyReco                            0.0105976     0.0580036     0.216316      0.0166781     0.0689286       10     
[art]:TriggerResults:TriggerResultInserter            3.3634e-05    6.20879e-05   0.000193545   4.2455e-05    4.75586e-05      10     
end_path:out1:RootOutput                              1.2093e-05    1.83897e-05   6.1536e-05    1.39315e-05   1.44054e-05      10     
end_path:out1:RootOutput(write)                        0.451852      0.758959       1.31041      0.697944      0.223971        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21782.8 MB
  Peak resident set size usage (VmHWM): 3069.99 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 219.913 | hit correction : 1.37438
hit initial : 194.639 | hit correction : 1.37419
hit initial : 118.902 | hit correction : 1.37378
 *** Total Charge : 3911.19
 Looking at view: 2 | Number of hits in view: 17
Begin processing the 9th record. run: 5026 subRun: 0 event: 10089 at 06-Dec-2025 10:00:00 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 21.5568
Completed MCParticle Loop
True Neutrino Energy: 26.9477
 Size of all hits: 0
Begin processing the 10th record. run: 5026 subRun: 0 event: 10090 at 06-Dec-2025 10:00:06 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  53
Added hit list length: 53 | Max was set to: -3.40282e+38
True Neutron Energy: 939.571
True Neutron Energy: 939.721
True Electron Energy: 31.4404
Completed MCParticle Loop
True Neutrino Energy: 40.3911
 Size of all hits: 53
hit initial : 274.882 | hit correction : 1.10781
hit initial : 271.8 | hit correction : 1.1077
hit initial : 239.83 | hit correction : 1.10751
hit initial : 38.8487 | hit correction : 1.10747
hit initial : 77.9777 | hit correction : 1.10783
hit initial : 215.752 | hit correction : 1.10676
hit initial : 217.141 | hit correction : 1.10654
hit initial : 27.6899 | hit correction : 1.10047
hit initial : 16.8146 | hit correction : 1.10046
hit initial : 240.798 | hit correction : 1.09774
hit initial : 58.9536 | hit correction : 1.09775
hit initial : 239.689 | hit correction : 1.0977
hit initial : 211.735 | hit correction : 1.0973
hit initial : 96.3465 | hit correction : 1.09727
hit initial : 148.155 | hit correction : 1.09675
hit initial : 40.0526 | hit correction : 1.09665
hit initial : 141.262 | hit correction : 1.09606
hit initial : 207.058 | hit correction : 1.09599
hit initial : 115.011 | hit correction : 1.09596
 *** Total Charge : 3173.09
 Looking at view: 0 | Number of hits in view: 19
hit initial : 38.6821 | hit correction : 1.09577
hit initial : 128.165 | hit correction : 1.09615
hit initial : 62.0877 | hit correction : 1.09773
hit initial : 99.034 | hit correction : 1.10783
hit initial : 333.365 | hit correction : 1.10778
hit initial : 346.762 | hit correction : 1.10764
hit initial : 92.4189 | hit correction : 1.10751
hit initial : 288.748 | hit correction : 1.10677
hit initial : 127.773 | hit correction : 1.10684
hit initial : 72.131 | hit correction : 1.10615
hit initial : 26.5451 | hit correction : 1.10622
hit initial : 221.245 | hit correction : 1.09773
hit initial : 300.568 | hit correction : 1.09768
hit initial : 280.883 | hit correction : 1.09738
hit initial : 257.11 | hit correction : 1.09695
hit initial : 183.016 | hit correction : 1.0967
hit initial : 280.173 | hit correction : 1.09606
hit initial : 174.482 | hit correction : 1.09589
 *** Total Charge : 3648.71
 Looking at view: 1 | Number of hits in view: 18
hit initial : 43.3263 | hit correction : 1.09683
hit initial : 41.8107 | hit correction : 1.09655
hit initial : 98.9245 | hit correction : 1.09484
hit initial : 554.623 | hit correction : 1.10777
hit initial : 510.894 | hit correction : 1.10748
hit initial : 297.404 | hit correction : 1.10649
hit initial : 63.0181 | hit correction : 1.10608
hit initial : 339.485 | hit correction : 1.09775
hit initial : 482.73 | hit correction : 1.09755
hit initial : 197.599 | hit correction : 1.09711
hit initial : 184.969 | hit correction : 1.09682
hit initial : 203.817 | hit correction : 1.09651
hit initial : 140.196 | hit correction : 1.09619
hit initial : 165.102 | hit correction : 1.09604
hit initial : 164.008 | hit correction : 1.09592
hit initial : 117.262 | hit correction : 1.0957
 *** Total Charge : 3969.16
 Looking at view: 2 | Number of hits in view: 16
06-Dec-2025 10:00:13 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          0.15849      0.193587       0.25265      0.181639      0.0295762       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000814456   0.00170516    0.00722308    0.000931702   0.00188986       10     
end_path:ana:AnalyzeEvents          0.15707       0.19146      0.249003      0.180097      0.0294858       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19472.8 MB
  Peak resident set size usage (VmHWM): 1534.04 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 13:14:13 UTC       justIN version: 01.05.03