justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262121.0@dunegpschedd02.fnal.gov

Jobsub ID262121.0@dunegpschedd02.fnal.gov
Workflow ID11171
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 09:58:29
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-13 16:46:22
From worker nodeHostnamedunegli-8047097-0-fnpc9060.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 09:59:31
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021528Z_gen_003797_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time6h (24400s)
CPU time6h (24009s = 98%)
Max RSS bytes2797838336 (2668 MiB)
Outputting started2025-12-13 16:46:11
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11171/1/001/analysisLowE_20251213T095932Z.root
Finished2025-12-13 16:46:22
Saved logsjustin-logs:262121.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

aloHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/osbiston/18_10_24_Official_LowE_Build/srcs/larpandoracontent/larpandoracontent/LArMonitoring/EventValidationBaseAlgorithm.cc line#: 64
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 6947, 6881, 6821
minwire 0: 40
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0930051, 
Output 1: 0.0541619, 0.0270059, 0.141042, 0.77779, 
Output 2: 0.118822, 0.223429, 0.627505, 0.0302445, 
Output 3: 0.950862, 0.0400089, 0.00398949, 0.00513955, 
Output 4: 0.193765, 0.696619, 0.101292, 0.00832428, 
Output 5: 0.77382, 0.21242, 0.0117736, 0.00198627, 
Output 6: 0.0247436, 0.0698929, 0.0744499, 0.830914, 

13-Dec-2025 16:45:06 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 16:45:06 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/32/a2/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T021528Z_gen_003797_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1521.74       2410.12       3207.91       2550.52       653.344        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000906248   0.00478178     0.0167411    0.00133829    0.00565947       10     
reco:pandora:StandardPandora                            1519.96       2406.85       3204.32       2547.38       652.331        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000590455    0.0763381     0.163396      0.0770774     0.0472784       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000822503   0.00955739     0.0514436    0.00544903     0.0141577       10     
reco:pandoracalo:Calorimetry                          0.000231704   0.00216705     0.0107301    0.000851707   0.00303709       10     
reco:pandorapid:Chi2ParticleID                        0.000123803   0.000643657   0.00435212     0.0002055    0.00124119       10     
reco:cvnmap:CVNMapper                                  0.0757538     0.139715       0.28865      0.117465      0.0627121       10     
reco:cvneva:CVNEvaluator                                1.14877       2.25691       6.82636       1.93015       1.58724        10     
reco:energyrecnumu:EnergyReco                          0.0788968     0.156103      0.409238      0.130859      0.0945184       10     
reco:energyrecnue:EnergyReco                          0.00147518    0.00445903     0.0163854    0.00272423    0.00443032       10     
reco:energyrecnc:EnergyReco                           0.00828705     0.0113336     0.0160053     0.0113869    0.00265055       10     
[art]:TriggerResults:TriggerResultInserter            1.8795e-05    4.48273e-05   0.00012422    3.77545e-05   3.02398e-05      10     
end_path:out1:RootOutput                               7.415e-06    1.26831e-05   3.3047e-05     9.365e-06    7.56829e-06      10     
end_path:out1:RootOutput(write)                        0.361945      0.603628      0.949732       0.62712      0.197269        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11912.8 MB
  Peak resident set size usage (VmHWM): 2797.84 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 450.543 | hit correction : 1.08529
hit initial : 29.5298 | hit correction : 1.08493
hit initial : 836.888 | hit correction : 1.08581
hit initial : 183.107 | hit correction : 1.08582
hit initial : 172.141 | hit correction : 1.09171
hit initial : 779.62 | hit correction : 1.0916
hit initial : 230.755 | hit correction : 1.09152
hit initial : 329.073 | hit correction : 1.09126
hit initial : 219.233 | hit correction : 1.0911
hit initial : 238.823 | hit correction : 1.09087
hit initial : 272.871 | hit correction : 1.09061
hit initial : 396.292 | hit correction : 1.09033
hit initial : 277.091 | hit correction : 1.09005
hit initial : 89.6746 | hit correction : 1.08993
 *** Total Charge : 4907.67
 Looking at view: 2 | Number of hits in view: 14
Begin processing the 8th record. run: 5027 subRun: 0 event: 37968 at 13-Dec-2025 16:45:57 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  16
Added hit list length: 16 | Max was set to: -3.40282e+38
True Neutron Energy: 945.996
True Neutron Energy: 941.562
True Neutron Energy: 940.446
True Electron Energy: 11.1656
Completed MCParticle Loop
True Neutrino Energy: 26.6465
 Size of all hits: 16
hit initial : 12.8785 | hit correction : 1.01338
hit initial : 339.155 | hit correction : 1.01388
hit initial : 292.654 | hit correction : 1.01388
hit initial : 304.498 | hit correction : 1.01387
hit initial : 272.516 | hit correction : 1.01396
 *** Total Charge : 1238.67
 Looking at view: 0 | Number of hits in view: 5
hit initial : 16.628 | hit correction : 1.01343
hit initial : 606.738 | hit correction : 1.01389
hit initial : 363.464 | hit correction : 1.0139
 *** Total Charge : 1000.53
 Looking at view: 1 | Number of hits in view: 3
hit initial : 32.9104 | hit correction : 1.01005
hit initial : 26.3495 | hit correction : 1.01336
hit initial : 299.106 | hit correction : 1.01391
hit initial : 430.916 | hit correction : 1.01386
hit initial : 169.289 | hit correction : 1.01379
hit initial : 146.339 | hit correction : 1.01384
hit initial : 148.354 | hit correction : 1.01396
hit initial : 163.593 | hit correction : 1.01411
 *** Total Charge : 1436.41
 Looking at view: 2 | Number of hits in view: 8
Begin processing the 9th record. run: 5027 subRun: 0 event: 37969 at 13-Dec-2025 16:46:02 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  6
This pfp has size:  7
Added hit list length: 7 | Max was set to: -3.40282e+38
This pfp has size:  5
This pfp has size:  5
True Electron Energy: 4.41642
Completed MCParticle Loop
True Neutrino Energy: 9.7943
 Size of all hits: 7
hit initial : 19.5239 | hit correction : 1.09412
hit initial : 92.5126 | hit correction : 1.09413
hit initial : 43.8769 | hit correction : 1.09414
 *** Total Charge : 170.59
 Looking at view: 0 | Number of hits in view: 3
hit initial : 65.2335 | hit correction : 1.09412
hit initial : 70.1557 | hit correction : 1.09413
 *** Total Charge : 148.133
 Looking at view: 1 | Number of hits in view: 2
hit initial : 151.423 | hit correction : 1.09412
hit initial : 13.8935 | hit correction : 1.09414
 *** Total Charge : 180.876
 Looking at view: 2 | Number of hits in view: 2
Begin processing the 10th record. run: 5027 subRun: 0 event: 37970 at 13-Dec-2025 16:46:06 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
True Neutron Energy: 939.578
True Electron Energy: 23.2173
Completed MCParticle Loop
True Neutrino Energy: 38.7301
 Size of all hits: 0
13-Dec-2025 16:46:10 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0762618     0.163885      0.284635      0.156696      0.0540841       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000418494   0.00631081     0.029188     0.000724762   0.00956633       10     
end_path:ana:AnalyzeEvents         0.0757127      0.15738      0.284011      0.151949      0.0557896       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9812.42 MB
  Peak resident set size usage (VmHWM): 1538.27 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:05:51 UTC       justIN version: 01.06.00