justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262031.1@dunegpschedd02.fnal.gov

Jobsub ID262031.1@dunegpschedd02.fnal.gov
Workflow ID11171
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 07:24:21
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 17:59:29
From worker nodeHostnamedunegli-7890498-0-fnpc18005.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 07:25:02
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T014414Z_gen_004235_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time10h (38041s)
CPU time10h (37330s = 98%)
Max RSS bytes2802085888 (2672 MiB)
Outputting started2025-12-13 17:59:04
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11171/1/001/analysisLowE_20251213T072506Z.root
Finished2025-12-13 17:59:29
Saved logsjustin-logs:262031.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

rc/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 17:57:42 UTC run: 5026 subRun: 0 event: 42350
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6836, 6768, 6707
minwire 0: 14
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1931, 2430
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0966354, 
Output 1: 0.0411307, 0.0486945, 0.0905105, 0.819664, 
Output 2: 0.148152, 0.224323, 0.58553, 0.0419946, 
Output 3: 0.97927, 0.0158354, 0.001049, 0.00384549, 
Output 4: 0.476024, 0.487213, 0.0334008, 0.00336275, 
Output 5: 0.736164, 0.227924, 0.0269224, 0.00898909, 
Output 6: 0.0154279, 0.0444329, 0.057784, 0.882355, 

13-Dec-2025 17:57:46 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 17:57:46 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/e2/af/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T014414Z_gen_004235_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2537.62       3761.38       5638.2        3172.75       1140.2         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00115617    0.00635643     0.028336     0.00173101    0.00910899       10     
reco:pandora:StandardPandora                            2534.9        3756.49       5632.05       3169.93       1138.04        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00113066     0.0895658     0.251679      0.0813153     0.0665799       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000833218    0.0147488     0.116616     0.00341203     0.0340031       10     
reco:pandoracalo:Calorimetry                          0.000424899    0.0035809     0.0253188    0.000822068   0.00733756       10     
reco:pandorapid:Chi2ParticleID                        0.000119565   0.00226377     0.0199565    0.000198635   0.00590535       10     
reco:cvnmap:CVNMapper                                  0.104318      0.160192      0.361782       0.11304      0.0865994       10     
reco:cvneva:CVNEvaluator                                1.9392        3.73901       13.459        2.08562       3.36668        10     
reco:energyrecnumu:EnergyReco                          0.0958528     0.137521      0.211575        0.119       0.0432962       10     
reco:energyrecnue:EnergyReco                          0.00201223    0.00628406     0.0181956    0.00324131    0.00547157       10     
reco:energyrecnc:EnergyReco                           0.00999746     0.0157437     0.0271646     0.0138088    0.00582763       10     
[art]:TriggerResults:TriggerResultInserter            2.7872e-05    9.41403e-05   0.00028709    4.93185e-05   8.88363e-05      10     
end_path:out1:RootOutput                              1.3005e-05    3.06949e-05   0.000112201   1.77035e-05   2.89185e-05      10     
end_path:out1:RootOutput(write)                        0.413034      0.717718       1.20419      0.705566      0.288892        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21581.8 MB
  Peak resident set size usage (VmHWM): 2802.09 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 239.773 | hit correction : 1.03488
hit initial : 213.715 | hit correction : 1.03477
hit initial : 221.893 | hit correction : 1.03484
hit initial : 277.233 | hit correction : 1.03452
hit initial : 248.274 | hit correction : 1.03367
hit initial : 206.488 | hit correction : 1.03406
hit initial : 236.364 | hit correction : 1.03327
hit initial : 194.654 | hit correction : 1.03286
 *** Total Charge : 2106.61
 Looking at view: 2 | Number of hits in view: 10
Begin processing the 10th record. run: 5026 subRun: 0 event: 42350 at 13-Dec-2025 17:58:59 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  5
This pfp has size:  52
Added hit list length: 52 | Max was set to: -3.40282e+38
This pfp has size:  17
This pfp has size:  5
True Neutron Energy: 939.571
True Neutron Energy: 942.1
True Neutron Energy: 940.577
True Electron Energy: 36.7018
Completed MCParticle Loop
True Neutrino Energy: 53.9503
 Size of all hits: 52
hit initial : 44.5627 | hit correction : 1.08718
hit initial : 171.017 | hit correction : 1.07691
hit initial : 55.4448 | hit correction : 1.0776
hit initial : 617.238 | hit correction : 1.07681
hit initial : 342.461 | hit correction : 1.07626
hit initial : 174.993 | hit correction : 1.07761
hit initial : 86.396 | hit correction : 1.0769
hit initial : 61.8891 | hit correction : 1.0776
hit initial : 407.493 | hit correction : 1.07584
hit initial : 282.108 | hit correction : 1.07516
hit initial : 246.082 | hit correction : 1.07476
hit initial : 232.733 | hit correction : 1.07443
hit initial : 308.518 | hit correction : 1.07397
hit initial : 403.886 | hit correction : 1.07347
hit initial : 221.813 | hit correction : 1.07342
hit initial : 158.025 | hit correction : 1.07289
hit initial : 118.008 | hit correction : 1.07299
 *** Total Charge : 4229.3
 Looking at view: 0 | Number of hits in view: 17
hit initial : 47.5631 | hit correction : 1.08718
hit initial : 20.3193 | hit correction : 1.08715
hit initial : 185.766 | hit correction : 1.07696
hit initial : 47.9858 | hit correction : 1.07698
hit initial : 294.28 | hit correction : 1.07668
hit initial : 140.455 | hit correction : 1.07629
hit initial : 146.033 | hit correction : 1.07761
hit initial : 261.426 | hit correction : 1.07688
hit initial : 117.6 | hit correction : 1.07758
hit initial : 447.849 | hit correction : 1.07599
hit initial : 27.9369 | hit correction : 1.07536
hit initial : 222.319 | hit correction : 1.07487
hit initial : 338.421 | hit correction : 1.07446
hit initial : 600.933 | hit correction : 1.07353
hit initial : 129.953 | hit correction : 1.07349
hit initial : 276.373 | hit correction : 1.07334
hit initial : 70.581 | hit correction : 1.07282
 *** Total Charge : 3630.67
 Looking at view: 1 | Number of hits in view: 17
hit initial : 35.8474 | hit correction : 1.08076
hit initial : 74.5825 | hit correction : 1.08719
hit initial : 25.8533 | hit correction : 1.08708
hit initial : 432.403 | hit correction : 1.0771
hit initial : 275.492 | hit correction : 1.07696
hit initial : 284.441 | hit correction : 1.07763
hit initial : 155.804 | hit correction : 1.07695
hit initial : 371.551 | hit correction : 1.07657
hit initial : 270.398 | hit correction : 1.07638
hit initial : 106.844 | hit correction : 1.07757
hit initial : 553.858 | hit correction : 1.07599
hit initial : 206.241 | hit correction : 1.07551
hit initial : 267.943 | hit correction : 1.0751
hit initial : 364.656 | hit correction : 1.07465
hit initial : 428.651 | hit correction : 1.07406
hit initial : 122.817 | hit correction : 1.07346
hit initial : 514.689 | hit correction : 1.07338
hit initial : 208.083 | hit correction : 1.07284
 *** Total Charge : 5056.5
 Looking at view: 2 | Number of hits in view: 18
13-Dec-2025 17:59:03 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0949277     0.165799       0.33718      0.128605      0.0838855       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00063686    0.00126063    0.00318809    0.000835544   0.00091275       10     
end_path:ana:AnalyzeEvents         0.0937061      0.16417      0.336106      0.127477      0.0840689       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477.6 MB
  Peak resident set size usage (VmHWM): 1538.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:04:55 UTC       justIN version: 01.06.00