justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 261927.1@dunegpschedd02.fnal.gov

Jobsub ID261927.1@dunegpschedd02.fnal.gov
Workflow ID11172
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 05:02:14
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-13 17:10:30
From worker nodeHostnamedunegli-8047001-0-fnpc19108.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 05:03:21
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232331Z_gen_001679_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time12h (43600s)
CPU time11h (42857s = 98%)
Max RSS bytes2832449536 (2701 MiB)
Outputting started2025-12-13 17:10:02
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11172/1/001/analysisLowE_20251213T050323Z.root
Finished2025-12-13 17:10:30
Saved logsjustin-logs:261927.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Impl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 17:09:00 UTC run: 5027 subRun: 0 event: 16790
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6660, 6656, 6643
minwire 0: 9
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.104379, 
Output 1: 0.0275254, 0.0425536, 0.119003, 0.810918, 
Output 2: 0.0598925, 0.168174, 0.749097, 0.0228368, 
Output 3: 0.973317, 0.0134195, 0.00238327, 0.0108802, 
Output 4: 0.210845, 0.694545, 0.0870297, 0.00758034, 
Output 5: 0.629993, 0.316142, 0.0441358, 0.00972848, 
Output 6: 0.0119019, 0.029094, 0.0435302, 0.915474, 

13-Dec-2025 17:09:03 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 17:09:03 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/56/94/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232331Z_gen_001679_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2603.51       4330.44       4969.18       4464.78       628.856        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00115529    0.00614724     0.0259504    0.00232974    0.00807292       10     
reco:pandora:StandardPandora                            2601.84       4327.09       4966.19       4461.48       628.357        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000945408    0.109769      0.220843      0.148221      0.0809723       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000774877    0.017027      0.101886     0.00688259     0.0290946       10     
reco:pandoracalo:Calorimetry                          0.000525644   0.00334438     0.0208477    0.00112808     0.0059129       10     
reco:pandorapid:Chi2ParticleID                        0.000129351   0.00136151     0.010177     0.000298162   0.00295164       10     
reco:cvnmap:CVNMapper                                  0.0612528     0.146647       0.2307        0.14598      0.0428561       10     
reco:cvneva:CVNEvaluator                               0.962563       2.35534       7.40453       1.93146       1.71696        10     
reco:energyrecnumu:EnergyReco                          0.117279      0.164557       0.2599       0.150077      0.0505811       10     
reco:energyrecnue:EnergyReco                          0.00149145     0.0075295      0.02299     0.00448782    0.00669362       10     
reco:energyrecnc:EnergyReco                            0.0088791     0.0143244     0.0179284     0.0140737    0.00258114       10     
[art]:TriggerResults:TriggerResultInserter            1.9731e-05    8.81329e-05   0.000439284   3.92105e-05   0.000121188      10     
end_path:out1:RootOutput                               9.12e-06     4.06423e-05   0.000140801   2.4985e-05    3.8196e-05       10     
end_path:out1:RootOutput(write)                        0.311088      0.523937       1.18498      0.455581      0.227534        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21573.9 MB
  Peak resident set size usage (VmHWM): 2832.45 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 22.5957 | hit correction : 1.16159
hit initial : 41.6937 | hit correction : 1.18024
hit initial : 12.8044 | hit correction : 1.1818
hit initial : 48.7931 | hit correction : 1.1825
hit initial : 298.996 | hit correction : 1.18293
hit initial : 316.268 | hit correction : 1.1831
hit initial : 153.342 | hit correction : 1.18404
hit initial : 454.184 | hit correction : 1.18413
hit initial : 257.251 | hit correction : 1.18427
hit initial : 83.2266 | hit correction : 1.19216
hit initial : 282.714 | hit correction : 1.1922
hit initial : 132.928 | hit correction : 1.19207
hit initial : 245.672 | hit correction : 1.19261
hit initial : 304.293 | hit correction : 1.19283
hit initial : 245.035 | hit correction : 1.19326
hit initial : 266.786 | hit correction : 1.19356
hit initial : 245.612 | hit correction : 1.19384
hit initial : 255.61 | hit correction : 1.19418
hit initial : 217.645 | hit correction : 1.19438
hit initial : 90.0549 | hit correction : 1.19446
 *** Total Charge : 5027.01
 Looking at view: 0 | Number of hits in view: 25
hit initial : 31.4285 | hit correction : 1.12658
hit initial : 74.9391 | hit correction : 1.12659
hit initial : 37.6999 | hit correction : 1.12659
hit initial : 39.9944 | hit correction : 1.13579
hit initial : 66.7723 | hit correction : 1.13579
hit initial : 24.0743 | hit correction : 1.18024
hit initial : 29.8852 | hit correction : 1.18023
hit initial : 20.0475 | hit correction : 1.18189
hit initial : 40.4002 | hit correction : 1.18245
hit initial : 126.26 | hit correction : 1.18271
hit initial : 376.345 | hit correction : 1.18297
hit initial : 152.959 | hit correction : 1.18342
hit initial : 392.923 | hit correction : 1.18406
hit initial : 397.056 | hit correction : 1.18418
hit initial : 86.0981 | hit correction : 1.18425
hit initial : 113.403 | hit correction : 1.19212
hit initial : 145.509 | hit correction : 1.19207
hit initial : 89.7255 | hit correction : 1.19207
hit initial : 156.217 | hit correction : 1.1926
hit initial : 36.3626 | hit correction : 1.19253
hit initial : 48.1237 | hit correction : 1.19263
hit initial : 191.888 | hit correction : 1.19269
hit initial : 510.794 | hit correction : 1.19359
hit initial : 97.0128 | hit correction : 1.19373
hit initial : 75.8788 | hit correction : 1.19439
hit initial : 222.425 | hit correction : 1.19437
hit initial : 69.3636 | hit correction : 1.19436
 *** Total Charge : 4327.81
 Looking at view: 1 | Number of hits in view: 27
hit initial : 111.609 | hit correction : 1.12659
hit initial : 62.613 | hit correction : 1.12656
hit initial : 88.8077 | hit correction : 1.13579
hit initial : 62.6633 | hit correction : 1.13578
hit initial : 38.9028 | hit correction : 1.1616
hit initial : 38.7541 | hit correction : 1.18025
hit initial : 24.9197 | hit correction : 1.18022
hit initial : 36.4863 | hit correction : 1.18189
hit initial : 114.896 | hit correction : 1.18241
hit initial : 181.357 | hit correction : 1.18281
hit initial : 431.063 | hit correction : 1.18333
hit initial : 204.745 | hit correction : 1.18379
hit initial : 348.348 | hit correction : 1.18426
hit initial : 296.474 | hit correction : 1.18429
hit initial : 206.864 | hit correction : 1.19208
hit initial : 222.976 | hit correction : 1.19209
hit initial : 36.8459 | hit correction : 1.19241
hit initial : 218.7 | hit correction : 1.19263
hit initial : 70.576 | hit correction : 1.19257
hit initial : 45.9335 | hit correction : 1.19265
hit initial : 213.86 | hit correction : 1.19292
hit initial : 191.148 | hit correction : 1.19322
hit initial : 197.256 | hit correction : 1.19348
hit initial : 203.489 | hit correction : 1.19367
hit initial : 161.059 | hit correction : 1.1939
hit initial : 130.948 | hit correction : 1.1941
hit initial : 153.396 | hit correction : 1.19425
hit initial : 233.005 | hit correction : 1.1944
hit initial : 127.096 | hit correction : 1.19453
 *** Total Charge : 5277.85
 Looking at view: 2 | Number of hits in view: 29
13-Dec-2025 17:10:01 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0764739     0.0955249     0.132821      0.0926499     0.0175332       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000449244   0.00118692    0.00664329    0.000523429    0.0018246       10     
end_path:ana:AnalyzeEvents         0.0754957     0.0941014     0.131997      0.0919802     0.0167239       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477.1 MB
  Peak resident set size usage (VmHWM): 1534.3 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:04:53 UTC       justIN version: 01.06.00