justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265279.37@dunegpschedd01.fnal.gov

Jobsub ID265279.37@dunegpschedd01.fnal.gov
Workflow ID11116
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-09 10:51:45
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-10 09:07:34
From worker nodeHostnamedunegli-48793-0-cmswn4069.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-09 16:37:07
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T030042Z_gen_004578_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time16h (59022s)
CPU time16h (58145s = 98%)
Max RSS bytes2563907584 (2445 MiB)
Outputting started2025-12-10 09:00:50
Output files
Finished2025-12-10 09:07:34
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  10-Dec-2025 09:00:15 UTC run: 5027 subRun: 0 event: 45780
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6811, 6846, 6697
minwire 0: 49
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 248, 747
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0500946, 
Output 1: 0.0217919, 0.0276793, 0.0637669, 0.886762, 
Output 2: 0.0795316, 0.205914, 0.68406, 0.030495, 
Output 3: 0.933995, 0.0505044, 0.00763577, 0.00786519, 
Output 4: 0.692022, 0.232985, 0.0591834, 0.0158094, 
Output 5: 0.781551, 0.190307, 0.0231703, 0.00497183, 
Output 6: 0.0152592, 0.0640152, 0.0712925, 0.849433, 

10-Dec-2025 09:00:17 UTC  Closed output file "matt_reco2.root"
10-Dec-2025 09:00:17 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/53/09/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T030042Z_gen_004578_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2431.06       5849.33       8284.2        6317.33       1637.38        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0135558     0.0289087     0.0788124     0.0236479     0.0184717       10     
reco:pandora:StandardPandora                            2429.84       5844.27       8260.24       6314.27       1633.67        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00132032     0.121066      0.659884      0.0556183     0.188848        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000532482    0.0178293     0.0843151    0.00410381     0.025942        10     
reco:pandoracalo:Calorimetry                          0.000313345   0.00998247     0.0336882    0.00223076     0.0124014       10     
reco:pandorapid:Chi2ParticleID                        9.9389e-05    0.00164909     0.0081813    0.000378605   0.00249347       10     
reco:cvnmap:CVNMapper                                  0.049469      0.183746      0.481844      0.154948      0.117358        10     
reco:cvneva:CVNEvaluator                               0.743124       3.55422       19.6473       1.98313       5.38491        10     
reco:energyrecnumu:EnergyReco                         0.00649007     0.143562      0.458936      0.129476       0.13382        10     
reco:energyrecnue:EnergyReco                          0.000902805   0.00909788     0.0243023    0.00566686    0.00779605       10     
reco:energyrecnc:EnergyReco                           0.00801371     0.0746981     0.268977      0.0249971     0.0834341       10     
[art]:TriggerResults:TriggerResultInserter            3.0017e-05    0.000162398   0.000593838   0.000110781   0.000158065      10     
end_path:out1:RootOutput                              1.2523e-05    6.47047e-05   0.000216481   5.3121e-05    5.52518e-05      10     
end_path:out1:RootOutput(write)                        0.249372      0.913636       2.51282      0.795518      0.605616        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21681.1 MB
  Peak resident set size usage (VmHWM): 2563.91 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 295.659 | hit correction : 1.18344
hit initial : 242.259 | hit correction : 1.18355
hit initial : 247.877 | hit correction : 1.18371
hit initial : 256.238 | hit correction : 1.18387
hit initial : 250.98 | hit correction : 1.18406
hit initial : 240.573 | hit correction : 1.18418
hit initial : 196.505 | hit correction : 1.18429
hit initial : 183.514 | hit correction : 1.18438
hit initial : 208.771 | hit correction : 1.18446
hit initial : 182.934 | hit correction : 1.18453
hit initial : 96.2854 | hit correction : 1.18464
hit initial : 25.6283 | hit correction : 1.18474
 *** Total Charge : 4158.98
 Looking at view: 1 | Number of hits in view: 18
hit initial : 71.9116 | hit correction : 1.18478
hit initial : 48.1094 | hit correction : 1.18312
hit initial : 279.497 | hit correction : 1.1833
hit initial : 971.798 | hit correction : 1.1836
hit initial : 55.4024 | hit correction : 1.18363
hit initial : 49.9054 | hit correction : 1.18391
hit initial : 179.519 | hit correction : 1.18373
hit initial : 195.112 | hit correction : 1.18357
hit initial : 108.173 | hit correction : 1.18357
hit initial : 104.598 | hit correction : 1.18612
hit initial : 94.6041 | hit correction : 1.18609
hit initial : 175.093 | hit correction : 1.18386
hit initial : 362.277 | hit correction : 1.18416
hit initial : 783.096 | hit correction : 1.18432
hit initial : 185.582 | hit correction : 1.18449
 *** Total Charge : 4339.01
 Looking at view: 2 | Number of hits in view: 15
Begin processing the 10th record. run: 5027 subRun: 0 event: 45780 at 10-Dec-2025 09:00:48 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  3
This pfp has size:  37
Added hit list length: 37 | Max was set to: -3.40282e+38
True Neutron Energy: 939.577
True Electron Energy: 48.2535
Completed MCParticle Loop
True Neutrino Energy: 54.9835
 Size of all hits: 37
hit initial : 69.8458 | hit correction : 1.43844
hit initial : 64.1619 | hit correction : 1.43846
hit initial : 156.02 | hit correction : 1.44246
hit initial : 134.743 | hit correction : 1.44143
hit initial : 175.588 | hit correction : 1.44244
hit initial : 65.7715 | hit correction : 1.44144
hit initial : 79.2572 | hit correction : 1.4424
hit initial : 63.6098 | hit correction : 1.44163
hit initial : 27.1294 | hit correction : 1.44852
hit initial : 44.3417 | hit correction : 1.44853
hit initial : 37.5501 | hit correction : 1.44846
hit initial : 31.6163 | hit correction : 1.44523
hit initial : 104.617 | hit correction : 1.4446
hit initial : 387.113 | hit correction : 1.44467
hit initial : 113.932 | hit correction : 1.4448
hit initial : 276.784 | hit correction : 1.44323
hit initial : 20.2083 | hit correction : 1.43659
hit initial : 7.67293 | hit correction : 1.44163
hit initial : 256.039 | hit correction : 1.4421
hit initial : 31.5719 | hit correction : 1.44111
hit initial : 421.982 | hit correction : 1.44068
hit initial : 469.495 | hit correction : 1.44001
hit initial : 355.956 | hit correction : 1.43989
 *** Total Charge : 4895.59
 Looking at view: 0 | Number of hits in view: 23
hit initial : 28.251 | hit correction : 1.4415
hit initial : 18.8677 | hit correction : 1.4415
hit initial : 94.0457 | hit correction : 1.44499
hit initial : 80.6961 | hit correction : 1.44498
hit initial : 160.696 | hit correction : 1.44462
hit initial : 77.3285 | hit correction : 1.44289
hit initial : 187.853 | hit correction : 1.44457
hit initial : 99.6266 | hit correction : 1.44326
hit initial : 48.976 | hit correction : 1.44444
hit initial : 135.262 | hit correction : 1.44203
hit initial : 311.853 | hit correction : 1.44155
hit initial : 177.031 | hit correction : 1.44078
hit initial : 253.222 | hit correction : 1.44036
hit initial : 281.931 | hit correction : 1.44001
 *** Total Charge : 2820.42
 Looking at view: 1 | Number of hits in view: 14
10-Dec-2025 09:00:50 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.049906      0.0679076     0.0943105     0.0658017     0.0129385       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00050007    0.000645117   0.000993798   0.000607354   0.000151722      10     
end_path:ana:AnalyzeEvents         0.048724       0.06698      0.0921959     0.0650188     0.0127094       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19473.1 MB
  Peak resident set size usage (VmHWM): 1526.98 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 23:17:46 UTC       justIN version: 01.05.03