justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267033.0@dunegpschedd01.fnal.gov

Jobsub ID267033.0@dunegpschedd01.fnal.gov
Workflow ID11108
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 21:46:37
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-11 23:32:31
From worker nodeHostnamedunegli-8035317-0-fnpc22006.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-11 21:48:16
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T231109Z_gen_001345_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time1h (6245s)
CPU time1h (6171s = 98%)
Max RSS bytes2918457344 (2783 MiB)
Outputting started2025-12-11 23:32:21
Output files
Finished2025-12-11 23:32:31
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.106934, 
Output 1: 0.0641295, 0.0585628, 0.112802, 0.764505, 
Output 2: 0.0663749, 0.209363, 0.692922, 0.0313401, 
Output 3: 0.971109, 0.0203863, 0.00243072, 0.0060744, 
Output 4: 0.42662, 0.511358, 0.0563083, 0.005713, 
Output 5: 0.597149, 0.306289, 0.0748932, 0.0216685, 
Output 6: 0.024991, 0.0697067, 0.0797303, 0.825572, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 13450 at 11-Dec-2025 23:21:30 UTC
Boundary wire vector sizes: 6843, 6836, 6768
minwire 0: 41
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.158971, 
Output 1: 0.065455, 0.0276868, 0.175378, 0.731481, 
Output 2: 0.121362, 0.178218, 0.660834, 0.0395866, 
Output 3: 0.979996, 0.0103261, 0.00141312, 0.00826527, 
Output 4: 0.139521, 0.823706, 0.0350881, 0.001685, 
Output 5: 0.500555, 0.436826, 0.0509221, 0.0116968, 
Output 6: 0.0124777, 0.0346568, 0.0493664, 0.903499, 

11-Dec-2025 23:31:54 UTC  Closed output file "matt_reco2.root"
11-Dec-2025 23:31:54 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/bf/53/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T231109Z_gen_001345_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              606.404       614.003       621.635       613.261       3.91653        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00109184    0.00549098     0.0194251    0.00358151    0.00513105       10     
reco:pandora:StandardPandora                            605.424       612.891       620.636       612.121       3.97248        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00140422     0.0111039     0.0455231    0.00353216     0.0131601       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00154077    0.00404859    0.00941188     0.0027679     0.0025837       10     
reco:pandoracalo:Calorimetry                          0.000160082   0.00521857     0.0146177    0.00253138    0.00600026       10     
reco:pandorapid:Chi2ParticleID                        4.5576e-05    0.000187423   0.000724619   0.000107062   0.000197057      10     
reco:cvnmap:CVNMapper                                  0.0146826     0.023642      0.0616032     0.0188598     0.0131378       10     
reco:cvneva:CVNEvaluator                               0.676058      0.783815       1.68909      0.684241      0.301789        10     
reco:energyrecnumu:EnergyReco                         0.00449085     0.0595809     0.0983762     0.0612042     0.0220587       10     
reco:energyrecnue:EnergyReco                          0.000758463   0.00315462    0.00471781    0.00448808    0.00177221       10     
reco:energyrecnc:EnergyReco                           0.00443561    0.00996676     0.0585938    0.00457346     0.0162092       10     
[art]:TriggerResults:TriggerResultInserter             8.636e-06    1.51466e-05   5.0245e-05    1.04395e-05   1.18971e-05      10     
end_path:out1:RootOutput                               2.114e-06    5.7938e-06    2.0759e-05     4.158e-06    5.18091e-06      10     
end_path:out1:RootOutput(write)                        0.194775       0.20526      0.230749      0.205378     0.00951693       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21683.2 MB
  Peak resident set size usage (VmHWM): 2918.46 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
True Neutron Energy: 939.626
True Electron Energy: 48.0905
Completed MCParticle Loop
True Neutrino Energy: 60.8316
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 13450 at 11-Dec-2025 23:32:19 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  61
Added hit list length: 61 | Max was set to: -3.40282e+38
True Neutron Energy: 939.566
True Electron Energy: 48.7092
Completed MCParticle Loop
True Neutrino Energy: 64.181
 Size of all hits: 61
hit initial : 17.4886 | hit correction : 1.46683
hit initial : 36.4494 | hit correction : 1.46686
hit initial : 21.4745 | hit correction : 1.46654
hit initial : 43.4626 | hit correction : 1.46646
hit initial : 42.1553 | hit correction : 1.46
hit initial : 99.7673 | hit correction : 1.46574
hit initial : 190.425 | hit correction : 1.46551
hit initial : 176.913 | hit correction : 1.46529
hit initial : 160.602 | hit correction : 1.46497
hit initial : 181.567 | hit correction : 1.4646
hit initial : 242.898 | hit correction : 1.46428
hit initial : 241.541 | hit correction : 1.46407
hit initial : 182.153 | hit correction : 1.46372
hit initial : 167.785 | hit correction : 1.46328
hit initial : 174.741 | hit correction : 1.46294
hit initial : 216.571 | hit correction : 1.46255
hit initial : 261.404 | hit correction : 1.46211
hit initial : 286.033 | hit correction : 1.4616
hit initial : 287.816 | hit correction : 1.46104
hit initial : 102.982 | hit correction : 1.4599
hit initial : 273.632 | hit correction : 1.46028
hit initial : 73.8791 | hit correction : 1.46012
 *** Total Charge : 5094.02
 Looking at view: 0 | Number of hits in view: 22
hit initial : 54.1521 | hit correction : 1.45992
hit initial : 61.7287 | hit correction : 1.45993
hit initial : 29.2968 | hit correction : 1.46648
hit initial : 46.0304 | hit correction : 1.46651
hit initial : 23.4693 | hit correction : 1.46655
hit initial : 39.4986 | hit correction : 1.46689
hit initial : 40.765 | hit correction : 1.46579
hit initial : 143.72 | hit correction : 1.46565
hit initial : 220.857 | hit correction : 1.46542
hit initial : 221.67 | hit correction : 1.4651
hit initial : 220.94 | hit correction : 1.46462
hit initial : 279.693 | hit correction : 1.46425
hit initial : 243.371 | hit correction : 1.46402
hit initial : 213.099 | hit correction : 1.4636
hit initial : 197.845 | hit correction : 1.4632
hit initial : 194.245 | hit correction : 1.46289
hit initial : 183.794 | hit correction : 1.46254
hit initial : 189.5 | hit correction : 1.46218
hit initial : 197.38 | hit correction : 1.46181
hit initial : 208.774 | hit correction : 1.46144
hit initial : 188.387 | hit correction : 1.46107
hit initial : 158.787 | hit correction : 1.46071
hit initial : 172.476 | hit correction : 1.46
 *** Total Charge : 5164.32
 Looking at view: 1 | Number of hits in view: 23
hit initial : 57.0358 | hit correction : 1.45986
hit initial : 70.5252 | hit correction : 1.46652
hit initial : 172.389 | hit correction : 1.45992
hit initial : 30.5685 | hit correction : 1.46646
hit initial : 586.223 | hit correction : 1.46293
hit initial : 44.2386 | hit correction : 1.46413
hit initial : 94.2702 | hit correction : 1.46316
hit initial : 43.1143 | hit correction : 1.46688
hit initial : 40.9325 | hit correction : 1.46686
hit initial : 226.772 | hit correction : 1.46554
hit initial : 613.699 | hit correction : 1.46495
hit initial : 389.739 | hit correction : 1.4642
hit initial : 59.0549 | hit correction : 1.46339
hit initial : 456.916 | hit correction : 1.46193
hit initial : 329.478 | hit correction : 1.46115
hit initial : 225.307 | hit correction : 1.46047
 *** Total Charge : 5033.69
 Looking at view: 2 | Number of hits in view: 16
11-Dec-2025 23:32:21 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.0486883     0.0588872     0.0862779     0.0584235     0.0109993       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000366813   0.000525961   0.000732594   0.000449108   0.000151654      10     
end_path:ana:AnalyzeEvents         0.0478191     0.0581792     0.085099      0.057842      0.0108887       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19473.7 MB
  Peak resident set size usage (VmHWM): 1534.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 04:23:20 UTC       justIN version: 01.05.03