Jobsub ID 29125.52@dunegpschedd02.fnal.gov

Jobsub ID	29125.52@dunegpschedd02.fnal.gov
Workflow ID	1892
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-29 17:25:53
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce01
Last heartbeat	2025-08-29 18:15:19
From worker node	Hostname	wn2207300.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7513 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-29 18:03:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_336_20231207T153626Z_gen_g4_detsim_hitreco__20240510T064229Z_reco2.root
Jobscript	Exit code	0
	Real time	11m (669s)
	CPU time	7m (451s = 67%)
	Max RSS bytes	1821413376 (1737 MiB)
Outputting started	2025-08-29 18:14:32
Output files
Finished	2025-08-29 18:15:19
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

5    0.092334       1.03795      0.0152289     0.169349        100    
reco:energyrecnue:EnergyReco                          0.000144221   0.00364892     0.115012     0.000763475    0.013266        100    
reco:energyrecnc:EnergyReco                           0.00304615     0.129611      0.532704      0.140908      0.0991496       100    
reco:energyrecnumurange:EnergyReco                    0.000406223    0.0452148      1.10511      0.0032813     0.150531        100    
reco:energyrecnumumcs:EnergyReco                      0.00049999     0.0454689      1.09893     0.00371156     0.148114        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000346662    0.0465046      1.09492     0.00373811     0.149779        100    
reco:opdec:Deconvolution                               0.103314      0.273966      0.498964      0.269877      0.0819171       100    
reco:ophitspe:OpHitFinderDeco                          0.104999       0.15976      0.328987      0.152633      0.034794        100    
reco:opflash:OpFlashFinder                            0.000397687   0.00165322    0.00627149    0.00157535    0.000836126      100    
reco:opslicer:OpSlicer                                0.00149634     0.295285       1.11641      0.231008      0.234283        100    
reco:rns:RandomNumberSaver                            2.5007e-05    8.56649e-05   0.000570312   6.5819e-05    8.05127e-05      100    
reco:anglereconue:NuAngularReco                       0.000205376   0.00250019     0.084681     0.00053674    0.00996293       100    
reco:anglereconumu:NuAngularReco                      8.8165e-05    0.00225522     0.0952537    0.000289189    0.0107394       100    
reco:anglereconuepfps:NuAngularReco                   0.000155502   0.00237197     0.0876822    0.000434017    0.0101506       100    
reco:anglereconumupfps:NuAngularReco                  0.000136637   0.00224965     0.084431     0.00037455    0.00993584       100    
reco:anglerecohits:NuAngularReco                      0.00035065    0.00311294     0.0907829    0.000869215    0.0106513       100    
[art]:TriggerResults:TriggerResultInserter             9.548e-06    1.74578e-05   7.7436e-05    1.6246e-05    8.09014e-06      100    
end_path:out1:RootOutput                               3.216e-06    6.82951e-06   3.0598e-05     5.225e-06    4.3688e-06       100    
end_path:out1:RootOutput(write)                        0.0229982     0.319809      0.749755      0.380972      0.181175        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20643.7 MB
  Peak resident set size usage (VmHWM): 1821.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:14:29 BST  run: 74568471 subRun: 1 event: 33700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 2
%MSG
29-Aug-2025 19:14:30 BST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_336_20231207T153626Z_gen_g4_detsim_hitreco__20240510T064229Z_reco2_truthvtx_2025-08-29T_181327Z.root"
29-Aug-2025 19:14:30 BST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_336_20231207T153626Z_gen_g4_detsim_hitreco__20240510T064229Z_reco2_truthvtx_2025-08-29T_180329Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 29-Aug-2025 19:14:30 BST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 37.492778 Real = 38.724432

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 19337.8 VmHWM = 1076.78

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
29125.52_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_336_20231207T153626Z_gen_g4_detsim_hitreco__20240510T064229Z_reco2_truthvtx_2025-08-29T_180329Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_336_20231207T153626Z_gen_g4_detsim_hitreco__20240510T064229Z_reco2_truthvtx_2025-08-29T_181327Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_336_20231207T153626Z_gen_g4_detsim_hitreco__20240510T064229Z_reco2_truthvtx_2025-08-29T_181327Z.root
caf_truthvtx_2025-08-29T_181327Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-29T_180329Z_vtx.log