Jobsub ID 29125.45@dunegpschedd02.fnal.gov

Jobsub ID	29125.45@dunegpschedd02.fnal.gov
Workflow ID	1892
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-29 17:25:53
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2025-08-29 18:11:14
From worker node	Hostname	heplnc171.pp.rl.ac.uk
	cpuinfo	AMD EPYC 9534 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-29 18:03:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_622_20231202T203540Z_gen_g4_detsim_hitreco__20240508T070156Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (416s)
	CPU time	4m (295s = 70%)
	Max RSS bytes	1953918976 (1863 MiB)
Outputting started	2025-08-29 18:10:51
Output files
Finished	2025-08-29 18:11:14
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

8007      0.0192153     0.132586        100    
reco:energyrecnue:EnergyReco                          0.000141973   0.00229886     0.0290093    0.00073038    0.00411701       100    
reco:energyrecnc:EnergyReco                           0.00129153     0.0964276     0.289591      0.113952      0.0740847       100    
reco:energyrecnumurange:EnergyReco                    0.000259663    0.0332309     0.989532     0.00222541     0.117951        100    
reco:energyrecnumumcs:EnergyReco                      0.00069841     0.0335009     0.987003     0.00284038     0.117015        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000485417    0.0321102     0.983724     0.00267842     0.115471        100    
reco:opdec:Deconvolution                               0.0464775     0.223841      0.459315      0.219958      0.0756765       100    
reco:ophitspe:OpHitFinderDeco                          0.0351126     0.0861178      0.42738      0.0773221     0.0533869       100    
reco:opflash:OpFlashFinder                            0.000211373   0.00123134    0.00575689    0.00116302    0.000700739      100    
reco:opslicer:OpSlicer                                0.000209813    0.223802       0.77828       0.18124      0.186193        100    
reco:rns:RandomNumberSaver                            2.2881e-05    5.22817e-05   0.000446026   4.4966e-05    4.26575e-05      100    
reco:anglereconue:NuAngularReco                       0.000169962   0.00137986     0.0207068    0.000440591   0.00279537       100    
reco:anglereconumu:NuAngularReco                      5.6151e-05    0.00109266     0.0195399    0.000184438   0.00265998       100    
reco:anglereconuepfps:NuAngularReco                   8.9001e-05    0.00126457     0.0196242    0.000301264   0.00277132       100    
reco:anglereconumupfps:NuAngularReco                   6.399e-05    0.00115061     0.020949     0.000254354   0.00275068       100    
reco:anglerecohits:NuAngularReco                      9.9402e-05    0.00173166     0.0228161    0.000641634   0.00301091       100    
[art]:TriggerResults:TriggerResultInserter             9.151e-06    1.45866e-05   6.2951e-05    1.32905e-05   6.47516e-06      100    
end_path:out1:RootOutput                               3.37e-06     5.88445e-06   2.0811e-05     5.355e-06    2.37584e-06      100    
end_path:out1:RootOutput(write)                        0.0121995     0.239063      0.547795      0.284916      0.144692        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37901.6 MB
  Peak resident set size usage (VmHWM): 1928.63 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:10:49 BST  run: 66006610 subRun: 1 event: 62300
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
29-Aug-2025 19:10:50 BST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_622_20231202T203540Z_gen_g4_detsim_hitreco__20240508T070156Z_reco2_truthvtx_2025-08-29T_181002Z.root"
29-Aug-2025 19:10:50 BST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_622_20231202T203540Z_gen_g4_detsim_hitreco__20240508T070156Z_reco2_truthvtx_2025-08-29T_180401Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 29-Aug-2025 19:10:50 BST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 27.497481 Real = 28.223167

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 36365.4 VmHWM = 870.154

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
29125.45_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_622_20231202T203540Z_gen_g4_detsim_hitreco__20240508T070156Z_reco2_truthvtx_2025-08-29T_180401Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_622_20231202T203540Z_gen_g4_detsim_hitreco__20240508T070156Z_reco2_truthvtx_2025-08-29T_181002Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_622_20231202T203540Z_gen_g4_detsim_hitreco__20240508T070156Z_reco2_truthvtx_2025-08-29T_181002Z.root
caf_truthvtx_2025-08-29T_181002Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-29T_180401Z_vtx.log