Jobsub ID 29125.56@dunegpschedd02.fnal.gov

Jobsub ID	29125.56@dunegpschedd02.fnal.gov
Workflow ID	1892
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-29 17:25:53
Site	UK_Oxford
Entry	DUNE_UK_SGrid_Oxford_arc01
Last heartbeat	2025-08-29 18:36:40
From worker node	Hostname	t2wn169.physics.ox.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-29 18:04:07
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_48_20231203T121933Z_gen_g4_detsim_hitreco__20240508T051550Z_reco2.root
Jobscript	Exit code	0
	Real time	31m (1894s)
	CPU time	9m (577s = 30%)
	Max RSS bytes	1940258816 (1850 MiB)
Outputting started	2025-08-29 18:35:42
Output files
Finished	2025-08-29 18:36:40
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0.000328579    0.327964       21.7939      0.0242705      2.16981        100    
reco:energyrecnue:EnergyReco                          0.000186752   0.00335122     0.0783885    0.000927456   0.00872046       100    
reco:energyrecnc:EnergyReco                           0.00258262     0.331324       2.5543       0.245725      0.371798        100    
reco:energyrecnumurange:EnergyReco                    0.00031326     0.210396       15.9599     0.00337738      1.59616        100    
reco:energyrecnumumcs:EnergyReco                      0.00140859     0.245835       19.6354     0.00433806      1.9585         100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00115806     0.238482       19.1209     0.00463313      1.90673        100    
reco:opdec:Deconvolution                               0.182449      0.602589       1.34004      0.561147      0.261928        100    
reco:ophitspe:OpHitFinderDeco                          0.216868       0.26549      0.672794      0.245882      0.0691144       100    
reco:opflash:OpFlashFinder                            0.000185188   0.00163074    0.00654985    0.00154007    0.000832154      100    
reco:opslicer:OpSlicer                                5.6727e-05     0.315434      0.899185      0.260898      0.235674        100    
reco:rns:RandomNumberSaver                            2.4627e-05    5.18036e-05   0.000577918   3.8742e-05    5.57493e-05      100    
reco:anglereconue:NuAngularReco                       0.000192081   0.00241282     0.0576786    0.000575037   0.00649594       100    
reco:anglereconumu:NuAngularReco                      8.3517e-05    0.00206313     0.0570359    0.000309683   0.00641748       100    
reco:anglereconuepfps:NuAngularReco                   0.000125987   0.00223444     0.0574866    0.00045959    0.00645537       100    
reco:anglereconumupfps:NuAngularReco                  0.00010633    0.00213614     0.0572546    0.00039043     0.006448        100    
reco:anglerecohits:NuAngularReco                      0.000185229   0.00305087     0.0644844    0.000992202   0.00726205       100    
[art]:TriggerResults:TriggerResultInserter            1.0018e-05    1.6469e-05    8.6022e-05    1.4362e-05    9.01191e-06      100    
end_path:out1:RootOutput                               3.476e-06    5.81999e-06   3.1079e-05     4.714e-06    3.29749e-06      100    
end_path:out1:RootOutput(write)                       0.00856838     0.307238      0.738984      0.359123      0.167138        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20682.2 MB
  Peak resident set size usage (VmHWM): 1940.26 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 19:35:40 BST  run: 50574832 subRun: 1 event: 4900
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 4
%MSG
29-Aug-2025 19:35:41 BST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_48_20231203T121933Z_gen_g4_detsim_hitreco__20240508T051550Z_reco2_truthvtx_2025-08-29T_183321Z.root"
29-Aug-2025 19:35:41 BST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_48_20231203T121933Z_gen_g4_detsim_hitreco__20240508T051550Z_reco2_truthvtx_2025-08-29T_180416Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 29-Aug-2025 19:35:41 BST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 107.962426 Real = 112.568689

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 19363 VmHWM = 1095.69

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
29125.56_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_48_20231203T121933Z_gen_g4_detsim_hitreco__20240508T051550Z_reco2_truthvtx_2025-08-29T_180416Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_48_20231203T121933Z_gen_g4_detsim_hitreco__20240508T051550Z_reco2_truthvtx_2025-08-29T_183321Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_48_20231203T121933Z_gen_g4_detsim_hitreco__20240508T051550Z_reco2_truthvtx_2025-08-29T_183321Z.root
caf_truthvtx_2025-08-29T_183321Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-29T_180416Z_vtx.log