Jobsub ID 36908.4@dunegpschedd01.fnal.gov

Jobsub ID	36908.4@dunegpschedd01.fnal.gov
Workflow ID	1895
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-30 23:53:29
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-08-31 00:37:37
From worker node	Hostname	wn-sate-036.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-30 23:53:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_652_20231204T035620Z_gen_g4_detsim_hitreco__20240509T210314Z_reco2.root
Jobscript	Exit code	0
	Real time	43m (2602s)
	CPU time	41m (2510s = 96%)
	Max RSS bytes	2022903808 (1929 MiB)
Outputting started	2025-08-31 00:37:18
Output files
Finished	2025-08-31 00:37:37
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

62.71      0.00867734      6.2381         100    
reco:energyrecnue:EnergyReco                          0.000187582   0.00410251     0.297401     0.000495776    0.0295233       100    
reco:energyrecnc:EnergyReco                           0.00272336     0.0154578     0.378685      0.0126785     0.0371878       100    
reco:energyrecnumurange:EnergyReco                    0.000326173     0.51198       50.0188     0.00291883      4.97595        100    
reco:energyrecnumumcs:EnergyReco                      0.00070241      0.51398       50.0952     0.00341051      4.98342        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000553721    0.514298       50.1094     0.00337718      4.98483        100    
reco:opdec:Deconvolution                              0.00696776     0.163559      0.353923      0.162926      0.0602525       100    
reco:ophitspe:OpHitFinderDeco                         0.00471597    0.00956661     0.0240048    0.00925499    0.00256344       100    
reco:opflash:OpFlashFinder                            0.000156383   0.00115751     0.0043278    0.00112641    0.000565473      100    
reco:opslicer:OpSlicer                                4.4193e-05      0.20893      0.820777      0.177106      0.162408        100    
reco:rns:RandomNumberSaver                            1.9236e-05    3.06327e-05   0.000365997   2.7046e-05    3.38116e-05      100    
reco:anglereconue:NuAngularReco                       0.000162315   0.00240842     0.184387     0.000316175    0.0183135       100    
reco:anglereconumu:NuAngularReco                      9.3526e-05    0.00223922     0.179864     0.000192682    0.0178761       100    
reco:anglereconuepfps:NuAngularReco                   0.000142037   0.00238463     0.182884     0.00029808     0.0181668       100    
reco:anglereconumupfps:NuAngularReco                  0.000132399   0.00239143     0.187538     0.000259848    0.0186328       100    
reco:anglerecohits:NuAngularReco                      0.000104146   0.00288478     0.191727     0.000612892    0.0190225       100    
[art]:TriggerResults:TriggerResultInserter             1.092e-05    1.25032e-05    5.848e-05    1.1817e-05    4.78166e-06      100    
end_path:out1:RootOutput                               3.116e-06    4.11469e-06   2.2162e-05     3.687e-06    2.27112e-06      100    
end_path:out1:RootOutput(write)                       0.00382152     0.259712      0.858262      0.294924      0.130651        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7847.63 MB
  Peak resident set size usage (VmHWM): 2022.9 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 02:37:16 CEST  run: 6420853 subRun: 1 event: 65300
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
31-Aug-2025 02:37:16 CEST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_652_20231204T035620Z_gen_g4_detsim_hitreco__20240509T210314Z_reco2_truthvtx_2025-08-31T_003332Z.root"
31-Aug-2025 02:37:16 CEST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_652_20231204T035620Z_gen_g4_detsim_hitreco__20240509T210314Z_reco2_truthvtx_2025-08-30T_235400Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 02:37:16 CEST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 201.487380 Real = 204.259282

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6696.14 VmHWM = 1336.3

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
36908.4_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_652_20231204T035620Z_gen_g4_detsim_hitreco__20240509T210314Z_reco2_truthvtx_2025-08-30T_235400Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_652_20231204T035620Z_gen_g4_detsim_hitreco__20240509T210314Z_reco2_truthvtx_2025-08-31T_003332Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_652_20231204T035620Z_gen_g4_detsim_hitreco__20240509T210314Z_reco2_truthvtx_2025-08-31T_003332Z.root
caf_truthvtx_2025-08-31T_003332Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-30T_235400Z_vtx.log