Jobsub ID 37763.30@dunegpschedd01.fnal.gov

Jobsub ID	37763.30@dunegpschedd01.fnal.gov
Workflow ID	1912
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-31 21:09:37
Site	US_Wisconsin
Entry	HCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat	2025-08-31 21:43:34
From worker node	Hostname	e2589
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	82800 (23 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-31 21:19:14
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_346_20231122T085732Z_gen_g4_detsim_hitreco__20240503T065538Z_reco2.root
Jobscript	Exit code	0
	Real time	24m (1448s)
	CPU time	5m (327s = 22%)
	Max RSS bytes	1944956928 (1854 MiB)
Outputting started	2025-08-31 21:43:23
Output files
Finished	2025-08-31 21:43:34
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0.000212731    0.145808       3.33148     0.00724742     0.361068        100    
reco:energyrecnue:EnergyReco                          0.000112191   0.00199677     0.0424731    0.000411316   0.00567015       100    
reco:energyrecnc:EnergyReco                           0.00141637     0.311669       1.17117       0.33891      0.259434        100    
reco:energyrecnumurange:EnergyReco                    0.000213823    0.0422236      2.72331      0.0015038     0.275601        100    
reco:energyrecnumumcs:EnergyReco                      0.000460058    0.0426965      2.7565       0.0019012     0.278625        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000340653    0.0421897      2.70525     0.00189622     0.273706        100    
reco:opdec:Deconvolution                               0.336212       0.46537       1.09742       0.4461       0.0987941       100    
reco:ophitspe:OpHitFinderDeco                          0.994805      0.999182       1.00458      0.999104     0.00196424       100    
reco:opflash:OpFlashFinder                            0.000208023   0.000974048    0.0027245    0.000906771   0.00044565       100    
reco:opslicer:OpSlicer                                0.00014292     0.167648      0.628646      0.130346      0.138299        100    
reco:rns:RandomNumberSaver                            1.8725e-05    3.23546e-05   0.000325725   2.70105e-05   3.04997e-05      100    
reco:anglereconue:NuAngularReco                       0.000150304   0.00126378     0.0297773    0.00032353    0.00365763       100    
reco:anglereconumu:NuAngularReco                      6.1927e-05    0.00107402     0.0285636    0.00017747    0.00352109       100    
reco:anglereconuepfps:NuAngularReco                   9.7243e-05     0.0011786     0.0286234    0.000256478    0.0035178       100    
reco:anglereconumupfps:NuAngularReco                  8.2275e-05     0.0011143     0.028887     0.000215902   0.00354384       100    
reco:anglerecohits:NuAngularReco                      8.7285e-05     0.001586      0.0311125    0.000493787   0.00397845       100    
[art]:TriggerResults:TriggerResultInserter             8.836e-06    1.12176e-05   7.2316e-05     9.944e-06    6.33502e-06      100    
end_path:out1:RootOutput                               2.696e-06    4.50189e-06   2.3364e-05    3.3365e-06    2.8511e-06       100    
end_path:out1:RootOutput(write)                       0.00705421      0.1946       0.454857      0.227392      0.103764        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37843.8 MB
  Peak resident set size usage (VmHWM): 1944.96 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 16:43:22 CDT  run: 6220954 subRun: 1 event: 34700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 2
%MSG
31-Aug-2025 16:43:22 CDT  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_346_20231122T085732Z_gen_g4_detsim_hitreco__20240503T065538Z_reco2_truthvtx_2025-08-31T_214237Z.root"
31-Aug-2025 16:43:22 CDT  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_346_20231122T085732Z_gen_g4_detsim_hitreco__20240503T065538Z_reco2_truthvtx_2025-08-31T_211935Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 16:43:22 CDT  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 29.026015 Real = 31.124549

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 36533.7 VmHWM = 1112.63

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
37763.30_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_346_20231122T085732Z_gen_g4_detsim_hitreco__20240503T065538Z_reco2_truthvtx_2025-08-31T_211935Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_346_20231122T085732Z_gen_g4_detsim_hitreco__20240503T065538Z_reco2_truthvtx_2025-08-31T_214237Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_346_20231122T085732Z_gen_g4_detsim_hitreco__20240503T065538Z_reco2_truthvtx_2025-08-31T_214237Z.root
caf_truthvtx_2025-08-31T_214237Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-31T_211935Z_vtx.log