Jobsub ID 31342.7@dunegpschedd02.fnal.gov

Jobsub ID	31342.7@dunegpschedd02.fnal.gov
Workflow ID	1901
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-09-01 05:23:01
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-01 06:09:39
From worker node	Hostname	wn46
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-01 05:24:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_296_20231207T041811Z_gen_g4_detsim_hitreco__20240510T033417Z_reco2.root
Jobscript	Exit code	0
	Real time	44m (2694s)
	CPU time	13m (822s = 30%)
	Max RSS bytes	1966125056 (1875 MiB)
Outputting started	2025-09-01 06:09:12
Output files
Finished	2025-09-01 06:09:39
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0.000349652    0.233781       1.85617      0.0307938     0.361625        100    
reco:energyrecnue:EnergyReco                          0.000174389   0.00519299     0.101493     0.000927108    0.0156859       100    
reco:energyrecnc:EnergyReco                           0.00194019     0.538375       3.26748      0.477667      0.499687        100    
reco:energyrecnumurange:EnergyReco                    0.000287522    0.073337       1.71234     0.00241861     0.243512        100    
reco:energyrecnumumcs:EnergyReco                      0.000586565    0.0732709      1.71734     0.00367889     0.241705        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000446676    0.0734988      1.71862     0.00368258     0.242481        100    
reco:opdec:Deconvolution                               0.470867      0.717942       1.5512       0.628668      0.232377        100    
reco:ophitspe:OpHitFinderDeco                           1.37982       1.38943       1.46669       1.3864       0.0120411       100    
reco:opflash:OpFlashFinder                            0.000274621   0.00137691     0.0038034    0.00124363    0.000733326      100    
reco:opslicer:OpSlicer                                5.3975e-05     0.251157       1.00061      0.179946      0.225207        100    
reco:rns:RandomNumberSaver                            1.9634e-05    3.52403e-05   0.000394044    2.682e-05    3.76776e-05      100    
reco:anglereconue:NuAngularReco                       0.000156531   0.00358394     0.0799583    0.000467069    0.012059        100    
reco:anglereconumu:NuAngularReco                      8.5607e-05    0.00339721     0.0792936    0.000307593    0.0120627       100    
reco:anglereconuepfps:NuAngularReco                   0.000129731   0.00354702     0.0800349    0.00039487     0.0120626       100    
reco:anglereconumupfps:NuAngularReco                  0.000115509   0.00345037     0.0793385    0.000367885    0.0118877       100    
reco:anglerecohits:NuAngularReco                      0.000106483   0.00424154     0.0856663    0.000734221    0.0128035       100    
[art]:TriggerResults:TriggerResultInserter            1.1216e-05    1.45804e-05   6.0667e-05    1.2338e-05    5.85923e-06      100    
end_path:out1:RootOutput                               2.766e-06    3.93924e-06    1.928e-05    3.5145e-06    1.92567e-06      100    
end_path:out1:RootOutput(write)                       0.00497672     0.282473       0.72931      0.304959       0.16119        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8877.42 MB
  Peak resident set size usage (VmHWM): 1966.13 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 01-Sep-2025 03:09:10 -03  run: 6467222 subRun: 1 event: 29700
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 5
%MSG
01-Sep-2025 03:09:10 -03  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_296_20231207T041811Z_gen_g4_detsim_hitreco__20240510T033417Z_reco2_truthvtx_2025-09-01T_060745Z.root"
01-Sep-2025 03:09:10 -03  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_296_20231207T041811Z_gen_g4_detsim_hitreco__20240510T033417Z_reco2_truthvtx_2025-09-01T_052424Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 01-Sep-2025 03:09:10 -03  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 47.990114 Real = 59.897370

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 7658.55 VmHWM = 1230.55

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
31342.7_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_296_20231207T041811Z_gen_g4_detsim_hitreco__20240510T033417Z_reco2_truthvtx_2025-09-01T_052424Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_296_20231207T041811Z_gen_g4_detsim_hitreco__20240510T033417Z_reco2_truthvtx_2025-09-01T_060745Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_296_20231207T041811Z_gen_g4_detsim_hitreco__20240510T033417Z_reco2_truthvtx_2025-09-01T_060745Z.root
caf_truthvtx_2025-09-01T_060745Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-09-01T_052424Z_vtx.log