Jobsub ID 278965.37@dunegpschedd01.fnal.gov

Jobsub ID	278965.37@dunegpschedd01.fnal.gov
Workflow ID	11954
Stage ID	1
User name	cdalmazz@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2026-01-15 09:29:41
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2026-01-15 18:47:45
From worker node	Hostname	hnode56.pic.es
	cpuinfo	AMD EPYC 7402P 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2026-01-15 09:30:57
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root
Jobscript	Exit code	0
	Real time	8h (29094s)
	CPU time	4h (14785s = 50%)
	Max RSS bytes	7222669312 (6888 MiB)
Outputting started	2026-01-15 17:35:52
Output files
Finished	2026-01-15 18:47:45
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Events option -n 99
Input PFN = root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/63/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
using fcl file for detsim /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/823e20dd2e5f60e9cad29386e43e37edfd68bdc2/fcls/detsyst_detsim_dune10kt_1x2x6.fcl
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e26-p3913-prof/lib/libXrdPosixPreload.so
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
using fcl file for reco2 /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/823e20dd2e5f60e9cad29386e43e37edfd68bdc2/fcls/detsyst_reco2_dune10kt_1x2x6.fcl
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e26-p3913-prof/lib/libXrdPosixPreload.so
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
using fcl file for CAFmaker cafmaker_atmos_dune10kt_1x2x6_runreco-nuenergy-nuangular_geov5.fcl
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
1+1 records in
1+1 records out
1028635 bytes (1.0 MB) copied, 0.00498504 s, 206 MB/s
=== Start last 100 lines of lar log file ===
Assuming the FD HD detector
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 7
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 15-Jan-2026 18:35:50 CET  run: 6430442 subRun: 1 event: 106999
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 7
%MSG
15-Jan-2026 18:35:50 CET  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_2026-01-15T_093307Z_g4_detsim_reco1_reco2.root"
%MSG-i NuRandomService:  CAFMaker:cafmaker@EndJob 15-Jan-2026 18:35:50 CET  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 99 passed = 99 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         99         99          0 cafmaker

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 19.349158 Real = 20.117217

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 8524.48 VmHWM = 1016.25

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0