Jobsub ID 30642.12@dunegpschedd02.fnal.gov

Jobsub ID	30642.12@dunegpschedd02.fnal.gov
Workflow ID	1901
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-31 10:21:06
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce04
Last heartbeat	2025-08-31 10:28:00
From worker node	Hostname	ccwcondor0052
	cpuinfo	AMD EPYC 9334 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	106200 (29 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-31 10:22:00
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_80_20231208T091505Z_gen_g4_detsim_hitreco__20240510T064522Z_reco2.root
Jobscript	Exit code	0
	Real time	5m (336s)
	CPU time	5m (312s = 92%)
	Max RSS bytes	1985576960 (1893 MiB)
Outputting started	2025-08-31 10:27:37
Output files
Finished	2025-08-31 10:28:00
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0.000299555    0.0295318     0.619609     0.00646506     0.0883639       100    
reco:energyrecnue:EnergyReco                          0.000153743   0.00206086     0.0232925    0.000501404   0.00412293       100    
reco:energyrecnc:EnergyReco                           0.00205567     0.0150646     0.0668248     0.0124418     0.0117167       100    
reco:energyrecnumurange:EnergyReco                    0.000272273     0.02308      0.553105     0.00255041     0.0779058       100    
reco:energyrecnumumcs:EnergyReco                      0.000631186    0.0233659     0.543645      0.0029409     0.0767929       100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000516492    0.022796      0.534977     0.00285085     0.0760642       100    
reco:opdec:Deconvolution                               0.025024      0.152598      0.307146      0.150381      0.0559916       100    
reco:ophitspe:OpHitFinderDeco                         0.00464522     0.0129316     0.0337534     0.011045     0.00626419       100    
reco:opflash:OpFlashFinder                            0.00030362    0.00140096    0.00430068    0.00132184    0.000643597      100    
reco:opslicer:OpSlicer                                 0.0011463     0.245957      0.746832      0.204423      0.184102        100    
reco:rns:RandomNumberSaver                            2.1022e-05    4.78619e-05   0.000572218   3.79625e-05   5.4343e-05       100    
reco:anglereconue:NuAngularReco                       0.000176347   0.00122006     0.0139836    0.00044133    0.00231394       100    
reco:anglereconumu:NuAngularReco                      6.5138e-05    0.000991431    0.0130692    0.000217866   0.00227972       100    
reco:anglereconuepfps:NuAngularReco                   0.00011147    0.00114949     0.0132775    0.000338483   0.00229229       100    
reco:anglereconumupfps:NuAngularReco                  8.9867e-05    0.00106796     0.0134221    0.000264641   0.00233488       100    
reco:anglerecohits:NuAngularReco                      0.000327727   0.00179013     0.0157334    0.000668187   0.00291723       100    
[art]:TriggerResults:TriggerResultInserter             9.364e-06    1.55882e-05   9.2962e-05    1.36205e-05   8.91034e-06      100    
end_path:out1:RootOutput                               3.596e-06    5.10962e-06   2.8433e-05     4.236e-06    2.90374e-06      100    
end_path:out1:RootOutput(write)                        0.0163776     0.280116      0.552982      0.314734      0.135284        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20716.6 MB
  Peak resident set size usage (VmHWM): 1982.11 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 2
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 3
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:27:35 CEST  run: 6472782 subRun: 1 event: 8100
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 3
%MSG
31-Aug-2025 12:27:36 CEST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_80_20231208T091505Z_gen_g4_detsim_hitreco__20240510T064522Z_reco2_truthvtx_2025-08-31T_102651Z.root"
31-Aug-2025 12:27:36 CEST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_80_20231208T091505Z_gen_g4_detsim_hitreco__20240510T064522Z_reco2_truthvtx_2025-08-31T_102208Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 12:27:36 CEST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 20.734192 Real = 22.121874

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 19170.1 VmHWM = 916.152

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
30642.12_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_80_20231208T091505Z_gen_g4_detsim_hitreco__20240510T064522Z_reco2_truthvtx_2025-08-31T_102208Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_80_20231208T091505Z_gen_g4_detsim_hitreco__20240510T064522Z_reco2_truthvtx_2025-08-31T_102651Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_80_20231208T091505Z_gen_g4_detsim_hitreco__20240510T064522Z_reco2_truthvtx_2025-08-31T_102651Z.root
caf_truthvtx_2025-08-31T_102651Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-31T_102208Z_vtx.log