Jobsub ID 30672.128@dunegpschedd02.fnal.gov

Jobsub ID	30672.128@dunegpschedd02.fnal.gov
Workflow ID	1901
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-31 10:23:12
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce02
Last heartbeat	2025-08-31 11:16:00
From worker node	Hostname	wn30
	cpuinfo	Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-31 10:23:46
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_237_20231207T054033Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2.root
Jobscript	Exit code	0
	Real time	51m (3104s)
	CPU time	10m (617s = 19%)
	Max RSS bytes	1949405184 (1859 MiB)
Outputting started	2025-08-31 11:15:31
Output files
Finished	2025-08-31 11:16:00
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

gyReco                    0.000502219    0.0792621      2.36385     0.00420764      0.28334        100    
reco:energyrecnumumcs:EnergyReco                      0.00115254     0.0804399      2.36492     0.00513955     0.282754        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00106834     0.0809269      2.36328     0.00513304     0.283227        100    
reco:opdec:Deconvolution                               0.505125      0.961857       1.86417      0.871655      0.307636        100    
reco:ophitspe:OpHitFinderDeco                           1.38154       1.39472       1.52129       1.39095      0.0209955       100    
reco:opflash:OpFlashFinder                            0.000518009   0.00237151    0.00532756    0.00233821    0.00113925       100    
reco:opslicer:OpSlicer                                0.00208726     0.398269       1.34107      0.344785      0.320028        100    
reco:rns:RandomNumberSaver                            3.7745e-05    0.00010378    0.00254824    4.6499e-05    0.00032421       100    
reco:anglereconue:NuAngularReco                       0.000274837   0.00312089     0.0500543    0.000657129   0.00745474       100    
reco:anglereconumu:NuAngularReco                      0.000175076   0.00282988     0.0515086    0.000421163   0.00746998       100    
reco:anglereconuepfps:NuAngularReco                   0.000245635   0.00309689     0.0519493    0.000641404   0.00759805       100    
reco:anglereconumupfps:NuAngularReco                  0.00024535    0.00298329     0.0507816    0.000594726   0.00741902       100    
reco:anglerecohits:NuAngularReco                      0.000257589   0.00420724     0.0582875    0.00146558    0.00852441       100    
[art]:TriggerResults:TriggerResultInserter            1.9341e-05    2.9325e-05    0.000124691   2.7503e-05    1.13896e-05      100    
end_path:out1:RootOutput                               4.166e-06    8.46547e-06   3.2563e-05     6.52e-06     5.45062e-06      100    
end_path:out1:RootOutput(write)                        0.0305235     0.426683      0.916769       0.53188      0.261155        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7780.46 MB
  Peak resident set size usage (VmHWM): 1949.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconuepfps@BeginModule  31-Aug-2025 08:15:28 -03 run: 50609765 subRun: 1 event: 23800
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconuepfps@BeginModule  31-Aug-2025 08:15:28 -03 run: 50609765 subRun: 1 event: 23800
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  31-Aug-2025 08:15:28 -03 run: 50609765 subRun: 1 event: 23800
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  31-Aug-2025 08:15:28 -03 run: 50609765 subRun: 1 event: 23800
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  31-Aug-2025 08:15:28 -03 run: 50609765 subRun: 1 event: 23800
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglerecohits@BeginModule  31-Aug-2025 08:15:28 -03 run: 50609765 subRun: 1 event: 23800
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglerecohits@BeginModule  31-Aug-2025 08:15:28 -03 run: 50609765 subRun: 1 event: 23800
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-i CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Geo name is: dune10kt_v5_1x2x6
%MSG
%MSG-i CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Assuming the FD HD detector
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 08:15:28 -03  run: 50609765 subRun: 1 event: 23800
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
31-Aug-2025 08:15:29 -03  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_237_20231207T054033Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2_truthvtx_2025-08-31T_111359Z.root"
31-Aug-2025 08:15:29 -03  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_237_20231207T054033Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2_truthvtx_2025-08-31T_102352Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 08:15:29 -03  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 37.559253 Real = 51.874547

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6273.27 VmHWM = 917.721

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
30672.128_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_237_20231207T054033Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2_truthvtx_2025-08-31T_102352Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_237_20231207T054033Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2_truthvtx_2025-08-31T_111359Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_237_20231207T054033Z_gen_g4_detsim_hitreco__20240510T063059Z_reco2_truthvtx_2025-08-31T_111359Z.root
caf_truthvtx_2025-08-31T_111359Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-31T_102352Z_vtx.log