Jobsub ID 29125.43@dunegpschedd02.fnal.gov

Jobsub ID	29125.43@dunegpschedd02.fnal.gov
Workflow ID	1892
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-29 17:25:53
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce515
Last heartbeat	2025-08-29 18:20:54
From worker node	Hostname	b9p10p0331.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-29 18:04:05
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_101_20231208T091547Z_gen_g4_detsim_hitreco__20240510T063232Z_reco2.root
Jobscript	Exit code	0
	Real time	16m (987s)
	CPU time	12m (778s = 78%)
	Max RSS bytes	2031992832 (1937 MiB)
Outputting started	2025-08-29 18:20:33
Output files
Finished	2025-08-29 18:20:54
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

059819      0.0749393       100    
reco:energyrecnumurange:EnergyReco                    0.000293741     0.22164       15.8151     0.00285434      1.5843         100    
reco:energyrecnumumcs:EnergyReco                      0.000720263    0.234345       16.5045     0.00332944      1.65795        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000519272    0.224966       15.6247      0.0033566      1.57148        100    
reco:opdec:Deconvolution                               0.074559      0.209355      0.442077      0.199263      0.0750129       100    
reco:ophitspe:OpHitFinderDeco                          0.142735      0.149148      0.168796      0.148827     0.00360518       100    
reco:opflash:OpFlashFinder                            0.000333912   0.00137777    0.00517592    0.00128327    0.000726633      100    
reco:opslicer:OpSlicer                                 0.002312      0.268528       1.07617       0.21055      0.229703        100    
reco:rns:RandomNumberSaver                             2.11e-05     4.60413e-05   0.000541402   3.75405e-05   5.15317e-05      100    
reco:anglereconue:NuAngularReco                       0.00017393    0.00378119     0.129171     0.000485292    0.0158162       100    
reco:anglereconumu:NuAngularReco                       8.151e-05    0.00363478     0.133537     0.000284076    0.0161774       100    
reco:anglereconuepfps:NuAngularReco                   0.00011146    0.00375165     0.134589     0.000396202    0.0161906       100    
reco:anglereconumupfps:NuAngularReco                  9.3401e-05    0.00374835     0.138905     0.000335686    0.0166092       100    
reco:anglerecohits:NuAngularReco                      0.000299071   0.00457671     0.146352     0.000791073    0.0177872       100    
[art]:TriggerResults:TriggerResultInserter             9.16e-06     1.44493e-05    6.279e-05     1.339e-05    6.29716e-06      100    
end_path:out1:RootOutput                               3.22e-06     5.5263e-06     5.743e-05     4.375e-06    5.74649e-06      100    
end_path:out1:RootOutput(write)                        0.0162449     0.268699      0.777997      0.317209      0.147048        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20785 MB
  Peak resident set size usage (VmHWM): 2031.99 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconuepfps@BeginModule  29-Aug-2025 20:20:30 CEST run: 6472782 subRun: 1 event: 10200
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  29-Aug-2025 20:20:30 CEST run: 6472782 subRun: 1 event: 10200
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  29-Aug-2025 20:20:30 CEST run: 6472782 subRun: 1 event: 10200
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  29-Aug-2025 20:20:30 CEST run: 6472782 subRun: 1 event: 10200
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w NeutrinoAngularRecoAlg:  NuAngularReco:anglereconumupfps@BeginModule  29-Aug-2025 20:20:30 CEST run: 6472782 subRun: 1 event: 10200
 Cannot access the muon track which is needed for this angular reconstruction method.
Not able to calculate neutrino direction
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglerecohits@BeginModule  29-Aug-2025 20:20:30 CEST run: 6472782 subRun: 1 event: 10200
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglerecohits@BeginModule  29-Aug-2025 20:20:30 CEST run: 6472782 subRun: 1 event: 10200
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-i CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Geo name is: dune10kt_v5_1x2x6
%MSG
%MSG-i CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Assuming the FD HD detector
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 29-Aug-2025 20:20:30 CEST  run: 6472782 subRun: 1 event: 10200
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
29-Aug-2025 20:20:31 CEST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_101_20231208T091547Z_gen_g4_detsim_hitreco__20240510T063232Z_reco2_truthvtx_2025-08-29T_181829Z.root"
29-Aug-2025 20:20:31 CEST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_101_20231208T091547Z_gen_g4_detsim_hitreco__20240510T063232Z_reco2_truthvtx_2025-08-29T_180412Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 29-Aug-2025 20:20:31 CEST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 99.507820 Real = 101.697930

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 19611.6 VmHWM = 1343.14

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
29125.43_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_101_20231208T091547Z_gen_g4_detsim_hitreco__20240510T063232Z_reco2_truthvtx_2025-08-29T_180412Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_101_20231208T091547Z_gen_g4_detsim_hitreco__20240510T063232Z_reco2_truthvtx_2025-08-29T_181829Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_101_20231208T091547Z_gen_g4_detsim_hitreco__20240510T063232Z_reco2_truthvtx_2025-08-29T_181829Z.root
caf_truthvtx_2025-08-29T_181829Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-29T_180412Z_vtx.log