Jobsub ID 37298.3@dunegpschedd01.fnal.gov

Jobsub ID	37298.3@dunegpschedd01.fnal.gov
Workflow ID	1895
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-31 10:21:04
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce04
Last heartbeat	2025-08-31 10:30:18
From worker node	Hostname	ccwcondor0732
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	106200 (29 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-31 10:22:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_511_20231207T072645Z_gen_g4_detsim_hitreco__20240510T053838Z_reco2.root
Jobscript	Exit code	0
	Real time	7m (441s)
	CPU time	6m (418s = 94%)
	Max RSS bytes	1936973824 (1847 MiB)
Outputting started	2025-08-31 10:29:55
Output files
Finished	2025-08-31 10:30:18
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

e:EnergyReco                    0.000299415    0.105408       8.28765     0.00291912     0.825118        100    
reco:energyrecnumumcs:EnergyReco                      0.000766686    0.111251       8.64467     0.00351858     0.861533        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00049325     0.111214       8.62608     0.00333277     0.859609        100    
reco:opdec:Deconvolution                               0.0395744     0.160883      0.277742      0.156048      0.0585357       100    
reco:ophitspe:OpHitFinderDeco                         0.00470645     0.0122745     0.0345056     0.0112451    0.00454224       100    
reco:opflash:OpFlashFinder                            0.000343618   0.00146982    0.00474754    0.00133929    0.000670948      100    
reco:opslicer:OpSlicer                                0.00701181     0.277384      0.757818      0.212165      0.218751        100    
reco:rns:RandomNumberSaver                            2.9756e-05    5.02305e-05   0.000515522   4.1218e-05    4.86219e-05      100    
reco:anglereconue:NuAngularReco                       0.000200789   0.00154681     0.0317716    0.000537548   0.00357306       100    
reco:anglereconumu:NuAngularReco                      7.2106e-05    0.00125789     0.0295269    0.000254033   0.00338961       100    
reco:anglereconuepfps:NuAngularReco                   0.000111851   0.00142592     0.0310231    0.000385983   0.00352662       100    
reco:anglereconumupfps:NuAngularReco                  0.000114475   0.00130237     0.0269375    0.000317443    0.0031847       100    
reco:anglerecohits:NuAngularReco                      0.000328569   0.00207346     0.0312897    0.000858462   0.00377295       100    
[art]:TriggerResults:TriggerResultInserter             9.929e-06    1.60565e-05   9.6903e-05    1.3476e-05    9.4729e-06       100    
end_path:out1:RootOutput                               3.296e-06    5.59782e-06   3.0578e-05     4.814e-06    3.37775e-06      100    
end_path:out1:RootOutput(write)                        0.0265957     0.289336      0.611278      0.329274      0.144711        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12079.4 MB
  Peak resident set size usage (VmHWM): 1936.97 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconuepfps@BeginModule  31-Aug-2025 12:29:53 CEST run: 6467376 subRun: 1 event: 51200
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconuepfps@BeginModule  31-Aug-2025 12:29:53 CEST run: 6467376 subRun: 1 event: 51200
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  31-Aug-2025 12:29:53 CEST run: 6467376 subRun: 1 event: 51200
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  31-Aug-2025 12:29:53 CEST run: 6467376 subRun: 1 event: 51200
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglereconumupfps@BeginModule  31-Aug-2025 12:29:53 CEST run: 6467376 subRun: 1 event: 51200
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglerecohits@BeginModule  31-Aug-2025 12:29:53 CEST run: 6467376 subRun: 1 event: 51200
 Please note: accessing PFParticle tracks through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetTrack.
%MSG
%MSG-w DUNEAna:  NuAngularReco:anglerecohits@BeginModule  31-Aug-2025 12:29:53 CEST run: 6467376 subRun: 1 event: 51200
 Please note: accessing PFParticle showers through this method is not the recommended workflow.
 Please use DUNEAnaEventUtils::GetPFParticles and access the tracks with DUNEAnaPFParticleUtils::GetShower.
%MSG
%MSG-i CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Geo name is: dune10kt_v5_1x2x6
%MSG
%MSG-i CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Assuming the FD HD detector
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:29:53 CEST  run: 6467376 subRun: 1 event: 51200
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
31-Aug-2025 12:29:54 CEST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_511_20231207T072645Z_gen_g4_detsim_hitreco__20240510T053838Z_reco2_truthvtx_2025-08-31T_102827Z.root"
31-Aug-2025 12:29:54 CEST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_511_20231207T072645Z_gen_g4_detsim_hitreco__20240510T053838Z_reco2_truthvtx_2025-08-31T_102240Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 12:29:54 CEST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 60.244020 Real = 63.192885

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 10680 VmHWM = 1019.76

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
37298.3_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_511_20231207T072645Z_gen_g4_detsim_hitreco__20240510T053838Z_reco2_truthvtx_2025-08-31T_102240Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_511_20231207T072645Z_gen_g4_detsim_hitreco__20240510T053838Z_reco2_truthvtx_2025-08-31T_102827Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_511_20231207T072645Z_gen_g4_detsim_hitreco__20240510T053838Z_reco2_truthvtx_2025-08-31T_102827Z.root
caf_truthvtx_2025-08-31T_102827Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-31T_102240Z_vtx.log