Jobsub ID 36861.107@dunegpschedd01.fnal.gov

Jobsub ID	36861.107@dunegpschedd01.fnal.gov
Workflow ID	1922
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-30 22:24:26
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce510
Last heartbeat	2025-08-30 22:43:53
From worker node	Hostname	b9p08p0553.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	343800 (95 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-30 22:33:12
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_901_20231202T122739Z_gen_g4_detsim_hitreco__20240507T212846Z_reco2.root
Jobscript	Exit code	0
	Real time	10m (612s)
	CPU time	7m (424s = 69%)
	Max RSS bytes	1917804544 (1828 MiB)
Outputting started	2025-08-30 22:43:26
Output files
Finished	2025-08-30 22:43:53
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

924099      0.0208838     0.102718        100    
reco:energyrecnue:EnergyReco                          0.000174793   0.00224505     0.0772751    0.000679006    0.0077758       100    
reco:energyrecnc:EnergyReco                           0.00252145     0.0598025      0.35179      0.0599418     0.0561696       100    
reco:energyrecnumurange:EnergyReco                    0.000338516    0.0248854     0.822089     0.00279133     0.0922796       100    
reco:energyrecnumumcs:EnergyReco                       0.0014137     0.0251747     0.814917     0.00332329     0.0906132       100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00108208     0.0252278     0.811604     0.00320676     0.0905234       100    
reco:opdec:Deconvolution                               0.0497197      0.1999       0.447433      0.199498      0.0729888       100    
reco:ophitspe:OpHitFinderDeco                          0.143484      0.150584      0.162361      0.150498     0.00338604       100    
reco:opflash:OpFlashFinder                            0.000197984   0.00129347    0.00422125    0.00120473    0.000659977      100    
reco:opslicer:OpSlicer                                5.7371e-05     0.244227       1.05515      0.154093      0.228533        100    
reco:rns:RandomNumberSaver                             2.197e-05    4.8232e-05    0.000425777   3.96105e-05   4.14642e-05      100    
reco:anglereconue:NuAngularReco                       0.000178303   0.00133452     0.0547818    0.000399922   0.00549007       100    
reco:anglereconumu:NuAngularReco                      8.2032e-05    0.00108617     0.0508994    0.000219978   0.00512047       100    
reco:anglereconuepfps:NuAngularReco                   0.000129113    0.0012248     0.0504728    0.000355666   0.00506629       100    
reco:anglereconumupfps:NuAngularReco                  0.000113432   0.00114222     0.0502661    0.000289075   0.00505322       100    
reco:anglerecohits:NuAngularReco                      0.000118522    0.0018466     0.0577609    0.000773438   0.00583324       100    
[art]:TriggerResults:TriggerResultInserter             9.51e-06     1.71971e-05   0.000139362   1.4351e-05    1.45381e-05      100    
end_path:out1:RootOutput                               3.34e-06     5.54202e-06   3.3241e-05    4.3755e-06    4.32934e-06      100    
end_path:out1:RootOutput(write)                       0.00494704     0.251355      0.798071      0.331449      0.168351        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20667.9 MB
  Peak resident set size usage (VmHWM): 1917.8 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 00:43:23 CEST  run: 6403947 subRun: 1 event: 90200
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
31-Aug-2025 00:43:24 CEST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_901_20231202T122739Z_gen_g4_detsim_hitreco__20240507T212846Z_reco2_truthvtx_2025-08-30T_224235Z.root"
31-Aug-2025 00:43:24 CEST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_901_20231202T122739Z_gen_g4_detsim_hitreco__20240507T212846Z_reco2_truthvtx_2025-08-30T_223321Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 00:43:24 CEST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 23.070061 Real = 24.313224

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 19479.1 VmHWM = 1211.08

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
36861.107_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_901_20231202T122739Z_gen_g4_detsim_hitreco__20240507T212846Z_reco2_truthvtx_2025-08-30T_223321Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_901_20231202T122739Z_gen_g4_detsim_hitreco__20240507T212846Z_reco2_truthvtx_2025-08-30T_224235Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_901_20231202T122739Z_gen_g4_detsim_hitreco__20240507T212846Z_reco2_truthvtx_2025-08-30T_224235Z.root
caf_truthvtx_2025-08-30T_224235Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-30T_223321Z_vtx.log