Jobsub ID 36756.23@dunegpschedd01.fnal.gov

Jobsub ID	36756.23@dunegpschedd01.fnal.gov
Workflow ID	1895
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-30 20:31:22
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-08-30 20:52:15
From worker node	Hostname	wn-sate-040.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-30 20:43:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_975_20231204T010113Z_gen_g4_detsim_hitreco__20240509T211722Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (509s)
	CPU time	8m (498s = 97%)
	Max RSS bytes	1935740928 (1846 MiB)
Outputting started	2025-08-30 20:51:56
Output files
Finished	2025-08-30 20:52:15
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

.31199     0.00873689     0.185274        100    
reco:energyrecnue:EnergyReco                          0.00019759    0.00354616      0.16638     0.000592226    0.0167683       100    
reco:energyrecnc:EnergyReco                           0.00283111     0.0152618     0.236811      0.012174      0.0241701       100    
reco:energyrecnumurange:EnergyReco                    0.000345699    0.0403859      1.16234     0.00288324     0.157008        100    
reco:energyrecnumumcs:EnergyReco                      0.000976663    0.0413621      1.16323     0.00362641     0.156382        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000802355    0.0413288      1.16168     0.00367566     0.156373        100    
reco:opdec:Deconvolution                               0.0433894     0.180396      0.309761      0.187101      0.0595743       100    
reco:ophitspe:OpHitFinderDeco                         0.00386845    0.00853921     0.0146505    0.00845865    0.00227277       100    
reco:opflash:OpFlashFinder                            0.000331843   0.00131384     0.0040264     0.0012813    0.00055098       100    
reco:opslicer:OpSlicer                                0.00536064     0.262047      0.876048      0.245813      0.180476        100    
reco:rns:RandomNumberSaver                            2.0438e-05    3.11404e-05   0.00035187    2.76665e-05   3.23201e-05      100    
reco:anglereconue:NuAngularReco                       0.000161373   0.00234831     0.118297     0.000371342    0.0119294       100    
reco:anglereconumu:NuAngularReco                      9.5379e-05    0.00221705     0.117756     0.000247895    0.0118943       100    
reco:anglereconuepfps:NuAngularReco                   0.000140123   0.00234077     0.118645     0.000342388     0.01197        100    
reco:anglereconumupfps:NuAngularReco                  0.000133471   0.00232085     0.120613     0.000307492    0.012172        100    
reco:anglerecohits:NuAngularReco                      0.000412704   0.00293836     0.126708     0.000694924    0.0127974       100    
[art]:TriggerResults:TriggerResultInserter             1.082e-05    1.33941e-05   6.5213e-05    1.2418e-05    5.44678e-06      100    
end_path:out1:RootOutput                               3.025e-06    3.8693e-06    2.0549e-05    3.6265e-06    1.79718e-06      100    
end_path:out1:RootOutput(write)                        0.0232027     0.276989      0.789621      0.307936      0.133341        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7757.94 MB
  Peak resident set size usage (VmHWM): 1935.74 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 30-Aug-2025 22:51:55 CEST  run: 6419587 subRun: 1 event: 97600
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
30-Aug-2025 22:51:55 CEST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_975_20231204T010113Z_gen_g4_detsim_hitreco__20240509T211722Z_reco2_truthvtx_2025-08-30T_205101Z.root"
30-Aug-2025 22:51:55 CEST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_975_20231204T010113Z_gen_g4_detsim_hitreco__20240509T211722Z_reco2_truthvtx_2025-08-30T_204331Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 30-Aug-2025 22:51:55 CEST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 34.025851 Real = 34.368574

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6547.96 VmHWM = 1193.36

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
36756.23_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_975_20231204T010113Z_gen_g4_detsim_hitreco__20240509T211722Z_reco2_truthvtx_2025-08-30T_204331Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_975_20231204T010113Z_gen_g4_detsim_hitreco__20240509T211722Z_reco2_truthvtx_2025-08-30T_205101Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_975_20231204T010113Z_gen_g4_detsim_hitreco__20240509T211722Z_reco2_truthvtx_2025-08-30T_205101Z.root
caf_truthvtx_2025-08-30T_205101Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-30T_204331Z_vtx.log