Jobsub ID 37436.8@dunegpschedd01.fnal.gov

Jobsub ID	37436.8@dunegpschedd01.fnal.gov
Workflow ID	1901
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-31 14:13:15
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce02
Last heartbeat	2025-08-31 16:00:13
From worker node	Hostname	wn30
	cpuinfo	Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-31 14:14:06
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_11_20231028T041140Z_gen_g4_detsim_hitreco__20240503T045314Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6321s)
	CPU time	43m (2580s = 40%)
	Max RSS bytes	1906323456 (1818 MiB)
Outputting started	2025-08-31 15:59:28
Output files
Finished	2025-08-31 16:00:13
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

6959       6.32631       1.85678      0.842048        200    
reco:energyrecnumu:EnergyReco                         0.000600395     0.83064       92.4722      0.0424557      6.5569         200    
reco:energyrecnue:EnergyReco                          0.000365822   0.00769449     0.334339     0.00167031     0.0281112       200    
reco:energyrecnc:EnergyReco                           0.00329664     0.764356       2.2456       0.940881      0.495074        200    
reco:energyrecnumurange:EnergyReco                    0.000565922    0.584078       81.1138     0.00458018      5.76773        200    
reco:energyrecnumumcs:EnergyReco                      0.00150633     0.605446       84.5663     0.00625295      6.01208        200    
reco:energyrecnumumcsllhd:EnergyReco                   0.0013981       0.606        85.3933     0.00619435      6.06534        200    
reco:opdec:Deconvolution                               0.482478      0.913344       1.7697       0.880684      0.264639        200    
reco:ophitspe:OpHitFinderDeco                           1.38155       1.39946       1.56565       1.39539      0.0201567       200    
reco:opflash:OpFlashFinder                            0.00041074    0.00274159    0.00835092    0.00261486    0.00123727       200    
reco:opslicer:OpSlicer                                0.000551452    0.464315       1.87941      0.373085      0.368775        200    
reco:rns:RandomNumberSaver                            3.7034e-05    8.40255e-05   0.000856662   6.3453e-05    0.000113412      200    
reco:anglereconue:NuAngularReco                       0.000286274   0.00470837     0.168865     0.000803132    0.0160467       200    
reco:anglereconumu:NuAngularReco                      0.000175965   0.00440769     0.172235     0.000580807    0.0159927       200    
reco:anglereconuepfps:NuAngularReco                   0.000255541   0.00464869     0.171017     0.000789234    0.0159956       200    
reco:anglereconumupfps:NuAngularReco                  0.000255442   0.00462404     0.169704     0.000736764    0.0159595       200    
reco:anglerecohits:NuAngularReco                      0.00025763    0.00604972     0.188483     0.00166597     0.0176175       200    
[art]:TriggerResults:TriggerResultInserter            1.7156e-05    3.32967e-05   0.000102028   3.1105e-05    9.27379e-06      200    
end_path:out1:RootOutput                               4.714e-06    1.05308e-05   3.7601e-05    1.0542e-05    5.13768e-06      200    
end_path:out1:RootOutput(write)                         0.01884       0.49913       1.23755      0.556232      0.246687        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7758.54 MB
  Peak resident set size usage (VmHWM): 1906.32 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 12:59:24 -03  run: 1794 subRun: 1 event: 2400
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
31-Aug-2025 12:59:26 -03  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_11_20231028T041140Z_gen_g4_detsim_hitreco__20240503T045314Z_reco2_truthvtx_2025-08-31T_155044Z.root"
31-Aug-2025 12:59:26 -03  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_11_20231028T041140Z_gen_g4_detsim_hitreco__20240503T045314Z_reco2_truthvtx_2025-08-31T_141413Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 12:59:26 -03  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 200 passed = 200 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        200        200          0 cafmaker
TrigReport        200        200          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 420.115738 Real = 472.778298

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6566.81 VmHWM = 1207.21

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
37436.8_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_11_20231028T041140Z_gen_g4_detsim_hitreco__20240503T045314Z_reco2_truthvtx_2025-08-31T_141413Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_11_20231028T041140Z_gen_g4_detsim_hitreco__20240503T045314Z_reco2_truthvtx_2025-08-31T_155044Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_11_20231028T041140Z_gen_g4_detsim_hitreco__20240503T045314Z_reco2_truthvtx_2025-08-31T_155044Z.root
caf_truthvtx_2025-08-31T_155044Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-31T_141413Z_vtx.log