Jobsub ID 31128.55@dunegpschedd02.fnal.gov

Jobsub ID	31128.55@dunegpschedd02.fnal.gov
Workflow ID	1912
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-31 22:14:40
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-08-31 23:18:53
From worker node	Hostname	wn54
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-31 22:16:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_695_20231207T113458Z_gen_g4_detsim_hitreco__20240510T041933Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3719s)
	CPU time	10m (643s = 17%)
	Max RSS bytes	1834053632 (1749 MiB)
Outputting started	2025-08-31 23:18:11
Output files
Finished	2025-08-31 23:18:53
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

5    0.226371       1.58976      0.0238683     0.341795        100    
reco:energyrecnue:EnergyReco                          0.000280059   0.00585673     0.134704     0.00111602     0.0194935       100    
reco:energyrecnc:EnergyReco                           0.00219181     0.843013       3.01295      0.944639      0.558188        100    
reco:energyrecnumurange:EnergyReco                    0.000491085    0.0645517      1.29067     0.00440524     0.174491        100    
reco:energyrecnumumcs:EnergyReco                      0.00148029     0.0625747      1.17736     0.00501289      0.16137        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00135069     0.0641272      1.28253     0.00519093     0.169132        100    
reco:opdec:Deconvolution                               0.516521       1.16745       1.83009       1.15845      0.307798        100    
reco:ophitspe:OpHitFinderDeco                           1.37966       1.40794       1.54286       1.40455      0.0261883       100    
reco:opflash:OpFlashFinder                            0.000177827   0.00214058    0.00734618    0.00197854    0.00103171       100    
reco:opslicer:OpSlicer                                4.7039e-05     0.374628       1.42365      0.276821       0.29461        100    
reco:rns:RandomNumberSaver                            1.9757e-05    5.0859e-05    0.000652403   4.15755e-05   6.20081e-05      100    
reco:anglereconue:NuAngularReco                       0.000167909   0.00383745     0.116474     0.00059612     0.014358        100    
reco:anglereconumu:NuAngularReco                      0.00010307    0.00365509     0.119032     0.000413319    0.0144373       100    
reco:anglereconuepfps:NuAngularReco                   0.000145082   0.00384015     0.118331     0.000564009    0.014388        100    
reco:anglereconumupfps:NuAngularReco                  0.000136294   0.00380763     0.120256     0.000574955    0.0145731       100    
reco:anglerecohits:NuAngularReco                      0.000111728    0.004811      0.129089     0.00116348     0.0158575       100    
[art]:TriggerResults:TriggerResultInserter            1.2076e-05    2.35165e-05   0.000108449   2.23885e-05   1.05553e-05      100    
end_path:out1:RootOutput                               3.232e-06    7.42193e-06   3.8151e-05     5.345e-06    6.0464e-06       100    
end_path:out1:RootOutput(write)                       0.00454721     0.433052       1.08879      0.481904      0.239749        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8832.59 MB
  Peak resident set size usage (VmHWM): 1834.05 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 20:18:08 -03  run: 50620606 subRun: 1 event: 69600
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 6
%MSG
31-Aug-2025 20:18:09 -03  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_695_20231207T113458Z_gen_g4_detsim_hitreco__20240510T041933Z_reco2_truthvtx_2025-08-31T_231701Z.root"
31-Aug-2025 20:18:09 -03  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_695_20231207T113458Z_gen_g4_detsim_hitreco__20240510T041933Z_reco2_truthvtx_2025-08-31T_221616Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 20:18:09 -03  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 32.673011 Real = 40.352720

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 7506.64 VmHWM = 1074.81

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
31128.55_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_695_20231207T113458Z_gen_g4_detsim_hitreco__20240510T041933Z_reco2_truthvtx_2025-08-31T_221616Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_695_20231207T113458Z_gen_g4_detsim_hitreco__20240510T041933Z_reco2_truthvtx_2025-08-31T_231701Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_695_20231207T113458Z_gen_g4_detsim_hitreco__20240510T041933Z_reco2_truthvtx_2025-08-31T_231701Z.root
caf_truthvtx_2025-08-31T_231701Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-31T_221616Z_vtx.log