Jobsub ID 37637.91@dunegpschedd01.fnal.gov

Jobsub ID	37637.91@dunegpschedd01.fnal.gov
Workflow ID	1912
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-31 18:19:29
Site	US_PuertoRico
Entry	CMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat	2025-08-31 19:03:23
From worker node	Hostname	compute-0-5.local
	cpuinfo	Intel(R) Xeon(R) CPU W3550 @ 3.07GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-31 18:20:46
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_393_20231201T085722Z_gen_g4_detsim_hitreco__20240507T203456Z_reco2.root
Jobscript	Exit code	0
	Real time	42m (2534s)
	CPU time	12m (752s = 29%)
	Max RSS bytes	1646542848 (1570 MiB)
Outputting started	2025-08-31 19:03:00
Output files
Finished	2025-08-31 19:03:23
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

0.000303271    0.342076       9.13367      0.0233917     0.961528        100    
reco:energyrecnue:EnergyReco                          0.000206718   0.00607713      0.21474     0.00084435     0.0243847       100    
reco:energyrecnc:EnergyReco                           0.00214358     0.634075       1.93468      0.669109      0.494778        100    
reco:energyrecnumurange:EnergyReco                     0.0002842     0.131762       7.20593     0.00297576      0.75599        100    
reco:energyrecnumumcs:EnergyReco                      0.000671939    0.140232       8.3814      0.00361264     0.861735        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000581925    0.145073       8.62004     0.00366152     0.887575        100    
reco:opdec:Deconvolution                               0.520518      0.825319       1.72754      0.824222      0.215928        100    
reco:ophitspe:OpHitFinderDeco                           1.46698       1.47478       1.51014       1.47432     0.00581009       100    
reco:opflash:OpFlashFinder                            0.000281316   0.00150405    0.00456436     0.0014567    0.000751635      100    
reco:opslicer:OpSlicer                                0.00211394     0.282054      0.949494      0.210512      0.228353        100    
reco:rns:RandomNumberSaver                            1.9996e-05    2.91534e-05   0.000365417   2.45455e-05   3.40696e-05      100    
reco:anglereconue:NuAngularReco                       0.000154406   0.00331068     0.113462     0.000372431    0.0131223       100    
reco:anglereconumu:NuAngularReco                      0.000103045   0.00316567     0.110225     0.00025928     0.0128446       100    
reco:anglereconuepfps:NuAngularReco                   0.000147567   0.00328949     0.109512     0.000366032    0.0128016       100    
reco:anglereconumupfps:NuAngularReco                  0.000147303   0.00326729     0.109677     0.000358141    0.0128218       100    
reco:anglerecohits:NuAngularReco                      0.000411155   0.00414266     0.124585     0.000825142    0.0144162       100    
[art]:TriggerResults:TriggerResultInserter            1.2406e-05    1.50112e-05   5.8061e-05    1.36125e-05   4.92535e-06      100    
end_path:out1:RootOutput                               2.83e-06     3.31377e-06   1.5274e-05     3.056e-06    1.24784e-06      100    
end_path:out1:RootOutput(write)                        0.0187524     0.321034      0.807391      0.335959      0.174767        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4545.25 MB
  Peak resident set size usage (VmHWM): 1646.54 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 4
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 5
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 6
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 15:02:58 AST  run: 6374820 subRun: 1 event: 39400
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 6
%MSG
31-Aug-2025 15:02:59 AST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_393_20231201T085722Z_gen_g4_detsim_hitreco__20240507T203456Z_reco2_truthvtx_2025-08-31T_190102Z.root"
31-Aug-2025 15:02:59 AST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_393_20231201T085722Z_gen_g4_detsim_hitreco__20240507T203456Z_reco2_truthvtx_2025-08-31T_182052Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 15:02:59 AST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 cafmaker
TrigReport        100        100          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 79.424909 Real = 93.011297

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3281.42 VmHWM = 952.57

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
37637.91_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_393_20231201T085722Z_gen_g4_detsim_hitreco__20240507T203456Z_reco2_truthvtx_2025-08-31T_182052Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_393_20231201T085722Z_gen_g4_detsim_hitreco__20240507T203456Z_reco2_truthvtx_2025-08-31T_190102Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_393_20231201T085722Z_gen_g4_detsim_hitreco__20240507T203456Z_reco2_truthvtx_2025-08-31T_190102Z.root
caf_truthvtx_2025-08-31T_190102Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-31T_182052Z_vtx.log