Jobsub ID 37379.26@dunegpschedd01.fnal.gov

Jobsub ID	37379.26@dunegpschedd01.fnal.gov
Workflow ID	1895
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-31 11:49:09
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce03
Last heartbeat	2025-08-31 12:10:44
From worker node	Hostname	ccwcondor0567
	cpuinfo	AMD EPYC 7453 28-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	106200 (29 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-08-31 11:50:04
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_443_20231120T135453Z_gen_g4_detsim_hitreco__20240507T173034Z_reco2.root
Jobscript	Exit code	0
	Real time	20m (1212s)
	CPU time	12m (758s = 62%)
	Max RSS bytes	1794502656 (1711 MiB)
Outputting started	2025-08-31 12:10:18
Output files
Finished	2025-08-31 12:10:44
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

85914        200    
reco:energyrecnue:EnergyReco                          0.00012256    0.00285807     0.0457018    0.00088103    0.00537994       200    
reco:energyrecnc:EnergyReco                           0.00166461     0.0693468     0.247947      0.0701034     0.0583905       200    
reco:energyrecnumurange:EnergyReco                    0.000321161    0.0488708      2.13222     0.00325577     0.223361        200    
reco:energyrecnumumcs:EnergyReco                      0.000642333    0.0499902      2.14631      0.0035775     0.226158        200    
reco:energyrecnumumcsllhd:EnergyReco                  0.00053367     0.0501581      2.3148      0.00361264     0.232145        200    
reco:opdec:Deconvolution                               0.0716313     0.223948      0.500521      0.225643      0.0784717       200    
reco:ophitspe:OpHitFinderDeco                          0.171348      0.179171      0.255816      0.178741      0.0067284       200    
reco:opflash:OpFlashFinder                            0.000264085   0.00158424    0.00796556    0.00147171    0.000931869      200    
reco:opslicer:OpSlicer                                0.000334907    0.278946       1.06064      0.212762      0.236369        200    
reco:rns:RandomNumberSaver                            2.5177e-05    4.33115e-05   0.000649918   3.7385e-05    4.42094e-05      200    
reco:anglereconue:NuAngularReco                       0.000200937   0.00154817     0.0260838    0.000448905   0.00339483       200    
reco:anglereconumu:NuAngularReco                      8.7874e-05    0.00135789     0.0240071    0.000272531   0.00330906       200    
reco:anglereconuepfps:NuAngularReco                   0.00011715    0.00147352     0.0241832    0.000361798    0.003342        200    
reco:anglereconumupfps:NuAngularReco                  0.000100398    0.0014331     0.024368     0.000332638   0.00335234       200    
reco:anglerecohits:NuAngularReco                      0.00011175    0.00217312     0.0302042    0.000778483   0.00408506       200    
[art]:TriggerResults:TriggerResultInserter             9.207e-06    1.72008e-05   9.7042e-05    1.5429e-05    8.67099e-06      200    
end_path:out1:RootOutput                               2.936e-06    5.32755e-06   3.3072e-05     4.373e-06    2.77698e-06      200    
end_path:out1:RootOutput(write)                        0.0130934     0.282983      0.676561      0.351396      0.167265        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18502.3 MB
  Peak resident set size usage (VmHWM): 1794.5 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
InputTag: label = 'regcnneval', instance = 'regcnnresult' does not correspond to a valid RegCNNResult product
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 0
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArPfoHierarchyFeatureTool_DaughterParentHitRatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArPfoHierarchyFeatureTool_NDaughterHits3D' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDChargeFeatureTool_EndFraction' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDChargeFeatureTool_FractionalSpread' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDLinearFitFeatureTool_DiffStraightLineMean' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDLinearFitFeatureTool_Length' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDLinearFitFeatureTool_MaxFitGapLength' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDLinearFitFeatureTool_SlidingLinearFitRMS' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDOpeningAngleFeatureTool_AngleDiff' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDPCAFeatureTool_SecondaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDPCAFeatureTool_TertiaryPCARatio' for PFP with ID 1
%MSG
%MSG-w CAFMaker:  CAFMaker:cafmaker@BeginModule 31-Aug-2025 14:10:15 CEST  run: 74251126 subRun: 1 event: 88800
Unknown property 'LArThreeDVertexDistanceFeatureTool_VertexDistance' for PFP with ID 1
%MSG
31-Aug-2025 14:10:16 CEST  Closed output file "caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_443_20231120T135453Z_gen_g4_detsim_hitreco__20240507T173034Z_reco2_truthvtx_2025-08-31T_120829Z.root"
31-Aug-2025 14:10:16 CEST  Closed input file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_443_20231120T135453Z_gen_g4_detsim_hitreco__20240507T173034Z_reco2_truthvtx_2025-08-31T_115014Z.root"
%MSG-i NuRandomService:  RootOutput:out@EndJob 31-Aug-2025 14:10:16 CEST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

TrigReport ---------- Event summary -------------
TrigReport Events total = 200 passed = 200 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        200        200          0 cafmaker
TrigReport        200        200          0 out

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 70.548514 Real = 79.736487

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 17110.1 VmHWM = 999.711

Art has completed and will exit with status 0.
=== End last  100 lines of lar log file ===
=== Generated output files ===
37379.26_dunegpschedd01.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_443_20231120T135453Z_gen_g4_detsim_hitreco__20240507T173034Z_reco2_truthvtx_2025-08-31T_115014Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_443_20231120T135453Z_gen_g4_detsim_hitreco__20240507T173034Z_reco2_truthvtx_2025-08-31T_120829Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_443_20231120T135453Z_gen_g4_detsim_hitreco__20240507T173034Z_reco2_truthvtx_2025-08-31T_120829Z.root
caf_truthvtx_2025-08-31T_120829Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
truthvtx_2025-08-31T_115014Z_vtx.log