Jobsub ID 38120.85@dunegpschedd02.fnal.gov

Jobsub ID	38120.85@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:00:34
Site	US_Wisconsin
Entry	HCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat	2025-09-11 11:30:53
From worker node	Hostname	e4049
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	82800 (23 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-11 10:08:07
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_634_20231203T105445Z_gen_g4_detsim_hitreco__20240509T200000Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_677_20231203T130901Z_gen_g4_detsim_hitreco__20240508T052120Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_174_20231203T072654Z_gen_g4_detsim_hitreco__20240509T203230Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_775_20231204T100513Z_gen_g4_detsim_hitreco__20240509T195930Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_419_20231203T022125Z_gen_g4_detsim_hitreco__20240508T071329Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_708_20231204T082807Z_gen_g4_detsim_hitreco__20240509T220851Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_66_20231204T120331Z_gen_g4_detsim_hitreco__20240509T201805Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_71_20231202T005731Z_gen_g4_detsim_hitreco__20240508T053526Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_47_20231203T163513Z_gen_g4_detsim_hitreco__20240509T215714Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_558_20231208T043135Z_gen_g4_detsim_hitreco__20240510T040724Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-11 11:30:53
Saved logs	justin-logs:38120.85-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

lhd:EnergyReco                  0.000324451    0.0221881     0.527947     0.00176865     0.0671862       400    
reco:opdec:Deconvolution                               0.318417      0.603792       1.13062      0.587371      0.140324        400    
reco:ophitspe:OpHitFinderDeco                          0.915576       1.02556       1.2812        1.02817      0.0660429       400    
reco:opflash:OpFlashFinder                            0.000126498   0.00130611     0.0134014    0.000940689   0.00157762       400    
reco:opslicer:OpSlicer                                3.2812e-05     0.193195      0.898382      0.146084      0.162129        400    
reco:rns:RandomNumberSaver                            1.5118e-05    3.84691e-05   0.000298182   3.7856e-05    1.72056e-05      400    
reco:anglereconue:NuAngularReco                       0.000132029   0.00158377      0.12608     0.000314559   0.00729734       400    
reco:anglereconumu:NuAngularReco                      6.1606e-05    0.00135643     0.112127     0.000156125   0.00668222       400    
reco:anglereconuepfps:NuAngularReco                   9.6582e-05    0.00152863      0.11879     0.000241942   0.00713707       400    
reco:anglereconumupfps:NuAngularReco                  8.2215e-05    0.00141651      0.12536     0.000200083   0.00733085       400    
reco:anglerecohits:NuAngularReco                      6.7437e-05    0.00190621     0.134829     0.000484683   0.00793353       400    
[art]:TriggerResults:TriggerResultInserter             7.975e-06    1.04027e-05   3.2331e-05     9.984e-06    2.50432e-06      400    
end_path:cafmaker:CAFMaker                            0.00106229     0.0300364      3.08648     0.00609501     0.176408        400    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3794.27 MB
  Peak resident set size usage (VmHWM): 2123.44 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 06:19:38 CDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/19/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_419_20231203T022125Z_gen_g4_detsim_hitreco__20240508T071329Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.902535, 0.019646, 0.0778189, 
Output 1: 0.00826127, 0.0471122, 0.101062, 0.843564, 
Output 2: 0.185803, 0.386327, 0.265419, 0.162451, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2182!
Begin processing the 394th record. run: 74517726 subRun: 1 event: 77594 at 11-Sep-2025 06:18:49 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 778!
Begin processing the 395th record. run: 74517726 subRun: 1 event: 77595 at 11-Sep-2025 06:18:55 CDT
Boundary wire vector sizes: 345, 411, 394
minwire 0: 65
minwire 1: 2106
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.742583, 0.00119298, 0.256224, 
Output 1: 0.107228, 0.871691, 0.0207046, 0.000377, 
Output 2: 0.076641, 0.74425, 0.173467, 0.00564148, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1207!
Begin processing the 396th record. run: 74517726 subRun: 1 event: 77596 at 11-Sep-2025 06:19:02 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 360!
Begin processing the 397th record. run: 74517726 subRun: 1 event: 77597 at 11-Sep-2025 06:19:07 CDT
Boundary wire vector sizes: 2260, 2177, 1442
minwire 0: 605
minwire 1: 810
minwire 2: 322
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00218923, 0.992689, 0.00512195, 
Output 1: 0.164047, 0.687505, 0.126903, 0.021545, 
Output 2: 0.0601107, 0.340437, 0.477466, 0.121987, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2985!
Begin processing the 398th record. run: 74517726 subRun: 1 event: 77598 at 11-Sep-2025 06:19:17 CDT
Boundary wire vector sizes: 171, 196, 139
minwire 0: 1357
minwire 1: 1149
minwire 2: 1543
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000337955, 0.998373, 0.00128896, 
Output 1: 0.545461, 0.447515, 0.00664708, 0.000376654, 
Output 2: 0.997881, 0.0020805, 3.72378e-05, 1.44067e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1963!
Begin processing the 399th record. run: 74517726 subRun: 1 event: 77599 at 11-Sep-2025 06:19:23 CDT
Boundary wire vector sizes: 347, 293, 284
minwire 0: 2797
minwire 1: 745
minwire 2: 2494
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.774595, 0.121192, 0.104213, 
Output 1: 0.0515096, 0.913011, 0.0345288, 0.000950215, 
Output 2: 0.536418, 0.413222, 0.0482471, 0.00211251, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1625!
Begin processing the 400th record. run: 74517726 subRun: 1 event: 77600 at 11-Sep-2025 06:19:29 CDT
Boundary wire vector sizes: 58, 58, 57
minwire 0: 776
minwire 1: 1555
minwire 2: 747
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0405892, 0.866499, 0.0929119, 
Output 1: 0.981658, 0.0169989, 0.00105407, 0.000289331, 
Output 2: 0.980233, 0.0191267, 0.000547422, 9.27995e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 854!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c2/ce/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_634_20231203T105445Z_gen_g4_detsim_hitreco__20240509T200000Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ff/86/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_677_20231203T130901Z_gen_g4_detsim_hitreco__20240508T052120Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/9f/ff/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_174_20231203T072654Z_gen_g4_detsim_hitreco__20240509T203230Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/87/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_775_20231204T100513Z_gen_g4_detsim_hitreco__20240509T195930Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/19/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_419_20231203T022125Z_gen_g4_detsim_hitreco__20240508T071329Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bb/49/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_708_20231204T082807Z_gen_g4_detsim_hitreco__20240509T220851Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c6/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_66_20231204T120331Z_gen_g4_detsim_hitreco__20240509T201805Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/b7/9b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_71_20231202T005731Z_gen_g4_detsim_hitreco__20240508T053526Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ed/64/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_47_20231203T163513Z_gen_g4_detsim_hitreco__20240509T215714Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/dc/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_558_20231208T043135Z_gen_g4_detsim_hitreco__20240510T040724Z_reco2.root
.:
total 21540
-rw-r--r-- 1 slot1_15 slot1_15 8152167 Sep 11 06:19 flatcaf.root
-rw-r--r-- 1 slot1_15 slot1_15 6753157 Sep 11 06:19 caf.root
-rw-r--r-- 1 slot1_15 slot1_15 6753157 Sep 11 06:19 caf_fd_hd_atmo_2501_20250911T100849Z.root
-rw-r--r-- 1 slot1_15 slot1_15  249278 Sep 11 06:19 caf_20250911T100849Z.log
-rw-r--r-- 1 slot1_15 slot1_15  102054 Sep 11 06:19 jobscript.log
-rw-r--r-- 1 slot1_15 slot1_15    1922 Sep 11 06:19 caf_20250911T100849Z.file
-rw-r--r-- 1 slot1_15 slot1_15    1922 Sep 11 06:19 caf_20250911T100849Z.pfns
-rw-r--r-- 1 slot1_15 slot1_15    1922 Sep 11 05:08 file.list
-rw-r--r-- 1 slot1_15 slot1_15    1922 Sep 11 06:19 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_15 slot1_15    1372 Sep 11 05:08 all-input-dids.txt
-rw-r--r-- 1 slot1_15 slot1_15    1372 Sep 11 06:19 caf_20250911T100849Z.did
-rw-r--r-- 1 slot1_15 slot1_15    1372 Sep 11 05:08 did.list
-rw-r--r-- 1 slot1_15 slot1_15       0 Sep 11 05:09 debugprod.log