Jobsub ID 44845.164@dunegpschedd01.fnal.gov

Jobsub ID	44845.164@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:04:34
Site	US_Wisconsin
Entry	HCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat	2025-09-11 11:04:58
From worker node	Hostname	e2551
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	82800 (23 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-11 10:41:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_925_20231205T041547Z_gen_g4_detsim_hitreco__20240510T021313Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_1042_20231203T070402Z_gen_g4_detsim_hitreco__20240508T073415Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_189_20231202T150359Z_gen_g4_detsim_hitreco__20240507T223924Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_994_20231202T075057Z_gen_g4_detsim_hitreco__20240507T204651Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_597_20231203T170637Z_gen_g4_detsim_hitreco__20240509T195739Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_302_20231207T050044Z_gen_g4_detsim_hitreco__20240510T053020Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_439_20231203T080334Z_gen_g4_detsim_hitreco__20240509T205003Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_376_20231122T052513Z_gen_g4_detsim_hitreco__20240503T065824Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_330_20231202T124738Z_gen_g4_detsim_hitreco__20240507T220609Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_421_20231202T113536Z_gen_g4_detsim_hitreco__20240507T214811Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-11 11:04:58
Saved logs	justin-logs:44845.164-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.551561      0.760966       0.55489      0.0769059       100    
reco:ophitspe:OpHitFinderDeco                          0.919057       1.04848       10.9475      0.946304      0.995005        100    
reco:opflash:OpFlashFinder                            0.000143127   0.00142688     0.0060491    0.00116837    0.00104724       100    
reco:opslicer:OpSlicer                                 4.242e-05     0.239291      0.788138      0.201229      0.187057        100    
reco:rns:RandomNumberSaver                             2.172e-05    5.50449e-05   0.000418822   4.5971e-05    4.61772e-05      100    
reco:anglereconue:NuAngularReco                       0.000166711   0.00151573     0.0181981    0.000397232   0.00323537       100    
reco:anglereconumu:NuAngularReco                      8.1352e-05    0.00120899     0.016376     0.000218864   0.00292218       100    
reco:anglereconuepfps:NuAngularReco                   0.000121537   0.00131966     0.0166738    0.000294109   0.00298292       100    
reco:anglereconumupfps:NuAngularReco                  0.000107711   0.00129012     0.0170403    0.000261849   0.00302754       100    
reco:anglerecohits:NuAngularReco                      8.6982e-05     0.0019696     0.019156     0.000666634   0.00359569       100    
[art]:TriggerResults:TriggerResultInserter             9.949e-06    1.30868e-05   4.2049e-05    1.1882e-05    4.36263e-06      100    
end_path:cafmaker:CAFMaker                            0.00152567     0.0346539      1.4261      0.00842383     0.145307        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3694.19 MB
  Peak resident set size usage (VmHWM): 2018.71 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 06:01:47 CDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6e/13/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_1042_20231203T070402Z_gen_g4_detsim_hitreco__20240508T073415Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 2: 1315
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996052, 0.000837319, 0.00311107, 
Output 1: 0.309439, 0.499498, 0.118441, 0.0726224, 
Output 2: 0.810096, 0.164119, 0.0223904, 0.00339539, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2913!
Begin processing the 95th record. run: 50601160 subRun: 1 event: 92595 at 11-Sep-2025 06:01:08 CDT
Boundary wire vector sizes: 46, 46, 38
minwire 0: 1288
minwire 1: 1585
minwire 2: 1170
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00278684, 0.00170655, 0.995507, 
Output 1: 0.904127, 0.0951628, 0.000612202, 9.75185e-05, 
Output 2: 0.99905, 0.000923939, 2.29023e-05, 2.74755e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1218!
Begin processing the 96th record. run: 50601160 subRun: 1 event: 92596 at 11-Sep-2025 06:01:14 CDT
Boundary wire vector sizes: 506, 311, 375
minwire 0: 814
minwire 1: 1602
minwire 2: 814
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0164811, 0.0453823, 0.938137, 
Output 1: 0.499181, 0.426458, 0.0676336, 0.00672713, 
Output 2: 0.0224657, 0.759373, 0.202584, 0.0155773, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2172!
Begin processing the 97th record. run: 50601160 subRun: 1 event: 92597 at 11-Sep-2025 06:01:21 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 666!
Begin processing the 98th record. run: 50601160 subRun: 1 event: 92598 at 11-Sep-2025 06:01:26 CDT
Boundary wire vector sizes: 226, 207, 189
minwire 0: 2681
minwire 1: 1034
minwire 2: 2348
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996233, 0.000583653, 0.00318295, 
Output 1: 0.00548584, 0.977551, 0.0166538, 0.000309722, 
Output 2: 0.988021, 0.0117847, 0.000191326, 2.6963e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1731!
Begin processing the 99th record. run: 50601160 subRun: 1 event: 92599 at 11-Sep-2025 06:01:33 CDT
Boundary wire vector sizes: 56, 114, 86
minwire 0: 243
minwire 1: 2131
minwire 2: 88
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00114599, 0.986413, 0.0124405, 
Output 1: 0.986568, 0.0132628, 9.08865e-05, 7.83418e-05, 
Output 2: 0.999217, 0.000750397, 2.1292e-05, 1.10439e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 565!
Begin processing the 100th record. run: 50601160 subRun: 1 event: 92600 at 11-Sep-2025 06:01:38 CDT
Boundary wire vector sizes: 200, 173, 218
minwire 0: 1130
minwire 1: 1415
minwire 2: 1281
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.994972, 0.000475433, 0.0045527, 
Output 1: 0.93332, 0.0663779, 0.000285199, 1.64175e-05, 
Output 2: 0.997023, 0.00291863, 5.50695e-05, 3.43559e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1874!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/5e/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_925_20231205T041547Z_gen_g4_detsim_hitreco__20240510T021313Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6e/13/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_1042_20231203T070402Z_gen_g4_detsim_hitreco__20240508T073415Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/49/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_189_20231202T150359Z_gen_g4_detsim_hitreco__20240507T223924Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_994_20231202T075057Z_gen_g4_detsim_hitreco__20240507T204651Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e2/ba/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_597_20231203T170637Z_gen_g4_detsim_hitreco__20240509T195739Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/7a/dd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_302_20231207T050044Z_gen_g4_detsim_hitreco__20240510T053020Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/34/92/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_439_20231203T080334Z_gen_g4_detsim_hitreco__20240509T205003Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5c/3a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_376_20231122T052513Z_gen_g4_detsim_hitreco__20240503T065824Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/11/e0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_330_20231202T124738Z_gen_g4_detsim_hitreco__20240507T220609Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cd/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_421_20231202T113536Z_gen_g4_detsim_hitreco__20240507T214811Z_reco2.root
.:
total 4696
-rw-r--r-- 1 slot1_2 slot1_2 1863515 Sep 11 06:01 flatcaf.root
-rw-r--r-- 1 slot1_2 slot1_2 1388826 Sep 11 06:01 caf.root
-rw-r--r-- 1 slot1_2 slot1_2 1388826 Sep 11 06:01 caf_fd_hd_atmo_2501_20250911T104155Z.root
-rw-r--r-- 1 slot1_2 slot1_2   69967 Sep 11 06:01 caf_20250911T104155Z.log
-rw-r--r-- 1 slot1_2 slot1_2   43261 Sep 11 06:01 jobscript.log
-rw-r--r-- 1 slot1_2 slot1_2    1976 Sep 11 06:01 caf_20250911T104155Z.file
-rw-r--r-- 1 slot1_2 slot1_2    1976 Sep 11 06:01 caf_20250911T104155Z.pfns
-rw-r--r-- 1 slot1_2 slot1_2    1976 Sep 11 05:41 file.list
-rw-r--r-- 1 slot1_2 slot1_2    1976 Sep 11 06:01 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_2 slot1_2    1376 Sep 11 05:41 all-input-dids.txt
-rw-r--r-- 1 slot1_2 slot1_2    1376 Sep 11 06:01 caf_20250911T104155Z.did
-rw-r--r-- 1 slot1_2 slot1_2    1376 Sep 11 05:41 did.list
-rw-r--r-- 1 slot1_2 slot1_2       0 Sep 11 05:42 debugprod.log