Jobsub ID 46603.92@dunegpschedd01.fnal.gov

Jobsub ID	46603.92@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:31:41
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2025-09-15 14:21:52
From worker node	Hostname	comp22-08
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:33:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_71_20231203T025006Z_gen_g4_detsim_hitreco__20240508T074757Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_1112_20231205T063616Z_gen_g4_detsim_hitreco__20240509T222135Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_446_20231201T110447Z_gen_g4_detsim_hitreco__20240507T202449Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_724_20231207T001621Z_gen_g4_detsim_hitreco__20240510T063525Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_252_20231203T075050Z_gen_g4_detsim_hitreco__20240509T204516Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_752_20231207T140625Z_gen_g4_detsim_hitreco__20240510T045136Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_364_20231120T135405Z_gen_g4_detsim_hitreco__20240503T070951Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_555_20231207T062930Z_gen_g4_detsim_hitreco__20240510T054638Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_75_20231204T080556Z_gen_g4_detsim_hitreco__20240509T234010Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_278_20231201T170020Z_gen_g4_detsim_hitreco__20240507T195128Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 14:21:52
Saved logs	justin-logs:46603.92-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

recnumumcsllhd:EnergyReco                  0.000452457    0.0510952      17.0122     0.00263408     0.557763       1000    
reco:opdec:Deconvolution                               0.0509257     0.254112       2.13172      0.254337      0.0999376      1000    
reco:ophitspe:OpHitFinderDeco                          0.144637      0.214682      0.867411      0.233035      0.0607266      1000    
reco:opflash:OpFlashFinder                            0.000282005   0.00171626    0.00596697    0.00167151    0.000545292     1000    
reco:opslicer:OpSlicer                                9.0544e-05     0.202119       1.02989      0.158064      0.168975       1000    
reco:rns:RandomNumberSaver                            2.0099e-05    2.6876e-05    0.000387442   2.53385e-05   1.22615e-05     1000    
reco:anglereconue:NuAngularReco                       0.000144923   0.00166875     0.107869     0.000319938   0.00674574      1000    
reco:anglereconumu:NuAngularReco                      6.3532e-05    0.00149167     0.100993     0.000177862   0.00651806      1000    
reco:anglereconuepfps:NuAngularReco                   0.000100815   0.00157143     0.0994615    0.00025945    0.00656777      1000    
reco:anglereconumupfps:NuAngularReco                  8.5556e-05    0.00155772     0.0995793    0.00023199    0.00656509      1000    
reco:anglerecohits:NuAngularReco                      8.0519e-05    0.00202996     0.106297     0.000546536   0.00711131      1000    
[art]:TriggerResults:TriggerResultInserter             8.473e-06    1.14416e-05    6.74e-05     1.06065e-05   3.22447e-06     1000    
end_path:cafmaker:CAFMaker                            0.00124275     0.0246416      5.18589      0.0035785     0.183674       1000    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4115.36 MB
  Peak resident set size usage (VmHWM): 2440.97 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 14:57:54 BST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/18/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_278_20231201T170020Z_gen_g4_detsim_hitreco__20240507T195128Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2214!
Begin processing the 992nd record. run: 6422672 subRun: 1 event: 7592 at 15-Sep-2025 14:57:28 BST
Boundary wire vector sizes: 123, 152, 122
minwire 0: 1466
minwire 1: 934
minwire 2: 1211
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.841431, 0.03388, 0.124689, 
Output 1: 0.207922, 0.764645, 0.0264079, 0.0010248, 
Output 2: 0.368727, 0.617629, 0.013513, 0.000131794, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2410!
Begin processing the 993rd record. run: 6422672 subRun: 1 event: 7593 at 15-Sep-2025 14:57:31 BST
Boundary wire vector sizes: 91, 77, 82
minwire 0: 2213
minwire 1: 476
minwire 2: 2407
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00182582, 0.873833, 0.124341, 
Output 1: 0.265813, 0.728489, 0.00558134, 0.000116066, 
Output 2: 0.998026, 0.00189615, 7.23975e-05, 5.22734e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 923!
Begin processing the 994th record. run: 6422672 subRun: 1 event: 7594 at 15-Sep-2025 14:57:33 BST
Boundary wire vector sizes: 1306, 807, 1118
minwire 0: 174
minwire 1: 1374
minwire 2: 197
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996772, 0.000638813, 0.00258969, 
Output 1: 0.445921, 0.441114, 0.0811358, 0.0318295, 
Output 2: 0.850497, 0.132092, 0.0154035, 0.00200758, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2315!
Begin processing the 995th record. run: 6422672 subRun: 1 event: 7595 at 15-Sep-2025 14:57:37 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 610!
Begin processing the 996th record. run: 6422672 subRun: 1 event: 7596 at 15-Sep-2025 14:57:39 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 215!
Begin processing the 997th record. run: 6422672 subRun: 1 event: 7597 at 15-Sep-2025 14:57:41 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 19!
Begin processing the 998th record. run: 6422672 subRun: 1 event: 7598 at 15-Sep-2025 14:57:43 BST
Boundary wire vector sizes: 2011, 2679, 2305
minwire 0: 2070
minwire 1: 393
minwire 2: 2097
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.229551, 0.49881, 0.271639, 
Output 1: 0.247787, 0.547237, 0.150574, 0.0544019, 
Output 2: 0.0295589, 0.13184, 0.340683, 0.497919, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2578!
Begin processing the 999th record. run: 6422672 subRun: 1 event: 7599 at 15-Sep-2025 14:57:49 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 482!
Begin processing the 1000th record. run: 6422672 subRun: 1 event: 7600 at 15-Sep-2025 14:57:51 BST
Boundary wire vector sizes: 173, 169, 116
minwire 0: 690
minwire 1: 3016
minwire 2: 26
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00299962, 0.981869, 0.0151313, 
Output 1: 0.00874945, 0.967875, 0.0227199, 0.000655332, 
Output 2: 0.997305, 0.00260368, 8.81775e-05, 3.03988e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1223!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/51/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_71_20231203T025006Z_gen_g4_detsim_hitreco__20240508T074757Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/71/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_1112_20231205T063616Z_gen_g4_detsim_hitreco__20240509T222135Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/75/16/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_446_20231201T110447Z_gen_g4_detsim_hitreco__20240507T202449Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/55/bf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_724_20231207T001621Z_gen_g4_detsim_hitreco__20240510T063525Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/4e/9a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_252_20231203T075050Z_gen_g4_detsim_hitreco__20240509T204516Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4d/17/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_752_20231207T140625Z_gen_g4_detsim_hitreco__20240510T045136Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/21/6b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_364_20231120T135405Z_gen_g4_detsim_hitreco__20240503T070951Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/b2/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_555_20231207T062930Z_gen_g4_detsim_hitreco__20240510T054638Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ea/33/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_75_20231204T080556Z_gen_g4_detsim_hitreco__20240509T234010Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/18/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_278_20231201T170020Z_gen_g4_detsim_hitreco__20240507T195128Z_reco2.root
.:
total 43320
-rw-r--r-- 1 pltdune002 pltdune 16299107 Sep 15 14:57 flatcaf.root
-rw-r--r-- 1 pltdune002 pltdune 13581791 Sep 15 14:57 caf.root
-rw-r--r-- 1 pltdune002 pltdune 13581791 Sep 15 14:57 caf_fd_hd_atmo_2501_20250915T123344Z.root
-rw-r--r-- 1 pltdune002 pltdune   595921 Sep 15 14:57 caf_20250915T123344Z.log
-rw-r--r-- 1 pltdune002 pltdune   216935 Sep 15 14:57 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune     2006 Sep 15 14:57 caf_20250915T123344Z.file
-rw-r--r-- 1 pltdune002 pltdune     2006 Sep 15 14:57 caf_20250915T123344Z.pfns
-rw-r--r-- 1 pltdune002 pltdune     2006 Sep 15 13:33 file.list
-rw-r--r-- 1 pltdune002 pltdune     2006 Sep 15 14:57 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune     1376 Sep 15 13:33 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune     1376 Sep 15 14:57 caf_20250915T123344Z.did
-rw-r--r-- 1 pltdune002 pltdune     1376 Sep 15 13:33 did.list
-rw-r--r-- 1 pltdune002 pltdune        0 Sep 15 13:33 debugprod.log