Jobsub ID 41104.79@dunegpschedd02.fnal.gov

Jobsub ID	41104.79@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 17:17:32
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce01
Last heartbeat	2025-09-17 17:31:50
From worker node	Hostname	wn2209251.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7513 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-17 17:18:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_778_20231207T103826Z_gen_g4_detsim_hitreco__20240510T052136Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_790_20231208T101452Z_gen_g4_detsim_hitreco__20240510T064958Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_61_20231203T171153Z_gen_g4_detsim_hitreco__20240509T215111Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_839_20231202T062803Z_gen_g4_detsim_hitreco__20240507T194718Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_735_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203414Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_874_20231204T004031Z_gen_g4_detsim_hitreco__20240509T220312Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-17 17:31:50
Saved logs	justin-logs:41104.79-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

:energyrecnc:EnergyReco                           0.00271228     0.133285      0.415933      0.175368      0.0981464       100    
reco:energyrecnumurange:EnergyReco                    0.00037745     0.027157      0.572098     0.00359482     0.0721931       100    
reco:energyrecnumumcs:EnergyReco                      0.000809293    0.0279438     0.582535     0.00438982      0.07234        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000543432    0.0277679     0.663525     0.00451303     0.0768834       100    
reco:opdec:Deconvolution                               0.132133      0.307026      0.599236      0.295211      0.0953451       100    
reco:ophitspe:OpHitFinderDeco                          0.243236      0.254139      0.283138      0.253673     0.00663192       100    
reco:opflash:OpFlashFinder                            0.000582737    0.0021662    0.00615534     0.0020383    0.00111778       100    
reco:opslicer:OpSlicer                                0.00667513     0.367086       1.4719       0.300119      0.300379        100    
reco:rns:RandomNumberSaver                            3.5547e-05    7.86829e-05   0.000696942   6.8033e-05    6.7232e-05       100    
reco:anglereconue:NuAngularReco                       0.000250882   0.00176849     0.0451353    0.000685099   0.00486429       100    
reco:anglereconumu:NuAngularReco                      8.7735e-05    0.00139181     0.0387723    0.000394788   0.00430488       100    
reco:anglereconuepfps:NuAngularReco                   0.000138641   0.00157017     0.0378386    0.000549544    0.004207        100    
reco:anglereconumupfps:NuAngularReco                  0.000139332    0.0014044     0.0351009    0.000451825   0.00397684       100    
reco:anglerecohits:NuAngularReco                      0.000484162   0.00246092     0.0427474    0.00113195    0.00496143       100    
[art]:TriggerResults:TriggerResultInserter            1.1291e-05    2.2174e-05    8.8777e-05     1.863e-05    1.06864e-05      100    
end_path:cafmaker:CAFMaker                            0.00218214     0.0240494     0.451076     0.00814676     0.0560391       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3812.91 MB
  Peak resident set size usage (VmHWM): 2116.92 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 18:31:24 BST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Begin processing the 95th record. run: 74550731 subRun: 1 event: 77895 at 17-Sep-2025 18:30:47 BST
Boundary wire vector sizes: 41, 42, 37
minwire 0: 526
minwire 1: 2260
minwire 2: 135
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 163, 462
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 38, 337
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00663823, 0.00329879, 0.990063, 
Output 1: 0.887483, 0.111583, 0.000823025, 0.000111395, 
Output 2: 0.998585, 0.00137933, 3.23296e-05, 3.37122e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 773!
Begin processing the 96th record. run: 74550731 subRun: 1 event: 77896 at 17-Sep-2025 18:30:50 BST
Boundary wire vector sizes: 875, 792, 652
minwire 0: 1577
minwire 1: 979
minwire 2: 1480
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00243902, 0.00484228, 0.992719, 
Output 1: 0.978624, 0.0209859, 0.000270107, 0.000120223, 
Output 2: 0.996948, 0.00297886, 6.03336e-05, 1.26251e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3023!
Begin processing the 97th record. run: 74550731 subRun: 1 event: 77897 at 17-Sep-2025 18:30:56 BST
Boundary wire vector sizes: 227, 229, 189
minwire 0: 2313
minwire 1: 1175
minwire 2: 1985
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00363522, 0.00448664, 0.991878, 
Output 1: 0.0116323, 0.962924, 0.0244432, 0.00100098, 
Output 2: 0.995889, 0.00400702, 0.000101392, 2.92406e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2112!
Begin processing the 98th record. run: 74550731 subRun: 1 event: 77898 at 17-Sep-2025 18:31:00 BST
Boundary wire vector sizes: 123, 192, 155
minwire 0: 265
minwire 1: 2230
minwire 2: 49
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000308099, 0.998288, 0.00140431, 
Output 1: 0.00543349, 0.989613, 0.0047929, 0.00016101, 
Output 2: 0.997216, 0.00271622, 6.59864e-05, 1.87841e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1347!
Begin processing the 99th record. run: 74550731 subRun: 1 event: 77899 at 17-Sep-2025 18:31:04 BST
Boundary wire vector sizes: 3504, 4085, 2074
minwire 0: 1199
minwire 1: 583
minwire 2: 1119
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0659556, 0.820298, 0.113746, 
Output 1: 0.401889, 0.445873, 0.103239, 0.0489987, 
Output 2: 0.0481116, 0.18576, 0.342419, 0.42371, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3277!
Begin processing the 100th record. run: 74550731 subRun: 1 event: 77900 at 17-Sep-2025 18:31:18 BST
Boundary wire vector sizes: 744, 1054, 709
minwire 0: 1749
minwire 1: 430
minwire 2: 1899
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.549672, 0.0797172, 0.370611, 
Output 1: 0.333436, 0.563866, 0.0756891, 0.0270083, 
Output 2: 0.0262957, 0.0873645, 0.270108, 0.616232, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1720!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_778_20231207T103826Z_gen_g4_detsim_hitreco__20240510T052136Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/06/71/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_790_20231208T101452Z_gen_g4_detsim_hitreco__20240510T064958Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/0f/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_61_20231203T171153Z_gen_g4_detsim_hitreco__20240509T215111Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/e7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_839_20231202T062803Z_gen_g4_detsim_hitreco__20240507T194718Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/0e/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_735_20231204T140409Z_gen_g4_detsim_hitreco__20240509T203414Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/bd/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_874_20231204T004031Z_gen_g4_detsim_hitreco__20240509T220312Z_reco2.root
.:
total 5340
-rw-r--r-- 1 dune01 dune01 2081096 Sep 17 18:31 flatcaf.root
-rw-r--r-- 1 dune01 dune01 1610165 Sep 17 18:31 caf.root
-rw-r--r-- 1 dune01 dune01 1610165 Sep 17 18:31 caf_fd_hd_atmo_2501_20250917T171843Z.root
-rw-r--r-- 1 dune01 dune01   76479 Sep 17 18:31 caf_20250917T171843Z.log
-rw-r--r-- 1 dune01 dune01   34232 Sep 17 18:31 jobscript.log
-rw-r--r-- 1 dune01 dune01    1397 Sep 17 18:31 caf_20250917T171843Z.file
-rw-r--r-- 1 dune01 dune01    1397 Sep 17 18:31 caf_20250917T171843Z.pfns
-rw-r--r-- 1 dune01 dune01    1397 Sep 17 18:18 file.list
-rw-r--r-- 1 dune01 dune01    1397 Sep 17 18:31 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01     962 Sep 17 18:18 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     962 Sep 17 18:31 caf_20250917T171843Z.did
-rw-r--r-- 1 dune01 dune01     962 Sep 17 18:18 did.list
-rw-r--r-- 1 dune01 dune01       0 Sep 17 18:19 debugprod.log