Jobsub ID 155570.187@dunegpschedd01.fnal.gov

Jobsub ID	155570.187@dunegpschedd01.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 15:41:55
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce02
Last heartbeat	2025-09-23 16:11:38
From worker node	Hostname	wn1205251.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 15:43:58
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_393_20231201T184301Z_gen_g4_detsim_hitreco__20240507T212740Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_646_20231204T120435Z_gen_g4_detsim_hitreco__20240509T202157Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_723_20231203T043942Z_gen_g4_detsim_hitreco__20240508T064347Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_402_20231203T041128Z_gen_g4_detsim_hitreco__20240508T045616Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_495_20231203T080502Z_gen_g4_detsim_hitreco__20240509T205725Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_1095_20231204T004550Z_gen_g4_detsim_hitreco__20240509T202119Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_783_20231207T151150Z_gen_g4_detsim_hitreco__20240510T064017Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_347_20231203T165339Z_gen_g4_detsim_hitreco__20240509T195200Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_209_20231207T110331Z_gen_g4_detsim_hitreco__20240510T052636Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_584_20231207T073245Z_gen_g4_detsim_hitreco__20240510T023328Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 16:11:38
Saved logs	justin-logs:155570.187-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0838292     0.222679      0.752308      0.221005      0.0702246       300    
reco:ophitspe:OpHitFinderDeco                          0.235923      0.240557      0.275997      0.240185      0.0028486       300    
reco:opflash:OpFlashFinder                            0.000182565   0.000849731   0.00251274    0.000770768   0.000417452      300    
reco:opslicer:OpSlicer                                2.9025e-05     0.168776      0.808548      0.118996      0.151101        300    
reco:rns:RandomNumberSaver                            1.4557e-05    2.02161e-05   0.000207139   1.8956e-05    1.13518e-05      300    
reco:anglereconue:NuAngularReco                       0.000107803   0.00126175     0.0361922    0.000216027   0.00390901       300    
reco:anglereconumu:NuAngularReco                      5.7208e-05    0.00115985     0.0357732    0.000127891   0.00387888       300    
reco:anglereconuepfps:NuAngularReco                   8.8827e-05    0.00124717     0.0376444    0.000202908   0.00398273       300    
reco:anglereconumupfps:NuAngularReco                  8.1112e-05    0.00121604     0.0363951    0.000173496   0.00393475       300    
reco:anglerecohits:NuAngularReco                       6.886e-05    0.00163002     0.0406869    0.000433317   0.00433842       300    
[art]:TriggerResults:TriggerResultInserter             7.254e-06    8.37736e-06   2.8293e-05    8.2055e-06    1.5788e-06       300    
end_path:cafmaker:CAFMaker                            0.000985399    0.0158882     0.454965     0.00263467     0.0482496       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3803.08 MB
  Peak resident set size usage (VmHWM): 2123.51 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 17:11:21 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/71/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_402_20231203T041128Z_gen_g4_detsim_hitreco__20240508T045616Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 0: 1433
minwire 1: 1796
minwire 2: 1115
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00369948, 0.925096, 0.0712041, 
Output 1: 0.0323181, 0.963449, 0.00412299, 0.000110418, 
Output 2: 0.996248, 0.00360381, 0.00014092, 7.06541e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1908!
Begin processing the 295th record. run: 6413332 subRun: 1 event: 72395 at 23-Sep-2025 17:03:48 BST
Boundary wire vector sizes: 196, 198, 186
minwire 0: 764
minwire 1: 2101
minwire 2: 535
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000312482, 0.9976, 0.00208753, 
Output 1: 0.00651535, 0.987253, 0.00605923, 0.000172119, 
Output 2: 0.9983, 0.00165667, 4.21939e-05, 1.23965e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1413!
Begin processing the 296th record. run: 6413332 subRun: 1 event: 72396 at 23-Sep-2025 17:03:51 BST
Boundary wire vector sizes: 75, 58, 30
minwire 0: 123
minwire 1: 2220
minwire 2: 69
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0062141, 0.00231832, 0.991468, 
Output 1: 0.0598707, 0.861418, 0.0768199, 0.00189147, 
Output 2: 0.996127, 0.00370066, 0.000167509, 5.00126e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 891!
Begin processing the 297th record. run: 6413332 subRun: 1 event: 72397 at 23-Sep-2025 17:03:53 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1509!
Begin processing the 298th record. run: 6413332 subRun: 1 event: 72398 at 23-Sep-2025 17:03:55 BST
Boundary wire vector sizes: 1705, 1957, 1144
minwire 0: 308
minwire 1: 2060
minwire 2: 8
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0377413, 0.32985, 0.632409, 
Output 1: 0.377703, 0.568368, 0.0448945, 0.00903444, 
Output 2: 0.0554106, 0.498039, 0.334363, 0.112187, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2511!
Begin processing the 299th record. run: 6413332 subRun: 1 event: 72399 at 23-Sep-2025 17:03:59 BST
Boundary wire vector sizes: 676, 826, 688
minwire 0: 785
minwire 1: 1468
minwire 2: 458
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.041129, 0.0285969, 0.930274, 
Output 1: 0.325595, 0.471344, 0.15948, 0.0435811, 
Output 2: 0.0912125, 0.499212, 0.334936, 0.0746398, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2670!
Begin processing the 300th record. run: 6413332 subRun: 1 event: 72400 at 23-Sep-2025 17:04:03 BST
Boundary wire vector sizes: 109, 184, 132
minwire 0: 11
minwire 1: 2294
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0538414, 0.00806523, 0.938093, 
Output 1: 0.00344183, 0.0185048, 0.0523461, 0.925707, 
Output 2: 0.557027, 0.390562, 0.0449868, 0.00742415, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1034!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e1/25/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6389847_393_20231201T184301Z_gen_g4_detsim_hitreco__20240507T212740Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e9/96/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_646_20231204T120435Z_gen_g4_detsim_hitreco__20240509T202157Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c7/c8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_723_20231203T043942Z_gen_g4_detsim_hitreco__20240508T064347Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/71/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_402_20231203T041128Z_gen_g4_detsim_hitreco__20240508T045616Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_495_20231203T080502Z_gen_g4_detsim_hitreco__20240509T205725Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/96/6a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_1095_20231204T004550Z_gen_g4_detsim_hitreco__20240509T202119Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/b7/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_783_20231207T151150Z_gen_g4_detsim_hitreco__20240510T064017Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/35/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_347_20231203T165339Z_gen_g4_detsim_hitreco__20240509T195200Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/94/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_209_20231207T110331Z_gen_g4_detsim_hitreco__20240510T052636Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c7/10/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_584_20231207T073245Z_gen_g4_detsim_hitreco__20240510T023328Z_reco2.root
.:
total 14988
-rw-r--r-- 1 dune01 dune01 5694459 Sep 23 17:11 flatcaf.root
-rw-r--r-- 1 dune01 dune01 4670116 Sep 23 17:11 caf.root
-rw-r--r-- 1 dune01 dune01 4670116 Sep 23 17:11 caf_fd_hd_atmo_3301_20250923T154406Z.root
-rw-r--r-- 1 dune01 dune01  176978 Sep 23 17:11 caf_20250923T154406Z.log
-rw-r--r-- 1 dune01 dune01   75008 Sep 23 17:11 jobscript.log
-rw-r--r-- 1 dune01 dune01    1992 Sep 23 17:11 caf_20250923T154406Z.file
-rw-r--r-- 1 dune01 dune01    1992 Sep 23 17:11 caf_20250923T154406Z.pfns
-rw-r--r-- 1 dune01 dune01    1992 Sep 23 16:44 file.list
-rw-r--r-- 1 dune01 dune01    1992 Sep 23 17:11 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01    1376 Sep 23 16:44 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01    1376 Sep 23 17:11 caf_20250923T154406Z.did
-rw-r--r-- 1 dune01 dune01    1376 Sep 23 16:44 did.list
-rw-r--r-- 1 dune01 dune01       0 Sep 23 16:44 debugprod.log