Jobsub ID 46606.0@dunegpschedd01.fnal.gov

Jobsub ID	46606.0@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:47:41
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2025-09-15 13:32:16
From worker node	Hostname	n134.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:48:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_464_20231203T041412Z_gen_g4_detsim_hitreco__20240508T071843Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_546_20231203T115311Z_gen_g4_detsim_hitreco__20240509T195151Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_601_20231202T125850Z_gen_g4_detsim_hitreco__20240507T221921Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1005_20231203T084741Z_gen_g4_detsim_hitreco__20240509T201917Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_815_20231202T072213Z_gen_g4_detsim_hitreco__20240508T035947Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_249_20231207T110854Z_gen_g4_detsim_hitreco__20240510T053228Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_101_20231203T150647Z_gen_g4_detsim_hitreco__20240508T080531Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_386_20231207T011959Z_gen_g4_detsim_hitreco__20240510T044329Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_830_20231203T162032Z_gen_g4_detsim_hitreco__20240508T035130Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_570_20231208T004742Z_gen_g4_detsim_hitreco__20240510T031225Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 13:32:16
Saved logs	justin-logs:46606.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000766118    0.0624057      2.57397     0.00333889      0.28263        100    
reco:opdec:Deconvolution                               0.152311       0.34489      0.643231      0.336751       0.10334        100    
reco:ophitspe:OpHitFinderDeco                          0.192415      0.262406      0.560644      0.241876      0.0587927       100    
reco:opflash:OpFlashFinder                            0.000234877   0.00159765     0.0202734    0.00133549    0.00198553       100    
reco:opslicer:OpSlicer                                0.000121618    0.234219      0.757011       0.1777       0.204738        100    
reco:rns:RandomNumberSaver                            2.2475e-05    3.60106e-05   0.000339554   2.89415e-05   3.1739e-05       100    
reco:anglereconue:NuAngularReco                       0.000178303   0.00261069     0.0608335    0.000434086   0.00834713       100    
reco:anglereconumu:NuAngularReco                      0.000107997   0.00247219     0.0608252    0.000263629   0.00839404       100    
reco:anglereconuepfps:NuAngularReco                   0.000149679   0.00273687     0.0608048    0.000363156   0.00854424       100    
reco:anglereconumupfps:NuAngularReco                  0.000141985   0.00256844     0.0607633    0.000351508   0.00846958       100    
reco:anglerecohits:NuAngularReco                       0.0001193    0.00382667     0.0670867    0.000755822    0.0108504       100    
[art]:TriggerResults:TriggerResultInserter            1.3162e-05    1.63085e-05   6.0549e-05    1.49965e-05   5.20529e-06      100    
end_path:cafmaker:CAFMaker                            0.00146005     0.0768228      4.50772     0.00543772     0.456007        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3669.06 MB
  Peak resident set size usage (VmHWM): 1996.02 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 14:10:23 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/ac/28/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_546_20231203T115311Z_gen_g4_detsim_hitreco__20240509T195151Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 141, 78, 104
minwire 0: 621
minwire 1: 2437
minwire 2: 310
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000607414, 0.997895, 0.00149792, 
Output 1: 0.907684, 0.0918653, 0.000380102, 7.017e-05, 
Output 2: 0.999178, 0.000803052, 1.66035e-05, 2.1989e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1234!
Begin processing the 95th record. run: 66006611 subRun: 1 event: 46495 at 15-Sep-2025 14:00:32 BST
Boundary wire vector sizes: 152, 288, 221
minwire 0: 2518
minwire 1: 623
minwire 2: 2433
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.998092, 0.000280862, 0.00162706, 
Output 1: 0.00452256, 0.208141, 0.767334, 0.0200018, 
Output 2: 0.988762, 0.0107015, 0.000529255, 7.24998e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1648!
Begin processing the 96th record. run: 66006611 subRun: 1 event: 46496 at 15-Sep-2025 14:00:37 BST
Boundary wire vector sizes: 10033, 8681, 5745
minwire 0: 682
minwire 1: 1038
minwire 2: 311
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0702619, 0.858221, 0.0715168, 
Output 1: 0.208128, 0.370925, 0.213567, 0.207379, 
Output 2: 0.0146598, 0.0501279, 0.219854, 0.715359, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3366!
Begin processing the 97th record. run: 66006611 subRun: 1 event: 46497 at 15-Sep-2025 14:02:43 BST
Boundary wire vector sizes: 1406, 1968, 1431
minwire 0: 293
minwire 1: 1801
minwire 2: 15
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.635621, 0.00598474, 0.358394, 
Output 1: 0.516535, 0.397342, 0.0544918, 0.0316314, 
Output 2: 0.0373494, 0.138562, 0.319468, 0.504621, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2048!
Begin processing the 98th record. run: 66006611 subRun: 1 event: 46498 at 15-Sep-2025 14:02:52 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 161!
Begin processing the 99th record. run: 66006611 subRun: 1 event: 46499 at 15-Sep-2025 14:02:56 BST
Boundary wire vector sizes: 362, 512, 151
minwire 0: 1462
minwire 1: 1689
minwire 2: 1173
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997098, 0.000501339, 0.00240094, 
Output 1: 0.00321933, 0.99223, 0.00445653, 9.36697e-05, 
Output 2: 0.996878, 0.00305934, 6.19167e-05, 1.07724e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2577!
Begin processing the 100th record. run: 66006611 subRun: 1 event: 46500 at 15-Sep-2025 14:03:01 BST
Boundary wire vector sizes: 1339, 936, 929
minwire 0: 540
minwire 1: 1177
minwire 2: 452
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0011712, 0.969458, 0.0293704, 
Output 1: 0.905049, 0.0894364, 0.00521571, 0.0002991, 
Output 2: 0.918976, 0.0791511, 0.00178548, 8.73582e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2275!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/af/0d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_464_20231203T041412Z_gen_g4_detsim_hitreco__20240508T071843Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/ac/28/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_546_20231203T115311Z_gen_g4_detsim_hitreco__20240509T195151Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/2d/1f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_601_20231202T125850Z_gen_g4_detsim_hitreco__20240507T221921Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/a2/e7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1005_20231203T084741Z_gen_g4_detsim_hitreco__20240509T201917Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/c2/4a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_815_20231202T072213Z_gen_g4_detsim_hitreco__20240508T035947Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/e3/c1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_249_20231207T110854Z_gen_g4_detsim_hitreco__20240510T053228Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/6c/88/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_101_20231203T150647Z_gen_g4_detsim_hitreco__20240508T080531Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/85/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_386_20231207T011959Z_gen_g4_detsim_hitreco__20240510T044329Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/bf/c2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_830_20231203T162032Z_gen_g4_detsim_hitreco__20240508T035130Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/eb/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_570_20231208T004742Z_gen_g4_detsim_hitreco__20240510T031225Z_reco2.root
.:
total 5668
-rw-r--r--. 1 dune001 dune 2208536 Sep 15 14:10 flatcaf.root
-rw-r--r--. 1 dune001 dune 1713267 Sep 15 14:10 caf.root
-rw-r--r--. 1 dune001 dune 1713267 Sep 15 14:10 caf_fd_hd_atmo_2501_20250915T124832Z.root
-rw-r--r--. 1 dune001 dune   73495 Sep 15 14:10 caf_20250915T124832Z.log
-rw-r--r--. 1 dune001 dune   37766 Sep 15 14:10 jobscript.log
-rw-r--r--. 1 dune001 dune    1907 Sep 15 14:10 caf_20250915T124832Z.file
-rw-r--r--. 1 dune001 dune    1907 Sep 15 14:10 caf_20250915T124832Z.pfns
-rw-r--r--. 1 dune001 dune    1907 Sep 15 13:48 file.list
-rw-r--r--. 1 dune001 dune    1907 Sep 15 14:10 justin-processed-pfns.txt
-rw-r--r--. 1 dune001 dune    1377 Sep 15 13:48 all-input-dids.txt
-rw-r--r--. 1 dune001 dune    1377 Sep 15 14:10 caf_20250915T124832Z.did
-rw-r--r--. 1 dune001 dune    1377 Sep 15 13:48 did.list
-rw-r--r--. 1 dune001 dune       0 Sep 15 13:48 debugprod.log