Jobsub ID 155570.189@dunegpschedd01.fnal.gov

Jobsub ID	155570.189@dunegpschedd01.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 15:41:55
Site	UK_Manchester
Entry	UBoone_T2_UK_Manchester_ce01
Last heartbeat	2025-09-23 16:02:10
From worker node	Hostname	wn1207241.tier2.hep.manchester.ac.uk
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 15:43:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_167_20231201T051732Z_gen_g4_detsim_hitreco__20240507T191715Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_1160_20231205T065923Z_gen_g4_detsim_hitreco__20240509T222645Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_315_20231123T011538Z_gen_g4_detsim_hitreco__20240507T174255Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_397_20231203T081014Z_gen_g4_detsim_hitreco__20240508T033513Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1029_20231202T183231Z_gen_g4_detsim_hitreco__20240508T060347Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_2_20231204T054707Z_gen_g4_detsim_hitreco__20240509T214759Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_473_20231206T230644Z_gen_g4_detsim_hitreco__20240510T063419Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_72_20231203T123900Z_gen_g4_detsim_hitreco__20240508T080247Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6447068_440_20231205T120806Z_gen_g4_detsim_hitreco__20240510T060811Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_341_20231206T234309Z_gen_g4_detsim_hitreco__20240510T025714Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 16:02:10
Saved logs	justin-logs:155570.189-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

convolution                               0.0912283     0.601648       1.5515       0.563447      0.267327        100    
reco:ophitspe:OpHitFinderDeco                          0.119168      0.277307       1.08965       0.17387      0.230129        100    
reco:opflash:OpFlashFinder                            0.00021201    0.000902358   0.00316118    0.000860812   0.000428514      100    
reco:opslicer:OpSlicer                                0.00159051     0.179859      0.653015      0.153138      0.140517        100    
reco:rns:RandomNumberSaver                            1.4968e-05    2.46132e-05   0.000292041   2.2613e-05    2.70777e-05      100    
reco:anglereconue:NuAngularReco                       9.4007e-05     0.0007489    0.00541982    0.000255848   0.00114815       100    
reco:anglereconumu:NuAngularReco                      6.0113e-05    0.000636073   0.00550938    0.000162486   0.00114855       100    
reco:anglereconuepfps:NuAngularReco                   9.2604e-05    0.000724556    0.0054337    0.000238455   0.00117566       100    
reco:anglereconumupfps:NuAngularReco                  8.3417e-05    0.000685755   0.00562262    0.000204254   0.00117902       100    
reco:anglerecohits:NuAngularReco                      6.6215e-05    0.00106572    0.00655678    0.000504951   0.00138649       100    
[art]:TriggerResults:TriggerResultInserter             7.764e-06    9.25835e-06   3.6338e-05     8.837e-06    2.81974e-06      100    
end_path:cafmaker:CAFMaker                            0.00104693     0.0097875     0.0882797    0.00306791     0.0160215       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3673.45 MB
  Peak resident set size usage (VmHWM): 2001.04 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 17:01:53 BST ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/fa/01/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_1160_20231205T065923Z_gen_g4_detsim_hitreco__20240509T222645Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0862226, 0.00593969, 0.907838, 
Output 1: 0.00924064, 0.0296051, 0.269933, 0.691221, 
Output 2: 0.991316, 0.00821378, 0.000407688, 6.22124e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2155!
Begin processing the 94th record. run: 65974923 subRun: 1 event: 16794 at 23-Sep-2025 16:53:38 BST
Boundary wire vector sizes: 638, 944, 909
minwire 0: 1957
minwire 1: 216
minwire 2: 1987
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997045, 0.000706294, 0.00224839, 
Output 1: 0.00544059, 0.989985, 0.00446681, 0.000107522, 
Output 2: 0.997294, 0.00263756, 6.67671e-05, 1.50882e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1891!
Begin processing the 95th record. run: 65974923 subRun: 1 event: 16795 at 23-Sep-2025 16:53:45 BST
Boundary wire vector sizes: 2878, 1116, 2020
minwire 0: 63
minwire 1: 1326
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.97919, 0.00398323, 0.0168268, 
Output 1: 0.230975, 0.422085, 0.225189, 0.121751, 
Output 2: 0.0310566, 0.132698, 0.418843, 0.417403, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2847!
Begin processing the 96th record. run: 65974923 subRun: 1 event: 16796 at 23-Sep-2025 16:54:02 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1376!
Begin processing the 97th record. run: 65974923 subRun: 1 event: 16797 at 23-Sep-2025 16:54:05 BST
Boundary wire vector sizes: 423, 608, 552
minwire 0: 2356
minwire 1: 654
minwire 2: 1956
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991891, 0.00371456, 0.00439479, 
Output 1: 0.00653391, 0.981416, 0.0118252, 0.000225335, 
Output 2: 0.970255, 0.0292192, 0.000518064, 7.9975e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2203!
Begin processing the 98th record. run: 65974923 subRun: 1 event: 16798 at 23-Sep-2025 16:54:10 BST
Boundary wire vector sizes: 3062, 3162, 2057
minwire 0: 459
minwire 1: 553
minwire 2: 525
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.110492, 0.481743, 0.407765, 
Output 1: 0.381315, 0.533421, 0.0597726, 0.0254916, 
Output 2: 0.0133023, 0.0517948, 0.231891, 0.703012, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3251!
Begin processing the 99th record. run: 65974923 subRun: 1 event: 16799 at 23-Sep-2025 16:54:29 BST
Boundary wire vector sizes: 57, 69, 46
minwire 0: 1551
minwire 1: 998
minwire 2: 1430
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00308227, 0.00172269, 0.995195, 
Output 1: 0.0815193, 0.912293, 0.00593057, 0.000257407, 
Output 2: 0.997925, 0.00199336, 7.69899e-05, 4.2633e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1484!
Begin processing the 100th record. run: 65974923 subRun: 1 event: 16800 at 23-Sep-2025 16:54:34 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 154!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/17/57/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_167_20231201T051732Z_gen_g4_detsim_hitreco__20240507T191715Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/fa/01/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_1160_20231205T065923Z_gen_g4_detsim_hitreco__20240509T222645Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/dd/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209163_315_20231123T011538Z_gen_g4_detsim_hitreco__20240507T174255Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/01/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_397_20231203T081014Z_gen_g4_detsim_hitreco__20240508T033513Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cb/19/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1029_20231202T183231Z_gen_g4_detsim_hitreco__20240508T060347Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d7/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_2_20231204T054707Z_gen_g4_detsim_hitreco__20240509T214759Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4a/92/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_473_20231206T230644Z_gen_g4_detsim_hitreco__20240510T063419Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/85/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_72_20231203T123900Z_gen_g4_detsim_hitreco__20240508T080247Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/06/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6447068_440_20231205T120806Z_gen_g4_detsim_hitreco__20240510T060811Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e9/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_341_20231206T234309Z_gen_g4_detsim_hitreco__20240510T025714Z_reco2.root
.:
total 4140
-rw-r--r-- 1 dune01 dune01 1633657 Sep 23 17:01 flatcaf.root
-rw-r--r-- 1 dune01 dune01 1222464 Sep 23 17:01 caf.root
-rw-r--r-- 1 dune01 dune01 1222464 Sep 23 17:01 caf_fd_hd_atmo_3301_20250923T154342Z.root
-rw-r--r-- 1 dune01 dune01   74147 Sep 23 17:01 caf_20250923T154342Z.log
-rw-r--r-- 1 dune01 dune01   34825 Sep 23 17:01 jobscript.log
-rw-r--r-- 1 dune01 dune01    1900 Sep 23 17:01 caf_20250923T154342Z.file
-rw-r--r-- 1 dune01 dune01    1900 Sep 23 17:01 caf_20250923T154342Z.pfns
-rw-r--r-- 1 dune01 dune01    1900 Sep 23 16:43 file.list
-rw-r--r-- 1 dune01 dune01    1900 Sep 23 17:01 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01    1376 Sep 23 16:43 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01    1376 Sep 23 17:01 caf_20250923T154342Z.did
-rw-r--r-- 1 dune01 dune01    1376 Sep 23 16:43 did.list
-rw-r--r-- 1 dune01 dune01       0 Sep 23 16:43 debugprod.log