Jobsub ID 46952.14@dunegpschedd01.fnal.gov

Jobsub ID	46952.14@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-16 07:34:40
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce02
Last heartbeat	2025-09-16 08:11:32
From worker node	Hostname	dune001-8142648.0-lcg2352.gridpp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-16 07:35:52
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_372_20231201T221058Z_gen_g4_detsim_hitreco__20240507T203647Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_681_20231207T133133Z_gen_g4_detsim_hitreco__20240510T060301Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_104_20231207T151621Z_gen_g4_detsim_hitreco__20240510T042351Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_218_20231201T143737Z_gen_g4_detsim_hitreco__20240507T203354Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_386_20231203T040719Z_gen_g4_detsim_hitreco__20240508T045533Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1094_20231203T085359Z_gen_g4_detsim_hitreco__20240509T202956Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_119_20231208T091627Z_gen_g4_detsim_hitreco__20240510T063237Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_821_20231201T213215Z_gen_g4_detsim_hitreco__20240508T050004Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_767_20231202T204206Z_gen_g4_detsim_hitreco__20240508T070101Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_196_20231205T031317Z_gen_g4_detsim_hitreco__20240509T222856Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-16 08:11:32
Saved logs	justin-logs:46952.14-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00050061     0.0588895      2.19017     0.00437521      0.22011        200    
reco:opdec:Deconvolution                               0.0145819     0.294359      0.870143      0.284812      0.127141        200    
reco:ophitspe:OpHitFinderDeco                         0.00379256     0.055843      0.359532      0.0287249     0.0527231       200    
reco:opflash:OpFlashFinder                            0.000164445   0.00173586    0.00657816    0.00160703    0.000869275      200    
reco:opslicer:OpSlicer                                4.9718e-05     0.339392       1.37081      0.273179      0.280877        200    
reco:rns:RandomNumberSaver                            1.9538e-05    4.16205e-05   0.000583523   3.2357e-05    5.14705e-05      200    
reco:anglereconue:NuAngularReco                       0.000150561   0.00265557     0.0497273    0.000464678   0.00722294       200    
reco:anglereconumu:NuAngularReco                      7.7623e-05    0.00249041     0.0493962    0.000297268   0.00722699       200    
reco:anglereconuepfps:NuAngularReco                   0.000116538   0.00261013     0.0496971    0.000390529   0.00723248       200    
reco:anglereconumupfps:NuAngularReco                  0.00010705    0.00261315     0.0502549    0.000372955   0.00735836       200    
reco:anglerecohits:NuAngularReco                      0.000121792   0.00342915     0.056133     0.000875352   0.00819127       200    
[art]:TriggerResults:TriggerResultInserter            1.0119e-05    1.53935e-05   6.5335e-05    1.4713e-05    4.51844e-06      200    
end_path:cafmaker:CAFMaker                            0.00131922     0.0372949     0.656499     0.00825738     0.0962953       200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3720.35 MB
  Peak resident set size usage (VmHWM): 1992.38 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 16-Sep-2025 08:00:18 UTC ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/13/f9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_104_20231207T151621Z_gen_g4_detsim_hitreco__20240510T042351Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 242, 275, 187
minwire 0: 1575
minwire 1: 780
minwire 2: 1179
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00135432, 0.992022, 0.00662424, 
Output 1: 0.587063, 0.381946, 0.0289057, 0.0020851, 
Output 2: 0.376554, 0.610284, 0.0129297, 0.000231701, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2123!
Begin processing the 195th record. run: 6456061 subRun: 1 event: 68195 at 16-Sep-2025 07:56:12 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 406!
Begin processing the 196th record. run: 6456061 subRun: 1 event: 68196 at 16-Sep-2025 07:56:13 UTC
Boundary wire vector sizes: 65, 102, 79
minwire 0: 2113
minwire 1: 964
minwire 2: 2061
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000287619, 0.998591, 0.00112118, 
Output 1: 0.0458811, 0.949818, 0.00414847, 0.00015263, 
Output 2: 0.998644, 0.00132508, 3.03393e-05, 8.6772e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1188!
Begin processing the 197th record. run: 6456061 subRun: 1 event: 68197 at 16-Sep-2025 07:56:16 UTC
Boundary wire vector sizes: 1824, 1484, 872
minwire 0: 681
minwire 1: 808
minwire 2: 646
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00427662, 0.97013, 0.0255932, 
Output 1: 0.0797506, 0.773959, 0.139806, 0.00648467, 
Output 2: 0.41769, 0.532092, 0.0488549, 0.00136322, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2589!
Begin processing the 198th record. run: 6456061 subRun: 1 event: 68198 at 16-Sep-2025 07:56:22 UTC
Boundary wire vector sizes: 3542, 4269, 3117
minwire 0: 415
minwire 1: 1857
minwire 2: 25
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00691151, 0.936913, 0.0561754, 
Output 1: 0.0542789, 0.142583, 0.226099, 0.577039, 
Output 2: 0.036469, 0.0951785, 0.271484, 0.596868, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3267!
Begin processing the 199th record. run: 6456061 subRun: 1 event: 68199 at 16-Sep-2025 07:56:36 UTC
Boundary wire vector sizes: 999, 962, 937
minwire 0: 1207
minwire 1: 1522
minwire 2: 1126
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0935164, 0.149153, 0.757331, 
Output 1: 0.00922275, 0.0618478, 0.159304, 0.769626, 
Output 2: 0.228626, 0.545204, 0.186486, 0.0396846, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3029!
Begin processing the 200th record. run: 6456061 subRun: 1 event: 68200 at 16-Sep-2025 07:56:44 UTC
Boundary wire vector sizes: 41, 148, 75
minwire 0: 1999
minwire 1: 913
minwire 2: 2032
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996919, 0.000881146, 0.00220014, 
Output 1: 0.00271602, 0.99368, 0.00351946, 8.41016e-05, 
Output 2: 0.997006, 0.00292428, 6.8497e-05, 1.49647e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1504!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/ad/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_372_20231201T221058Z_gen_g4_detsim_hitreco__20240507T203647Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/35/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_681_20231207T133133Z_gen_g4_detsim_hitreco__20240510T060301Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/13/f9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_104_20231207T151621Z_gen_g4_detsim_hitreco__20240510T042351Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/ba/ff/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_218_20231201T143737Z_gen_g4_detsim_hitreco__20240507T203354Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/36/44/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_386_20231203T040719Z_gen_g4_detsim_hitreco__20240508T045533Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0d/25/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1094_20231203T085359Z_gen_g4_detsim_hitreco__20240509T202956Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/30/8f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_119_20231208T091627Z_gen_g4_detsim_hitreco__20240510T063237Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/d3/61/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_821_20231201T213215Z_gen_g4_detsim_hitreco__20240508T050004Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c6/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_767_20231202T204206Z_gen_g4_detsim_hitreco__20240508T070101Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d2/98/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_196_20231205T031317Z_gen_g4_detsim_hitreco__20240509T222856Z_reco2.root
.:
total 11144
-rw-r--r-- 1 dune001 dune 4254241 Sep 16 08:00 flatcaf.root
-rw-r--r-- 1 dune001 dune 3461606 Sep 16 08:00 caf.root
-rw-r--r-- 1 dune001 dune 3461606 Sep 16 08:00 caf_fd_hd_atmo_2501_20250916T073601Z.root
-rw-r--r-- 1 dune001 dune  134749 Sep 16 08:00 caf_20250916T073601Z.log
-rw-r--r-- 1 dune001 dune   60902 Sep 16 08:00 jobscript.log
-rw-r--r-- 1 dune001 dune    1930 Sep 16 08:00 caf_20250916T073601Z.file
-rw-r--r-- 1 dune001 dune    1930 Sep 16 08:00 caf_20250916T073601Z.pfns
-rw-r--r-- 1 dune001 dune    1930 Sep 16 07:36 file.list
-rw-r--r-- 1 dune001 dune    1930 Sep 16 08:00 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune    1376 Sep 16 07:36 all-input-dids.txt
-rw-r--r-- 1 dune001 dune    1376 Sep 16 08:00 caf_20250916T073601Z.did
-rw-r--r-- 1 dune001 dune    1376 Sep 16 07:36 did.list
-rw-r--r-- 1 dune001 dune       0 Sep 16 07:36 debugprod.log