Jobsub ID 44845.9@dunegpschedd01.fnal.gov

Jobsub ID	44845.9@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:04:34
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw6
Last heartbeat	2025-09-11 11:08:28
From worker node	Hostname	mh-7763-5.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-11 10:13:51
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_114_20231202T093453Z_gen_g4_detsim_hitreco__20240507T200056Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_8_20231204T032213Z_gen_g4_detsim_hitreco__20240509T211540Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_640_20231202T151037Z_gen_g4_detsim_hitreco__20240508T024154Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_680_20231206T235340Z_gen_g4_detsim_hitreco__20240510T042933Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_892_20231202T161015Z_gen_g4_detsim_hitreco__20240508T043821Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_414_20231202T101230Z_gen_g4_detsim_hitreco__20240508T070720Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_187_20231208T053124Z_gen_g4_detsim_hitreco__20240510T044203Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_694_20231202T090147Z_gen_g4_detsim_hitreco__20240507T210632Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_418_20231208T095830Z_gen_g4_detsim_hitreco__20240510T064939Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_364_20231204T034027Z_gen_g4_detsim_hitreco__20240509T194855Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-11 11:08:28
Saved logs	justin-logs:44845.9-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

reco:opdec:Deconvolution                               0.428393      0.819705       1.51747       0.77386       0.21675        200    
reco:ophitspe:OpHitFinderDeco                           1.23749       1.29392       1.67547       1.32191      0.0634693       200    
reco:opflash:OpFlashFinder                            0.00030399    0.00167999    0.00553216    0.00156066    0.000802453      200    
reco:opslicer:OpSlicer                                0.000332099    0.346871       1.22942      0.296281      0.269627        200    
reco:rns:RandomNumberSaver                             1.771e-05    6.00547e-05   0.000591789   4.3695e-05    5.88557e-05      200    
reco:anglereconue:NuAngularReco                       0.00013761    0.00304744     0.0936005    0.000556104   0.00954668       200    
reco:anglereconumu:NuAngularReco                       5.906e-05    0.00276127     0.089395     0.000288514   0.00929859       200    
reco:anglereconuepfps:NuAngularReco                   9.2449e-05    0.00288514     0.0907607    0.000441485   0.00927735       200    
reco:anglereconumupfps:NuAngularReco                   7.98e-05     0.00280836     0.0900937    0.000372234   0.00925649       200    
reco:anglerecohits:NuAngularReco                      0.00019878    0.00376045     0.100013     0.000977818    0.0103241       200    
[art]:TriggerResults:TriggerResultInserter              9.1e-06     1.92374e-05   9.2519e-05     1.676e-05    1.14008e-05      200    
end_path:cafmaker:CAFMaker                            0.00154446     0.0379949      1.02411     0.00678574     0.112514        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3687.53 MB
  Peak resident set size usage (VmHWM): 1977.95 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 04:08:06 PDT ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/80/7d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_640_20231202T151037Z_gen_g4_detsim_hitreco__20240508T024154Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 940!
Begin processing the 195th record. run: 6420853 subRun: 1 event: 895 at 11-Sep-2025 03:59:56 PDT
Boundary wire vector sizes: 190, 172, 144
minwire 0: 1592
minwire 1: 1239
minwire 2: 1590
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000923888, 0.981387, 0.0176887, 
Output 1: 0.407017, 0.584904, 0.00760715, 0.000471576, 
Output 2: 0.993819, 0.00604829, 0.000126031, 6.96821e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1997!
Begin processing the 196th record. run: 6420853 subRun: 1 event: 896 at 11-Sep-2025 04:00:04 PDT
Boundary wire vector sizes: 105, 95, 105
minwire 0: 2229
minwire 1: 824
minwire 2: 2035
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000518276, 0.997675, 0.00180661, 
Output 1: 0.278792, 0.713697, 0.00724524, 0.000266264, 
Output 2: 0.998699, 0.00127137, 2.83987e-05, 9.27231e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1538!
Begin processing the 197th record. run: 6420853 subRun: 1 event: 897 at 11-Sep-2025 04:00:12 PDT
Boundary wire vector sizes: 62, 60, 51
minwire 0: 2151
minwire 1: 320
minwire 2: 2393
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.532629, 0.141387, 0.325984, 
Output 1: 0.925619, 0.0708459, 0.00332882, 0.000206459, 
Output 2: 0.736945, 0.25763, 0.00525348, 0.000171239, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 689!
Begin processing the 198th record. run: 6420853 subRun: 1 event: 898 at 11-Sep-2025 04:00:19 PDT
Boundary wire vector sizes: 244, 147, 166
minwire 0: 1410
minwire 1: 696
minwire 2: 1667
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000336735, 0.998388, 0.00127577, 
Output 1: 0.0721727, 0.893475, 0.0321706, 0.00218121, 
Output 2: 0.998178, 0.00175659, 6.28387e-05, 2.69286e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1455!
Begin processing the 199th record. run: 6420853 subRun: 1 event: 899 at 11-Sep-2025 04:00:27 PDT
Boundary wire vector sizes: 76, 81, 70
minwire 0: 1250
minwire 1: 1794
minwire 2: 981
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.736448, 0.0261956, 0.237356, 
Output 1: 0.99483, 0.00463795, 0.000299111, 0.000232558, 
Output 2: 0.904627, 0.0932494, 0.0018779, 0.000245416, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1275!
Begin processing the 200th record. run: 6420853 subRun: 1 event: 900 at 11-Sep-2025 04:00:34 PDT
Boundary wire vector sizes: 221, 144, 171
minwire 0: 2940
minwire 1: 633
minwire 2: 2745
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.973863, 0.00164902, 0.0244877, 
Output 1: 0.439273, 0.557196, 0.00336758, 0.000164149, 
Output 2: 0.931284, 0.0671407, 0.00153032, 4.50158e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1267!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/96/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_114_20231202T093453Z_gen_g4_detsim_hitreco__20240507T200056Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/b3/33/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_8_20231204T032213Z_gen_g4_detsim_hitreco__20240509T211540Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/80/7d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_640_20231202T151037Z_gen_g4_detsim_hitreco__20240508T024154Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8f/07/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_680_20231206T235340Z_gen_g4_detsim_hitreco__20240510T042933Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/94/d5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_892_20231202T161015Z_gen_g4_detsim_hitreco__20240508T043821Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_414_20231202T101230Z_gen_g4_detsim_hitreco__20240508T070720Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/76/04/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_187_20231208T053124Z_gen_g4_detsim_hitreco__20240510T044203Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_694_20231202T090147Z_gen_g4_detsim_hitreco__20240507T210632Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/96/51/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_418_20231208T095830Z_gen_g4_detsim_hitreco__20240510T064939Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/9c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_364_20231204T034027Z_gen_g4_detsim_hitreco__20240509T194855Z_reco2.root
.:
total 12040
-rw-r--r--. 1 cuser cuser 4555811 Sep 11 04:08 flatcaf.root
-rw-r--r--. 1 cuser cuser 3727381 Sep 11 04:08 caf.root
-rw-r--r--. 1 cuser cuser 3727381 Sep 11 04:08 caf_fd_hd_atmo_2501_20250911T101359Z.root
-rw-r--r--. 1 cuser cuser  132319 Sep 11 04:08 caf_20250911T101359Z.log
-rw-r--r--. 1 cuser cuser   67872 Sep 11 04:08 jobscript.log
-rw-r--r--. 1 cuser cuser    1921 Sep 11 04:08 caf_20250911T101359Z.file
-rw-r--r--. 1 cuser cuser    1921 Sep 11 04:08 caf_20250911T101359Z.pfns
-rw-r--r--. 1 cuser cuser    1921 Sep 11 03:13 file.list
-rw-r--r--. 1 cuser cuser    1921 Sep 11 04:08 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser    1371 Sep 11 03:13 all-input-dids.txt
-rw-r--r--. 1 cuser cuser    1371 Sep 11 04:08 caf_20250911T101359Z.did
-rw-r--r--. 1 cuser cuser    1371 Sep 11 03:13 did.list
-rw-r--r--. 1 cuser cuser       0 Sep 11 03:14 debugprod.log