Jobsub ID 39988.118@dunegpschedd02.fnal.gov

Jobsub ID	39988.118@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:33:41
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw6
Last heartbeat	2025-09-15 13:43:09
From worker node	Hostname	mh-7763-12.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:36:05
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_560_20231204T100452Z_gen_g4_detsim_hitreco__20240509T215948Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_161_20231207T043406Z_gen_g4_detsim_hitreco__20240510T052915Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_420_20231207T083000Z_gen_g4_detsim_hitreco__20240510T034024Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_928_20231202T163530Z_gen_g4_detsim_hitreco__20240508T053139Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_561_20231204T140355Z_gen_g4_detsim_hitreco__20240509T202313Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_47_20231201T120729Z_gen_g4_detsim_hitreco__20240507T190140Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_683_20231202T081559Z_gen_g4_detsim_hitreco__20240507T195426Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_86_20231201T113732Z_gen_g4_detsim_hitreco__20240507T185502Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_1143_20231204T050450Z_gen_g4_detsim_hitreco__20240509T202346Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_653_20231208T050619Z_gen_g4_detsim_hitreco__20240510T041213Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 13:43:09
Saved logs	justin-logs:39988.118-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

91     0.105835        300    
reco:opdec:Deconvolution                               0.456583      0.812946       3.04296      0.792984       0.23123        300    
reco:ophitspe:OpHitFinderDeco                           1.34031        1.37         1.7315        1.36807      0.0443544       300    
reco:opflash:OpFlashFinder                            0.000156068   0.00167961    0.00625616    0.00152095    0.000914352      300    
reco:opslicer:OpSlicer                                6.7339e-05     0.299427       1.12856      0.231466      0.236816        300    
reco:rns:RandomNumberSaver                            2.3669e-05    6.00886e-05   0.000721762   5.32195e-05   4.52682e-05      300    
reco:anglereconue:NuAngularReco                       0.000192197   0.00166579     0.0857409    0.000508509   0.00582847       300    
reco:anglereconumu:NuAngularReco                      8.1489e-05    0.00145281     0.0928324    0.000306117   0.00611148       300    
reco:anglereconuepfps:NuAngularReco                   0.000114419   0.00156126     0.0819739    0.000438559   0.00560749       300    
reco:anglereconumupfps:NuAngularReco                  0.000106459   0.00141023     0.0591141    0.000367471   0.00454843       300    
reco:anglerecohits:NuAngularReco                      0.000139808   0.00222024     0.0732832     0.0008834    0.00555628       300    
[art]:TriggerResults:TriggerResultInserter             9.76e-06     1.60163e-05   6.8879e-05    1.47345e-05   6.79474e-06      300    
end_path:cafmaker:CAFMaker                            0.00146138     0.0217718      0.67754     0.00762944     0.0597522       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3669.3 MB
  Peak resident set size usage (VmHWM): 2106.03 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 06:42:44 PDT ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/61/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_928_20231202T163530Z_gen_g4_detsim_hitreco__20240508T053139Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Classifier summary: 
Output 0: 0.120871, 0.755963, 0.123166, 
Output 1: 0.0534276, 0.857945, 0.0862317, 0.00239543, 
Output 2: 0.945528, 0.0526232, 0.00180823, 4.08383e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1839!
Begin processing the 294th record. run: 50610358 subRun: 1 event: 42094 at 15-Sep-2025 06:33:50 PDT
Boundary wire vector sizes: 33, 37, 31
minwire 0: 720
minwire 1: 1537
minwire 2: 786
Used alternate method to get min and max wires due to vertex determination failure: 28, 327
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 102, 401
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00290159, 0.00199673, 0.995102, 
Output 1: 0.915867, 0.0835522, 0.000508576, 7.24286e-05, 
Output 2: 0.999083, 0.000894865, 1.98637e-05, 2.22058e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 484!
Begin processing the 295th record. run: 50610358 subRun: 1 event: 42095 at 15-Sep-2025 06:33:58 PDT
Boundary wire vector sizes: 382, 300, 282
minwire 0: 1884
minwire 1: 491
minwire 2: 1808
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00108721, 0.99716, 0.00175308, 
Output 1: 0.0116322, 0.0383993, 0.18581, 0.764158, 
Output 2: 0.995649, 0.00387905, 0.000360353, 0.000111866, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1731!
Begin processing the 296th record. run: 50610358 subRun: 1 event: 42096 at 15-Sep-2025 06:34:08 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 460!
Begin processing the 297th record. run: 50610358 subRun: 1 event: 42097 at 15-Sep-2025 06:34:16 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1311!
Begin processing the 298th record. run: 50610358 subRun: 1 event: 42098 at 15-Sep-2025 06:34:24 PDT
Boundary wire vector sizes: 57, 124, 75
minwire 0: 458
minwire 1: 1904
minwire 2: 21
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.88944, 0.0016774, 0.108882, 
Output 1: 0.864476, 0.133538, 0.00190302, 8.30392e-05, 
Output 2: 0.843609, 0.15425, 0.00209812, 4.31237e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1057!
Begin processing the 299th record. run: 50610358 subRun: 1 event: 42099 at 15-Sep-2025 06:34:32 PDT
Boundary wire vector sizes: 547, 581, 366
minwire 0: 1252
minwire 1: 807
minwire 2: 728
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0547551, 0.0589416, 0.886303, 
Output 1: 0.0703265, 0.154599, 0.398008, 0.377067, 
Output 2: 0.182252, 0.513535, 0.260099, 0.0441139, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3065!
Begin processing the 300th record. run: 50610358 subRun: 1 event: 42100 at 15-Sep-2025 06:34:45 PDT
Boundary wire vector sizes: 136, 104, 111
minwire 0: 1461
minwire 1: 683
minwire 2: 1693
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.015227, 0.0044905, 0.980283, 
Output 1: 0.0810164, 0.595498, 0.305443, 0.0180428, 
Output 2: 0.993037, 0.00665347, 0.000298869, 1.02313e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1512!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/25/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_560_20231204T100452Z_gen_g4_detsim_hitreco__20240509T215948Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_161_20231207T043406Z_gen_g4_detsim_hitreco__20240510T052915Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/99/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_420_20231207T083000Z_gen_g4_detsim_hitreco__20240510T034024Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/61/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_928_20231202T163530Z_gen_g4_detsim_hitreco__20240508T053139Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f1/15/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_561_20231204T140355Z_gen_g4_detsim_hitreco__20240509T202313Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/32/8d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_47_20231201T120729Z_gen_g4_detsim_hitreco__20240507T190140Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/13/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_683_20231202T081559Z_gen_g4_detsim_hitreco__20240507T195426Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5b/74/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_86_20231201T113732Z_gen_g4_detsim_hitreco__20240507T185502Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ca/09/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_1143_20231204T050450Z_gen_g4_detsim_hitreco__20240509T202346Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/74/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_653_20231208T050619Z_gen_g4_detsim_hitreco__20240510T041213Z_reco2.root
.:
total 13232
-rw-r--r-- 1 cuser cuser 5030111 Sep 15 06:42 flatcaf.root
-rw-r--r-- 1 cuser cuser 4077723 Sep 15 06:42 caf.root
-rw-r--r-- 1 cuser cuser 4077723 Sep 15 06:42 caf_fd_hd_atmo_2501_20250915T123615Z.root
-rw-r--r-- 1 cuser cuser  191593 Sep 15 06:42 caf_20250915T123615Z.log
-rw-r--r-- 1 cuser cuser   79414 Sep 15 06:42 jobscript.log
-rw-r--r-- 1 cuser cuser    1916 Sep 15 06:42 caf_20250915T123615Z.file
-rw-r--r-- 1 cuser cuser    1916 Sep 15 06:42 caf_20250915T123615Z.pfns
-rw-r--r-- 1 cuser cuser    1916 Sep 15 05:36 file.list
-rw-r--r-- 1 cuser cuser    1916 Sep 15 06:42 justin-processed-pfns.txt
-rw-r--r-- 1 cuser cuser    1378 Sep 15 05:36 all-input-dids.txt
-rw-r--r-- 1 cuser cuser    1378 Sep 15 06:42 caf_20250915T123615Z.did
-rw-r--r-- 1 cuser cuser    1378 Sep 15 05:36 did.list
-rw-r--r-- 1 cuser cuser       0 Sep 15 05:36 debugprod.log