Jobsub ID 39988.130@dunegpschedd02.fnal.gov

Jobsub ID	39988.130@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:33:41
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-09-15 13:01:42
From worker node	Hostname	mh-7763-3.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:35:06
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_633_20231203T145238Z_gen_g4_detsim_hitreco__20240508T080222Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_711_20231204T223800Z_gen_g4_detsim_hitreco__20240510T032612Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1007_20231203T051438Z_gen_g4_detsim_hitreco__20240508T071008Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_1188_20231203T123657Z_gen_g4_detsim_hitreco__20240508T075435Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_239_20231202T202525Z_gen_g4_detsim_hitreco__20240508T061713Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_577_20231204T223216Z_gen_g4_detsim_hitreco__20240509T221544Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_413_20231202T054450Z_gen_g4_detsim_hitreco__20240508T053136Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_330_20231122T063804Z_gen_g4_detsim_hitreco__20240503T060044Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_342_20231203T040647Z_gen_g4_detsim_hitreco__20240508T045028Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_622_20231203T043428Z_gen_g4_detsim_hitreco__20240508T065036Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 13:01:42
Saved logs	justin-logs:39988.130-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.596019     0.00347934     0.0913407       100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000746436    0.0273408     0.598353     0.00362583     0.0917304       100    
reco:opdec:Deconvolution                               0.503729      0.780209       1.22524      0.787611      0.156013        100    
reco:ophitspe:OpHitFinderDeco                           1.40508       1.41173       1.42689       1.41155     0.00327474       100    
reco:opflash:OpFlashFinder                            0.000392037   0.00145759    0.00428396    0.00135534    0.000622861      100    
reco:opslicer:OpSlicer                                0.00387014     0.293217      0.888355      0.237099      0.221827        100    
reco:rns:RandomNumberSaver                             1.873e-05    4.45054e-05   0.000418588    3.763e-05    4.05772e-05      100    
reco:anglereconue:NuAngularReco                       0.000173149   0.00108013     0.0217639    0.000430232   0.00235694       100    
reco:anglereconumu:NuAngularReco                      8.4079e-05    0.000896809    0.0215674    0.000267694   0.00234276       100    
reco:anglereconuepfps:NuAngularReco                   0.000129249   0.00103517     0.0215423    0.000368973   0.00235449       100    
reco:anglereconumupfps:NuAngularReco                  0.000109009   0.000960782    0.0214552    0.000321828   0.00233336       100    
reco:anglerecohits:NuAngularReco                      0.000302847   0.00157689     0.0243174    0.000775835    0.0027068       100    
[art]:TriggerResults:TriggerResultInserter             9.06e-06     1.50056e-05   8.8219e-05    1.39495e-05   8.58393e-06      100    
end_path:cafmaker:CAFMaker                            0.00138457     0.155652       14.1458      0.0060815      1.40632        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3648.88 MB
  Peak resident set size usage (VmHWM): 1846.62 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 06:01:18 PDT ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/1d/d7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_711_20231204T223800Z_gen_g4_detsim_hitreco__20240510T032612Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 463!
Begin processing the 92nd record. run: 74511302 subRun: 1 event: 63392 at 15-Sep-2025 05:52:28 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 664!
Begin processing the 93rd record. run: 74511302 subRun: 1 event: 63393 at 15-Sep-2025 05:52:35 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1237!
Begin processing the 94th record. run: 74511302 subRun: 1 event: 63394 at 15-Sep-2025 05:52:42 PDT
Boundary wire vector sizes: 172, 154, 163
minwire 0: 1194
minwire 1: 1024
minwire 2: 1316
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.355775, 0.012195, 0.632031, 
Output 1: 0.0242514, 0.347121, 0.582862, 0.045766, 
Output 2: 0.0251363, 0.219745, 0.625589, 0.129529, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1657!
Begin processing the 95th record. run: 74511302 subRun: 1 event: 63395 at 15-Sep-2025 05:52:51 PDT
Boundary wire vector sizes: 329, 322, 359
minwire 0: 1993
minwire 1: 294
minwire 2: 2271
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992965, 0.000521308, 0.00651413, 
Output 1: 0.00801079, 0.980717, 0.0107015, 0.000570427, 
Output 2: 0.948364, 0.0502999, 0.00131128, 2.50273e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1192!
Begin processing the 96th record. run: 74511302 subRun: 1 event: 63396 at 15-Sep-2025 05:52:59 PDT
Boundary wire vector sizes: 1714, 2040, 1330
minwire 0: 1616
minwire 1: 506
minwire 2: 1954
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.243365, 0.543851, 0.212784, 
Output 1: 0.150449, 0.82096, 0.0275274, 0.00106348, 
Output 2: 0.0481858, 0.40568, 0.452813, 0.0933206, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2342!
Begin processing the 97th record. run: 74511302 subRun: 1 event: 63397 at 15-Sep-2025 05:53:13 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 385!
Begin processing the 98th record. run: 74511302 subRun: 1 event: 63398 at 15-Sep-2025 05:53:20 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 149!
Begin processing the 99th record. run: 74511302 subRun: 1 event: 63399 at 15-Sep-2025 05:53:27 PDT
Boundary wire vector sizes: 1061, 1077, 1129
minwire 0: 2350
minwire 1: 659
minwire 2: 2040
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.896543, 0.00268783, 0.100769, 
Output 1: 0.343963, 0.642452, 0.0130468, 0.000538433, 
Output 2: 0.0959832, 0.550804, 0.319009, 0.0342034, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2485!
Begin processing the 100th record. run: 74511302 subRun: 1 event: 63400 at 15-Sep-2025 05:53:37 PDT
Boundary wire vector sizes: 46, 51, 40
minwire 0: 525
minwire 1: 2400
minwire 2: 260
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00187586, 0.996572, 0.00155163, 
Output 1: 0.435105, 0.560808, 0.00393147, 0.000154759, 
Output 2: 0.998641, 0.00133, 2.78673e-05, 1.19973e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 782!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/79/0e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_633_20231203T145238Z_gen_g4_detsim_hitreco__20240508T080222Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/1d/d7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_711_20231204T223800Z_gen_g4_detsim_hitreco__20240510T032612Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ab/f9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1007_20231203T051438Z_gen_g4_detsim_hitreco__20240508T071008Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/aa/5c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_1188_20231203T123657Z_gen_g4_detsim_hitreco__20240508T075435Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d1/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_239_20231202T202525Z_gen_g4_detsim_hitreco__20240508T061713Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6e/5c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_577_20231204T223216Z_gen_g4_detsim_hitreco__20240509T221544Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/89/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_413_20231202T054450Z_gen_g4_detsim_hitreco__20240508T053136Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fb/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_330_20231122T063804Z_gen_g4_detsim_hitreco__20240503T060044Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/5f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_342_20231203T040647Z_gen_g4_detsim_hitreco__20240508T045028Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1e/e6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_622_20231203T043428Z_gen_g4_detsim_hitreco__20240508T065036Z_reco2.root
.:
total 4664
-rw-r--r--. 1 cuser cuser 1841287 Sep 15 06:01 flatcaf.root
-rw-r--r--. 1 cuser cuser 1395547 Sep 15 06:01 caf.root
-rw-r--r--. 1 cuser cuser 1395547 Sep 15 06:01 caf_fd_hd_atmo_2501_20250915T123517Z.root
-rw-r--r--. 1 cuser cuser   71965 Sep 15 06:01 caf_20250915T123517Z.log
-rw-r--r--. 1 cuser cuser   33075 Sep 15 06:01 jobscript.log
-rw-r--r--. 1 cuser cuser    1871 Sep 15 06:01 caf_20250915T123517Z.file
-rw-r--r--. 1 cuser cuser    1871 Sep 15 06:01 caf_20250915T123517Z.pfns
-rw-r--r--. 1 cuser cuser    1871 Sep 15 05:35 file.list
-rw-r--r--. 1 cuser cuser    1871 Sep 15 06:01 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser    1379 Sep 15 05:35 all-input-dids.txt
-rw-r--r--. 1 cuser cuser    1379 Sep 15 06:01 caf_20250915T123517Z.did
-rw-r--r--. 1 cuser cuser    1379 Sep 15 05:35 did.list
-rw-r--r--. 1 cuser cuser       0 Sep 15 05:35 debugprod.log