Jobsub ID 103192.0@dunegpschedd02.fnal.gov

Jobsub ID	103192.0@dunegpschedd02.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 10:29:39
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-23 11:53:20
From worker node	Hostname	wn77
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:37:23
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_292_20231201T071109Z_gen_g4_detsim_hitreco__20240507T183933Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_23_20231208T052548Z_gen_g4_detsim_hitreco__20240510T052858Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_203_20231207T113617Z_gen_g4_detsim_hitreco__20240510T053119Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_298_20231203T232042Z_gen_g4_detsim_hitreco__20240509T204742Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_689_20231208T053834Z_gen_g4_detsim_hitreco__20240510T065456Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_638_20231207T073602Z_gen_g4_detsim_hitreco__20240510T063027Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_0_20231202T001343Z_gen_g4_detsim_hitreco__20240508T051809Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_88_20231204T232421Z_gen_g4_detsim_hitreco__20240510T020246Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_579_20231202T070300Z_gen_g4_detsim_hitreco__20240508T034352Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_243_20231207T122254Z_gen_g4_detsim_hitreco__20240510T060924Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 11:53:20
Saved logs	justin-logs:103192.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.591603      0.989725       1.96049      0.992041      0.179008        300    
reco:ophitspe:OpHitFinderDeco                           1.76581       1.80402       1.86806       1.77888      0.0394199       300    
reco:opflash:OpFlashFinder                            0.000180658    0.0016487    0.00732775    0.00154331    0.000827707      300    
reco:opslicer:OpSlicer                                6.6802e-05      0.3397        1.30861      0.256389      0.282652        300    
reco:rns:RandomNumberSaver                            2.5281e-05    3.78905e-05    0.0004659    3.6401e-05    2.53886e-05      300    
reco:anglereconue:NuAngularReco                       0.000206589   0.00141979     0.0645821    0.000453885   0.00447569       300    
reco:anglereconumu:NuAngularReco                      0.000114715   0.00127767     0.0662955    0.000324824   0.00454037       300    
reco:anglereconuepfps:NuAngularReco                   0.000180818   0.00139942     0.0678969    0.000427959   0.00462163       300    
reco:anglereconumupfps:NuAngularReco                  0.000156947   0.00136984     0.0671191    0.000408568   0.00459031       300    
reco:anglerecohits:NuAngularReco                      0.000152277   0.00209905     0.0741054    0.000922925   0.00520538       300    
[art]:TriggerResults:TriggerResultInserter             1.158e-05    1.77039e-05   6.0852e-05    1.6966e-05    5.61904e-06      300    
end_path:cafmaker:CAFMaker                            0.00153761     0.0176331     0.786436     0.00483244     0.0559057       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3650.32 MB
  Peak resident set size usage (VmHWM): 1964.62 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 08:49:05 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/67/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_298_20231203T232042Z_gen_g4_detsim_hitreco__20240509T204742Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.946883, 0.0101892, 0.0429275, 
Output 1: 0.26908, 0.674737, 0.0529458, 0.00323724, 
Output 2: 0.0356253, 0.390961, 0.469918, 0.103495, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1690!
Begin processing the 294th record. run: 6455553 subRun: 1 event: 20394 at 23-Sep-2025 08:40:11 -03
Boundary wire vector sizes: 490, 830, 519
minwire 0: 2078
minwire 1: 888
minwire 2: 1774
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.997121, 0.000469474, 0.00240932, 
Output 1: 0.00219496, 0.994188, 0.00354099, 7.61369e-05, 
Output 2: 0.995732, 0.00419157, 7.53957e-05, 1.26787e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2600!
Begin processing the 295th record. run: 6455553 subRun: 1 event: 20395 at 23-Sep-2025 08:40:24 -03
Boundary wire vector sizes: 43, 66, 46
minwire 0: 1046
minwire 1: 1578
minwire 2: 1072
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.107493, 0.0435744, 0.848933, 
Output 1: 0.0616928, 0.662682, 0.263455, 0.0121706, 
Output 2: 0.626161, 0.363125, 0.0105654, 0.000147764, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1366!
Begin processing the 296th record. run: 6455553 subRun: 1 event: 20396 at 23-Sep-2025 08:40:35 -03
Boundary wire vector sizes: 586, 152, 271
minwire 0: 1126
minwire 1: 2151
minwire 2: 819
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996117, 0.000696002, 0.00318656, 
Output 1: 0.0670981, 0.923569, 0.00895484, 0.000377793, 
Output 2: 0.984247, 0.0154605, 0.000286326, 6.47963e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2190!
Begin processing the 297th record. run: 6455553 subRun: 1 event: 20397 at 23-Sep-2025 08:40:46 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 622!
Begin processing the 298th record. run: 6455553 subRun: 1 event: 20398 at 23-Sep-2025 08:40:56 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 854!
Begin processing the 299th record. run: 6455553 subRun: 1 event: 20399 at 23-Sep-2025 08:41:05 -03
Boundary wire vector sizes: 730, 720, 732
minwire 0: 2008
minwire 1: 163
minwire 2: 1940
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000454449, 0.997121, 0.00242471, 
Output 1: 0.0518747, 0.917722, 0.0293596, 0.0010435, 
Output 2: 0.931197, 0.0678885, 0.000901625, 1.34087e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1657!
Begin processing the 300th record. run: 6455553 subRun: 1 event: 20400 at 23-Sep-2025 08:41:16 -03
Boundary wire vector sizes: 2059, 2903, 2189
minwire 0: 1275
minwire 1: 178
minwire 2: 1885
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0839653, 0.2094, 0.706634, 
Output 1: 0.257703, 0.568299, 0.143515, 0.0304818, 
Output 2: 0.0295701, 0.103172, 0.401612, 0.465646, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3072!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d6/71/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515154_292_20231201T071109Z_gen_g4_detsim_hitreco__20240507T183933Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/95/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_23_20231208T052548Z_gen_g4_detsim_hitreco__20240510T052858Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e3/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_203_20231207T113617Z_gen_g4_detsim_hitreco__20240510T053119Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/67/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_298_20231203T232042Z_gen_g4_detsim_hitreco__20240509T204742Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4f/58/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_689_20231208T053834Z_gen_g4_detsim_hitreco__20240510T065456Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/21/6b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_638_20231207T073602Z_gen_g4_detsim_hitreco__20240510T063027Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/10/32/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_0_20231202T001343Z_gen_g4_detsim_hitreco__20240508T051809Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c3/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_88_20231204T232421Z_gen_g4_detsim_hitreco__20240510T020246Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7c/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_579_20231202T070300Z_gen_g4_detsim_hitreco__20240508T034352Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1b/0d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_243_20231207T122254Z_gen_g4_detsim_hitreco__20240510T060924Z_reco2.root
.:
total 11728
-rw-r--r-- 1 nobody nobody 4454556 Sep 23 08:49 flatcaf.root
-rw-r--r-- 1 nobody nobody 3595106 Sep 23 08:49 caf.root
-rw-r--r-- 1 nobody nobody 3595106 Sep 23 08:49 caf_fd_hd_atmo_3302_20250923T103737Z.root
-rw-r--r-- 1 nobody nobody  186943 Sep 23 08:49 caf_20250923T103737Z.log
-rw-r--r-- 1 nobody nobody   75213 Sep 23 08:49 jobscript.log
-rw-r--r-- 1 nobody nobody    1929 Sep 23 08:49 caf_20250923T103737Z.file
-rw-r--r-- 1 nobody nobody    1929 Sep 23 08:49 caf_20250923T103737Z.pfns
-rw-r--r-- 1 nobody nobody    1929 Sep 23 07:37 file.list
-rw-r--r-- 1 nobody nobody    1929 Sep 23 08:49 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 23 07:37 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 23 08:49 caf_20250923T103737Z.did
-rw-r--r-- 1 nobody nobody    1375 Sep 23 07:37 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:37 debugprod.log