Jobsub ID 103647.15@dunegpschedd02.fnal.gov

Jobsub ID	103647.15@dunegpschedd02.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 10:34:39
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-23 12:14:06
From worker node	Hostname	wn68
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:39:24
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_602_20231207T080206Z_gen_g4_detsim_hitreco__20240510T063004Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_839_20231203T070057Z_gen_g4_detsim_hitreco__20240508T080238Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_688_20231207T044412Z_gen_g4_detsim_hitreco__20240510T060656Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_626_20231208T053243Z_gen_g4_detsim_hitreco__20240510T065407Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_640_20231205T030217Z_gen_g4_detsim_hitreco__20240510T024925Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_456_20231205T030557Z_gen_g4_detsim_hitreco__20240510T020833Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_461_20231208T080546Z_gen_g4_detsim_hitreco__20240510T062210Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_223_20231202T095922Z_gen_g4_detsim_hitreco__20240507T200323Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_76_20231202T150339Z_gen_g4_detsim_hitreco__20240508T031805Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_224_20231208T080153Z_gen_g4_detsim_hitreco__20240510T052630Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 12:14:06
Saved logs	justin-logs:103647.15-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.605293       1.00805       1.98497       1.00698      0.202864        300    
reco:ophitspe:OpHitFinderDeco                           1.76562       1.82766       1.88896       1.84937      0.0379298       300    
reco:opflash:OpFlashFinder                            0.000188238   0.00161184     0.0100269     0.0015484    0.000886138      300    
reco:opslicer:OpSlicer                                8.9314e-05     0.335265       1.67024      0.266973      0.285706        300    
reco:rns:RandomNumberSaver                            2.2661e-05    3.86418e-05   0.000433837   3.6331e-05    2.47873e-05      300    
reco:anglereconue:NuAngularReco                       0.000197678   0.00214436     0.155929     0.000461818    0.0116481       300    
reco:anglereconumu:NuAngularReco                      0.000106024   0.00198501     0.153089     0.000289157    0.0116388       300    
reco:anglereconuepfps:NuAngularReco                   0.000155826   0.00208806     0.153249     0.000422755    0.0115479       300    
reco:anglereconumupfps:NuAngularReco                  0.000151946   0.00205475     0.154211     0.000384395    0.0115206       300    
reco:anglerecohits:NuAngularReco                      0.000175547   0.00280723      0.16365     0.000925631     0.01237        300    
[art]:TriggerResults:TriggerResultInserter             1.056e-05    1.75142e-05   0.000128465   1.6821e-05    7.70165e-06      300    
end_path:cafmaker:CAFMaker                             0.0015035     0.0267756      1.8405      0.00474334     0.136543        300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3744.92 MB
  Peak resident set size usage (VmHWM): 2052.31 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 09:09:52 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/06/90/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_626_20231208T053243Z_gen_g4_detsim_hitreco__20240510T065407Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 0: 103
minwire 1: 2248
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.268187, 0.6042, 0.127612, 
Output 1: 0.466433, 0.498111, 0.0324843, 0.00297209, 
Output 2: 0.252146, 0.724617, 0.0227143, 0.000523292, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1182!
Begin processing the 295th record. run: 50607041 subRun: 1 event: 68895 at 23-Sep-2025 09:01:31 -03
Boundary wire vector sizes: 255, 455, 400
minwire 0: 1031
minwire 1: 1265
minwire 2: 876
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.990544, 0.000907387, 0.00854822, 
Output 1: 0.00658258, 0.0235955, 0.117581, 0.852241, 
Output 2: 0.938776, 0.0580858, 0.00277187, 0.00036679, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1922!
Begin processing the 296th record. run: 50607041 subRun: 1 event: 68896 at 23-Sep-2025 09:01:42 -03
Boundary wire vector sizes: 121, 91, 94
minwire 0: 1872
minwire 1: 715
minwire 2: 1672
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 396, 695
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000659368, 0.996079, 0.00326138, 
Output 1: 0.957725, 0.0419136, 0.000311473, 4.98811e-05, 
Output 2: 0.998667, 0.00129634, 3.20239e-05, 4.59439e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1689!
Begin processing the 297th record. run: 50607041 subRun: 1 event: 68897 at 23-Sep-2025 09:01:53 -03
Boundary wire vector sizes: 197, 255, 220
minwire 0: 2774
minwire 1: 750
minwire 2: 2508
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000208873, 0.99868, 0.00111136, 
Output 1: 0.0616231, 0.927039, 0.0109484, 0.000389261, 
Output 2: 0.998609, 0.00135497, 3.49245e-05, 9.73859e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1458!
Begin processing the 298th record. run: 50607041 subRun: 1 event: 68898 at 23-Sep-2025 09:02:04 -03
Boundary wire vector sizes: 338, 490, 285
minwire 0: 855
minwire 1: 1971
minwire 2: 506
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.973958, 0.0143141, 0.0117279, 
Output 1: 0.279729, 0.548259, 0.142306, 0.0297066, 
Output 2: 0.809241, 0.181776, 0.00851067, 0.000472264, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2073!
Begin processing the 299th record. run: 50607041 subRun: 1 event: 68899 at 23-Sep-2025 09:02:16 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 301!
Begin processing the 300th record. run: 50607041 subRun: 1 event: 68900 at 23-Sep-2025 09:02:26 -03
Boundary wire vector sizes: 61, 74, 64
minwire 0: 2210
minwire 1: 722
minwire 2: 2244
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992001, 0.000551389, 0.00744786, 
Output 1: 0.540689, 0.455895, 0.00331401, 0.000101942, 
Output 2: 0.993196, 0.00667788, 0.000122279, 3.88733e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1195!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/eb/44/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_602_20231207T080206Z_gen_g4_detsim_hitreco__20240510T063004Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/85/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_839_20231203T070057Z_gen_g4_detsim_hitreco__20240508T080238Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/15/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_688_20231207T044412Z_gen_g4_detsim_hitreco__20240510T060656Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/06/90/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_626_20231208T053243Z_gen_g4_detsim_hitreco__20240510T065407Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/13/9b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_640_20231205T030217Z_gen_g4_detsim_hitreco__20240510T024925Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/a8/91/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_456_20231205T030557Z_gen_g4_detsim_hitreco__20240510T020833Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5e/59/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_461_20231208T080546Z_gen_g4_detsim_hitreco__20240510T062210Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/75/34/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_223_20231202T095922Z_gen_g4_detsim_hitreco__20240507T200323Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e2/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_76_20231202T150339Z_gen_g4_detsim_hitreco__20240508T031805Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_224_20231208T080153Z_gen_g4_detsim_hitreco__20240510T052630Z_reco2.root
.:
total 11884
-rw-r--r-- 1 nobody nobody 4507281 Sep 23 09:09 flatcaf.root
-rw-r--r-- 1 nobody nobody 3645649 Sep 23 09:09 caf.root
-rw-r--r-- 1 nobody nobody 3645649 Sep 23 09:09 caf_fd_hd_atmo_3301_20250923T103937Z.root
-rw-r--r-- 1 nobody nobody  189637 Sep 23 09:09 caf_20250923T103937Z.log
-rw-r--r-- 1 nobody nobody   72337 Sep 23 09:09 jobscript.log
-rw-r--r-- 1 nobody nobody    1950 Sep 23 09:09 caf_20250923T103937Z.file
-rw-r--r-- 1 nobody nobody    1950 Sep 23 09:09 caf_20250923T103937Z.pfns
-rw-r--r-- 1 nobody nobody    1950 Sep 23 07:39 file.list
-rw-r--r-- 1 nobody nobody    1950 Sep 23 09:09 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:39 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 09:09 caf_20250923T103937Z.did
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:39 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:40 debugprod.log