Jobsub ID 106434.16@dunegpschedd01.fnal.gov

Jobsub ID	106434.16@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:57:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-23 12:50:53
From worker node	Hostname	wn54
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:02:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_494_20231207T124654Z_gen_g4_detsim_hitreco__20240510T062045Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_852_20231203T113855Z_gen_g4_detsim_hitreco__20240509T200811Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_794_20231207T053144Z_gen_g4_detsim_hitreco__20240510T055133Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6454800_46_20231207T090644Z_gen_g4_detsim_hitreco__20240510T062035Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_408_20231202T190846Z_gen_g4_detsim_hitreco__20240508T062957Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_490_20231205T033904Z_gen_g4_detsim_hitreco__20240510T020307Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_453_20231207T025240Z_gen_g4_detsim_hitreco__20240510T044338Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_406_20231202T113227Z_gen_g4_detsim_hitreco__20240507T214847Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_741_20231203T121104Z_gen_g4_detsim_hitreco__20240509T200544Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_95_20231205T025543Z_gen_g4_detsim_hitreco__20240510T024601Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 12:50:53
Saved logs	justin-logs:106434.16-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gyReco                  0.000784329    0.0598054      2.9249      0.00459911     0.235326        600    
reco:opdec:Deconvolution                               0.617387       1.11102       2.46543       1.02924      0.290882        600    
reco:ophitspe:OpHitFinderDeco                           1.76382       1.81458       1.87622       1.81832      0.0381564       600    
reco:opflash:OpFlashFinder                            0.000207176   0.00202466    0.00752104    0.00190895    0.000905613      600    
reco:opslicer:OpSlicer                                7.3683e-05     0.354813       1.57993       0.26775      0.297406        600    
reco:rns:RandomNumberSaver                            2.6638e-05    6.41137e-05   0.000732775   4.9766e-05    8.33162e-05      600    
reco:anglereconue:NuAngularReco                       0.000243584   0.00258348     0.0987224    0.000653072   0.00774114       600    
reco:anglereconumu:NuAngularReco                      0.000139019   0.00225227     0.0969189    0.000421619   0.00766595       600    
reco:anglereconuepfps:NuAngularReco                   0.000199547   0.00247015     0.0966792    0.00058493    0.00773707       600    
reco:anglereconumupfps:NuAngularReco                  0.000184273   0.00241952     0.0962959    0.000562806   0.00776236       600    
reco:anglerecohits:NuAngularReco                      0.000181081   0.00337723     0.105335     0.00120085    0.00866597       600    
[art]:TriggerResults:TriggerResultInserter            1.2539e-05    2.29849e-05   9.1314e-05    2.1413e-05    6.03548e-06      600    
end_path:cafmaker:CAFMaker                            0.00165696     0.0346262      1.19386     0.00916949     0.0998472       600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3688.55 MB
  Peak resident set size usage (VmHWM): 2014.26 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 09:46:35 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/6c/61/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_453_20231207T025240Z_gen_g4_detsim_hitreco__20240510T044338Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.000240722, 0.99861, 0.00114897, 
Output 1: 0.0302785, 0.958932, 0.0103303, 0.000459647, 
Output 2: 0.991727, 0.00808533, 0.000182143, 5.24988e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2480!
Begin processing the 593rd record. run: 66020489 subRun: 1 event: 49093 at 23-Sep-2025 09:37:09 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 777!
Begin processing the 594th record. run: 66020489 subRun: 1 event: 49094 at 23-Sep-2025 09:37:19 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 520!
Begin processing the 595th record. run: 66020489 subRun: 1 event: 49095 at 23-Sep-2025 09:37:30 -03
Boundary wire vector sizes: 512, 544, 507
minwire 0: 335
minwire 1: 1998
minwire 2: 20
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00151754, 0.995014, 0.00346812, 
Output 1: 0.0663426, 0.331608, 0.397578, 0.204471, 
Output 2: 0.994772, 0.0049607, 0.000247159, 2.04386e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1669!
Begin processing the 596th record. run: 66020489 subRun: 1 event: 49096 at 23-Sep-2025 09:37:46 -03
Boundary wire vector sizes: 176, 313, 263
minwire 0: 2020
minwire 1: 69
minwire 2: 2587
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.979917, 0.000972063, 0.0191111, 
Output 1: 0.0730095, 0.385743, 0.444482, 0.0967653, 
Output 2: 0.599539, 0.384233, 0.0158441, 0.000384469, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1003!
Begin processing the 597th record. run: 66020489 subRun: 1 event: 49097 at 23-Sep-2025 09:37:59 -03
Boundary wire vector sizes: 135, 108, 118
minwire 0: 2085
minwire 1: 436
minwire 2: 2172
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.111861, 0.0151059, 0.873033, 
Output 1: 0.526161, 0.352042, 0.109886, 0.0119118, 
Output 2: 0.215086, 0.74716, 0.036879, 0.000875012, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1453!
Begin processing the 598th record. run: 66020489 subRun: 1 event: 49098 at 23-Sep-2025 09:38:11 -03
Boundary wire vector sizes: 114, 143, 121
minwire 0: 707
minwire 1: 1658
minwire 2: 744
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00119109, 0.995704, 0.00310462, 
Output 1: 0.0166211, 0.147762, 0.759456, 0.0761601, 
Output 2: 0.911596, 0.0850877, 0.00325262, 6.37022e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1340!
Begin processing the 599th record. run: 66020489 subRun: 1 event: 49099 at 23-Sep-2025 09:38:24 -03
Boundary wire vector sizes: 2072, 2437, 1466
minwire 0: 210
minwire 1: 1679
minwire 2: 183
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00344263, 0.863947, 0.13261, 
Output 1: 0.603877, 0.361498, 0.0266537, 0.00797184, 
Output 2: 0.319201, 0.556579, 0.101538, 0.022682, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2733!
Begin processing the 600th record. run: 66020489 subRun: 1 event: 49100 at 23-Sep-2025 09:38:53 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 689!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552621_494_20231207T124654Z_gen_g4_detsim_hitreco__20240510T062045Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/af/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_852_20231203T113855Z_gen_g4_detsim_hitreco__20240509T200811Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/97/46/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_794_20231207T053144Z_gen_g4_detsim_hitreco__20240510T055133Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e2/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6454800_46_20231207T090644Z_gen_g4_detsim_hitreco__20240510T062035Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e9/93/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_408_20231202T190846Z_gen_g4_detsim_hitreco__20240508T062957Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/01/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_490_20231205T033904Z_gen_g4_detsim_hitreco__20240510T020307Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/6c/61/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_453_20231207T025240Z_gen_g4_detsim_hitreco__20240510T044338Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/70/69/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_406_20231202T113227Z_gen_g4_detsim_hitreco__20240507T214847Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_741_20231203T121104Z_gen_g4_detsim_hitreco__20240509T200544Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/59/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_95_20231205T025543Z_gen_g4_detsim_hitreco__20240510T024601Z_reco2.root
.:
total 25288
-rw-r--r-- 1 nobody nobody 9537677 Sep 23 09:46 flatcaf.root
-rw-r--r-- 1 nobody nobody 7894324 Sep 23 09:46 caf.root
-rw-r--r-- 1 nobody nobody 7894324 Sep 23 09:46 caf_fd_hd_atmo_3302_20250923T100222Z.root
-rw-r--r-- 1 nobody nobody  363706 Sep 23 09:46 caf_20250923T100222Z.log
-rw-r--r-- 1 nobody nobody  150976 Sep 23 09:46 jobscript.log
-rw-r--r-- 1 nobody nobody    1915 Sep 23 09:46 caf_20250923T100222Z.file
-rw-r--r-- 1 nobody nobody    1915 Sep 23 09:46 caf_20250923T100222Z.pfns
-rw-r--r-- 1 nobody nobody    1915 Sep 23 07:02 file.list
-rw-r--r-- 1 nobody nobody    1915 Sep 23 09:46 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1373 Sep 23 07:02 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1373 Sep 23 09:46 caf_20250923T100222Z.did
-rw-r--r-- 1 nobody nobody    1373 Sep 23 07:02 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:02 debugprod.log