Jobsub ID 46603.141@dunegpschedd01.fnal.gov

Jobsub ID	46603.141@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:31:41
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-15 14:29:54
From worker node	Hostname	wn39
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 13:35:00
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_1187_20231203T163342Z_gen_g4_detsim_hitreco__20240508T052550Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_740_20231208T084350Z_gen_g4_detsim_hitreco__20240510T060944Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_870_20231203T050845Z_gen_g4_detsim_hitreco__20240508T072631Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_778_20231207T022936Z_gen_g4_detsim_hitreco__20240510T062625Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_849_20231205T060739Z_gen_g4_detsim_hitreco__20240510T024133Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_185_20231208T045000Z_gen_g4_detsim_hitreco__20240510T065426Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220762_458_20231122T074959Z_gen_g4_detsim_hitreco__20240503T064749Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_398_20231201T115920Z_gen_g4_detsim_hitreco__20240507T194443Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_208_20231203T131601Z_gen_g4_detsim_hitreco__20240508T072534Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_891_20231203T123938Z_gen_g4_detsim_hitreco__20240509T201226Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 14:29:54
Saved logs	justin-logs:46603.141-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

o:opdec:Deconvolution                               0.641767       1.05584       1.91836       1.05045      0.177385        200    
reco:ophitspe:OpHitFinderDeco                           1.83474       1.87464       1.92224       1.85077      0.034123        200    
reco:opflash:OpFlashFinder                            0.000225995   0.00204646    0.00960624    0.00192497    0.00101947       200    
reco:opslicer:OpSlicer                                0.000164218    0.370018       1.33188      0.282465      0.292166        200    
reco:rns:RandomNumberSaver                            2.3899e-05    6.07824e-05   0.000515856   5.46145e-05   3.78464e-05      200    
reco:anglereconue:NuAngularReco                       0.000221895   0.00161755     0.0391505    0.000605164   0.00337907       200    
reco:anglereconumu:NuAngularReco                      0.000139016   0.00133146     0.0379654    0.000364243    0.003245        200    
reco:anglereconuepfps:NuAngularReco                   0.000206769   0.00153996     0.0380732    0.000557717   0.00328221       200    
reco:anglereconumupfps:NuAngularReco                  0.000205259   0.00148124     0.0393683    0.000499414   0.00334497       200    
reco:anglerecohits:NuAngularReco                      0.000183043   0.00235641     0.043761     0.00113794    0.00392568       200    
[art]:TriggerResults:TriggerResultInserter            1.1455e-05    2.36347e-05   8.6068e-05    2.26225e-05   7.29265e-06      200    
end_path:cafmaker:CAFMaker                            0.00171366     0.0226823     0.447351     0.00831054     0.043882        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3652.26 MB
  Peak resident set size usage (VmHWM): 1986.92 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 11:22:14 -03 ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_870_20231203T050845Z_gen_g4_detsim_hitreco__20240508T072631Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00246204, 0.9098, 0.0877376, 
Output 1: 0.00959057, 0.980579, 0.00961597, 0.000214416, 
Output 2: 0.996925, 0.00299214, 7.96571e-05, 3.23452e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 991!
Begin processing the 195th record. run: 6472706 subRun: 1 event: 74095 at 15-Sep-2025 11:21:02 -03
Boundary wire vector sizes: 38, 40, 41
minwire 0: 858
minwire 1: 1717
minwire 2: 770
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 71, 370
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0370332, 0.00337775, 0.959589, 
Output 1: 0.0248761, 0.153014, 0.63215, 0.18996, 
Output 2: 0.97502, 0.0237417, 0.00117573, 6.2136e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1282!
Begin processing the 196th record. run: 6472706 subRun: 1 event: 74096 at 15-Sep-2025 11:21:12 -03
Boundary wire vector sizes: 1477, 1442, 919
minwire 0: 1230
minwire 1: 691
minwire 2: 1121
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0023456, 0.960697, 0.0369571, 
Output 1: 0.662465, 0.304111, 0.0308853, 0.00253898, 
Output 2: 0.0877151, 0.836167, 0.0728877, 0.00322982, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2933!
Begin processing the 197th record. run: 6472706 subRun: 1 event: 74097 at 15-Sep-2025 11:21:30 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 205!
Begin processing the 198th record. run: 6472706 subRun: 1 event: 74098 at 15-Sep-2025 11:21:39 -03
Boundary wire vector sizes: 149, 158, 92
minwire 0: 2308
minwire 1: 848
minwire 2: 2197
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000774982, 0.988509, 0.0107161, 
Output 1: 0.106621, 0.873094, 0.0195687, 0.000717037, 
Output 2: 0.997404, 0.00250353, 8.83801e-05, 4.2231e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1675!
Begin processing the 199th record. run: 6472706 subRun: 1 event: 74099 at 15-Sep-2025 11:21:50 -03
Boundary wire vector sizes: 310, 360, 367
minwire 0: 382
minwire 1: 1577
minwire 2: 387
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995595, 0.000816471, 0.00358868, 
Output 1: 0.830207, 0.169129, 0.000579107, 8.53014e-05, 
Output 2: 0.997999, 0.00192886, 6.44727e-05, 7.87559e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1441!
Begin processing the 200th record. run: 6472706 subRun: 1 event: 74100 at 15-Sep-2025 11:22:02 -03
Boundary wire vector sizes: 37, 36, 28
minwire 0: 1139
minwire 1: 1888
minwire 2: 771
Used alternate method to get min and max wires due to vertex determination failure: 230, 529
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 230, 529
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00319602, 0.0025709, 0.994233, 
Output 1: 0.963317, 0.0358626, 0.000701616, 0.000118574, 
Output 2: 0.997333, 0.00261114, 5.0429e-05, 5.41052e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 981!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/54/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_1187_20231203T163342Z_gen_g4_detsim_hitreco__20240508T052550Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c9/59/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_740_20231208T084350Z_gen_g4_detsim_hitreco__20240510T060944Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_870_20231203T050845Z_gen_g4_detsim_hitreco__20240508T072631Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ea/5f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_778_20231207T022936Z_gen_g4_detsim_hitreco__20240510T062625Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/d1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_849_20231205T060739Z_gen_g4_detsim_hitreco__20240510T024133Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4b/8a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_185_20231208T045000Z_gen_g4_detsim_hitreco__20240510T065426Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/07/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220762_458_20231122T074959Z_gen_g4_detsim_hitreco__20240503T064749Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3a/ae/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481119_398_20231201T115920Z_gen_g4_detsim_hitreco__20240507T194443Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7d/59/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_208_20231203T131601Z_gen_g4_detsim_hitreco__20240508T072534Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c1/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_891_20231203T123938Z_gen_g4_detsim_hitreco__20240509T201226Z_reco2.root
.:
total 7104
-rw-r--r-- 1 nobody nobody 2758721 Sep 15 11:22 flatcaf.root
-rw-r--r-- 1 nobody nobody 2134694 Sep 15 11:22 caf.root
-rw-r--r-- 1 nobody nobody 2134694 Sep 15 11:22 caf_fd_hd_atmo_2501_20250915T133518Z.root
-rw-r--r-- 1 nobody nobody  134284 Sep 15 11:22 caf_20250915T133518Z.log
-rw-r--r-- 1 nobody nobody   53558 Sep 15 11:22 jobscript.log
-rw-r--r-- 1 nobody nobody    1866 Sep 15 11:22 caf_20250915T133518Z.file
-rw-r--r-- 1 nobody nobody    1866 Sep 15 11:22 caf_20250915T133518Z.pfns
-rw-r--r-- 1 nobody nobody    1866 Sep 15 10:35 file.list
-rw-r--r-- 1 nobody nobody    1866 Sep 15 11:22 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 15 10:35 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 15 11:22 caf_20250915T133518Z.did
-rw-r--r-- 1 nobody nobody    1376 Sep 15 10:35 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 15 10:35 debugprod.log