Jobsub ID 40104.11@dunegpschedd02.fnal.gov

Jobsub ID	40104.11@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 15:47:50
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-09-15 16:11:03
From worker node	Hostname	wn-lot-031.farm.nikhef.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 15:49:46
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_1019_20231204T073824Z_gen_g4_detsim_hitreco__20240509T202119Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_858_20231203T235058Z_gen_g4_detsim_hitreco__20240509T211514Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_795_20231202T112635Z_gen_g4_detsim_hitreco__20240507T211031Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_65_20231207T022350Z_gen_g4_detsim_hitreco__20240510T042353Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_703_20231208T033513Z_gen_g4_detsim_hitreco__20240510T064741Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_315_20231205T024901Z_gen_g4_detsim_hitreco__20240509T222848Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_944_20231203T084526Z_gen_g4_detsim_hitreco__20240509T211746Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_750_20231208T063316Z_gen_g4_detsim_hitreco__20240510T051813Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_361_20231208T092527Z_gen_g4_detsim_hitreco__20240510T063619Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 16:11:03
Saved logs	justin-logs:40104.11-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

997        200    
reco:opdec:Deconvolution                               0.0476489     0.225514       0.45015      0.225848      0.0776518       200    
reco:ophitspe:OpHitFinderDeco                          0.128439      0.133456      0.141475      0.133412     0.00214521       200    
reco:opflash:OpFlashFinder                            0.000188402   0.00118315    0.00345005    0.00114689    0.000545372      200    
reco:opslicer:OpSlicer                                 5.319e-05     0.243383       1.07526       0.20622      0.193259        200    
reco:rns:RandomNumberSaver                            1.7452e-05    2.91465e-05   0.000351357   2.64445e-05   2.32398e-05      200    
reco:anglereconue:NuAngularReco                       0.000141534   0.00174441     0.0652282    0.000348767    0.0066184       200    
reco:anglereconumu:NuAngularReco                      7.4739e-05    0.00159257     0.0618842    0.000215407   0.00653043       200    
reco:anglereconuepfps:NuAngularReco                   0.000115526   0.00170262     0.0631294    0.000306533   0.00660643       200    
reco:anglereconumupfps:NuAngularReco                  0.000107742   0.00164974     0.0634257    0.000272264    0.0065291       200    
reco:anglerecohits:NuAngularReco                      0.000123982   0.00219483     0.0682109    0.000636559   0.00707216       200    
[art]:TriggerResults:TriggerResultInserter             9.958e-06    1.20547e-05   4.0346e-05    1.1852e-05    2.58974e-06      200    
end_path:cafmaker:CAFMaker                            0.00133835     0.0223367     0.779336     0.00424895     0.0809374       200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3682.35 MB
  Peak resident set size usage (VmHWM): 2022.44 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 18:10:41 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1789!
Begin processing the 193rd record. run: 66009532 subRun: 1 event: 85893 at 15-Sep-2025 18:10:18 CEST
Boundary wire vector sizes: 30, 60, 42
minwire 0: 1491
minwire 1: 1786
minwire 2: 1186
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.152879, 0.831507, 0.0156133, 
Output 1: 0.196028, 0.795636, 0.00804985, 0.000286348, 
Output 2: 0.992075, 0.0076981, 0.00021697, 1.00908e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1298!
Begin processing the 194th record. run: 66009532 subRun: 1 event: 85894 at 15-Sep-2025 18:10:21 CEST
Boundary wire vector sizes: 63, 77, 39
minwire 0: 805
minwire 1: 2109
minwire 2: 320
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.31385, 0.00352548, 0.682624, 
Output 1: 0.112954, 0.859717, 0.0260275, 0.00130212, 
Output 2: 0.296943, 0.694375, 0.00860031, 8.17906e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1393!
Begin processing the 195th record. run: 66009532 subRun: 1 event: 85895 at 15-Sep-2025 18:10:23 CEST
Boundary wire vector sizes: 204, 1422, 880
minwire 0: 1142
minwire 1: 666
minwire 2: 909
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996445, 0.000522028, 0.00303272, 
Output 1: 0.359317, 0.63601, 0.00439234, 0.000280209, 
Output 2: 0.99371, 0.00611285, 0.000169626, 7.99902e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2588!
Begin processing the 196th record. run: 66009532 subRun: 1 event: 85896 at 15-Sep-2025 18:10:29 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 288!
Begin processing the 197th record. run: 66009532 subRun: 1 event: 85897 at 15-Sep-2025 18:10:31 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 919!
Begin processing the 198th record. run: 66009532 subRun: 1 event: 85898 at 15-Sep-2025 18:10:33 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1423!
Begin processing the 199th record. run: 66009532 subRun: 1 event: 85899 at 15-Sep-2025 18:10:35 CEST
Boundary wire vector sizes: 50, 152, 79
minwire 0: 1451
minwire 1: 967
minwire 2: 1636
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.967517, 0.0011899, 0.0312928, 
Output 1: 0.880128, 0.116444, 0.00323562, 0.00019189, 
Output 2: 0.917871, 0.0805212, 0.00155125, 5.62744e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 847!
Begin processing the 200th record. run: 66009532 subRun: 1 event: 85900 at 15-Sep-2025 18:10:38 CEST
Boundary wire vector sizes: 69, 59, 71
minwire 0: 824
minwire 1: 1305
minwire 2: 933
Used alternate method to get min and max wires due to vertex determination failure: 64, 363
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 133, 432
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 8, 307
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00277046, 0.00196306, 0.995267, 
Output 1: 0.748342, 0.250067, 0.00147614, 0.00011547, 
Output 2: 0.998868, 0.00109071, 3.82012e-05, 3.23603e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1199!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3b/5c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_1019_20231204T073824Z_gen_g4_detsim_hitreco__20240509T202119Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_858_20231203T235058Z_gen_g4_detsim_hitreco__20240509T211514Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bb/7c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_795_20231202T112635Z_gen_g4_detsim_hitreco__20240507T211031Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6d/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_65_20231207T022350Z_gen_g4_detsim_hitreco__20240510T042353Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e7/9d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_703_20231208T033513Z_gen_g4_detsim_hitreco__20240510T064741Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/83/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_315_20231205T024901Z_gen_g4_detsim_hitreco__20240509T222848Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/f6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_944_20231203T084526Z_gen_g4_detsim_hitreco__20240509T211746Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/da/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_750_20231208T063316Z_gen_g4_detsim_hitreco__20240510T051813Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/04/fa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_361_20231208T092527Z_gen_g4_detsim_hitreco__20240510T063619Z_reco2.root
.:
total 10596
-rw-r--r--. 1 dune012 dune 4042589 Sep 15 18:10 flatcaf.root
-rw-r--r--. 1 dune012 dune 3282377 Sep 15 18:10 caf.root
-rw-r--r--. 1 dune012 dune 3282377 Sep 15 18:10 caf_fd_hd_atmo_2501_20250915T154956Z.root
-rw-r--r--. 1 dune012 dune  137668 Sep 15 18:10 caf_20250915T154956Z.log
-rw-r--r--. 1 dune012 dune   55842 Sep 15 18:10 jobscript.log
-rw-r--r--. 1 dune012 dune    1864 Sep 15 18:10 caf_20250915T154956Z.file
-rw-r--r--. 1 dune012 dune    1864 Sep 15 18:10 caf_20250915T154956Z.pfns
-rw-r--r--. 1 dune012 dune    1864 Sep 15 17:49 file.list
-rw-r--r--. 1 dune012 dune    1864 Sep 15 18:10 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune    1374 Sep 15 17:49 all-input-dids.txt
-rw-r--r--. 1 dune012 dune    1374 Sep 15 18:10 caf_20250915T154956Z.did
-rw-r--r--. 1 dune012 dune    1374 Sep 15 17:49 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 15 17:50 debugprod.log