Jobsub ID 40051.31@dunegpschedd02.fnal.gov

Jobsub ID	40051.31@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 14:31:47
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-15 14:44:06
From worker node	Hostname	wn-lot-029.farm.nikhef.nl
	cpuinfo	AMD EPYC 7702P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 14:33:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_40_20231202T160523Z_gen_g4_detsim_hitreco__20240508T041702Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_215_20231203T093005Z_gen_g4_detsim_hitreco__20240508T080317Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_474_20231203T124714Z_gen_g4_detsim_hitreco__20240508T050911Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_30_20231202T004132Z_gen_g4_detsim_hitreco__20240508T053156Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_90_20231202T075132Z_gen_g4_detsim_hitreco__20240507T210613Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_985_20231202T140222Z_gen_g4_detsim_hitreco__20240507T223630Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_393_20231207T010813Z_gen_g4_detsim_hitreco__20240510T040217Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_376_20231202T054342Z_gen_g4_detsim_hitreco__20240508T034043Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_170_20231207T022517Z_gen_g4_detsim_hitreco__20240510T033409Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 14:44:06
Saved logs	justin-logs:40051.31-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3    0.0416636     0.934454     0.00261719      0.13519        100    
reco:energyrecnumumcs:EnergyReco                      0.000667065    0.0435894     0.925457     0.00349965     0.138631        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000518628    0.0420664     0.927573     0.00344692     0.133354        100    
reco:opdec:Deconvolution                               0.0517677     0.260968      0.517415      0.262276      0.0939742       100    
reco:ophitspe:OpHitFinderDeco                          0.130262      0.136034      0.150725       0.13569      0.0028992       100    
reco:opflash:OpFlashFinder                            0.000175578   0.00125066    0.00352484    0.00120729    0.000611061      100    
reco:opslicer:OpSlicer                                0.00015005      0.27988      0.814887      0.244454      0.222565        100    
reco:rns:RandomNumberSaver                            1.8675e-05    2.73845e-05   0.000290562   2.49415e-05   2.65768e-05      100    
reco:anglereconue:NuAngularReco                       0.000152464   0.00195604     0.0424071    0.000387428   0.00641983       100    
reco:anglereconumu:NuAngularReco                      7.9148e-05    0.00187528     0.0435912    0.000261949   0.00660413       100    
reco:anglereconuepfps:NuAngularReco                   0.000122399    0.0019497      0.0432      0.000349046   0.00651421       100    
reco:anglereconumupfps:NuAngularReco                  0.000115115   0.00191999     0.0433288    0.000316391    0.006567        100    
reco:anglerecohits:NuAngularReco                      8.7603e-05    0.00256738     0.0483296    0.000770363   0.00721724       100    
[art]:TriggerResults:TriggerResultInserter             9.858e-06    1.14474e-05   4.3241e-05     1.096e-05    3.2776e-06       100    
end_path:cafmaker:CAFMaker                            0.00125347     0.0255884     0.542509     0.00439589     0.0793849       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3653.9 MB
  Peak resident set size usage (VmHWM): 1988.38 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 16:43:30 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.000434655, 0.997334, 0.00223178, 
Output 1: 0.964927, 0.0345695, 0.000292657, 0.000210937, 
Output 2: 0.999022, 0.000934214, 3.2443e-05, 1.15998e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1725!
Begin processing the 93rd record. run: 66004350 subRun: 1 event: 4093 at 15-Sep-2025 16:42:29 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 845!
Begin processing the 94th record. run: 66004350 subRun: 1 event: 4094 at 15-Sep-2025 16:42:31 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 805!
Begin processing the 95th record. run: 66004350 subRun: 1 event: 4095 at 15-Sep-2025 16:42:33 CEST
Boundary wire vector sizes: 212, 222, 209
minwire 0: 1752
minwire 1: 1049
minwire 2: 1814
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00152059, 0.993839, 0.00464073, 
Output 1: 0.014726, 0.249518, 0.658242, 0.0775142, 
Output 2: 0.522837, 0.455611, 0.0213196, 0.000232607, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1844!
Begin processing the 96th record. run: 66004350 subRun: 1 event: 4096 at 15-Sep-2025 16:42:36 CEST
Boundary wire vector sizes: 35, 48, 37
minwire 0: 2393
minwire 1: 922
minwire 2: 2301
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0225316, 0.973917, 0.00355183, 
Output 1: 0.0983451, 0.896966, 0.00453655, 0.000151999, 
Output 2: 0.998296, 0.00164094, 6.00765e-05, 3.31931e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 690!
Begin processing the 97th record. run: 66004350 subRun: 1 event: 4097 at 15-Sep-2025 16:42:39 CEST
Boundary wire vector sizes: 6042, 10854, 5359
minwire 0: 1277
minwire 1: 128
minwire 2: 1012
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0351914, 0.727345, 0.237464, 
Output 1: 0.296076, 0.408813, 0.152911, 0.1422, 
Output 2: 0.0229512, 0.0866647, 0.247679, 0.642705, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3416!
Begin processing the 98th record. run: 66004350 subRun: 1 event: 4098 at 15-Sep-2025 16:43:21 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 605!
Begin processing the 99th record. run: 66004350 subRun: 1 event: 4099 at 15-Sep-2025 16:43:23 CEST
Boundary wire vector sizes: 104, 87, 81
minwire 0: 785
minwire 1: 2330
minwire 2: 424
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.027067, 0.513793, 0.45914, 
Output 1: 0.0285309, 0.96729, 0.00407117, 0.000108263, 
Output 2: 0.995253, 0.00457074, 0.000169432, 7.06321e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1503!
Begin processing the 100th record. run: 66004350 subRun: 1 event: 4100 at 15-Sep-2025 16:43:26 CEST
Boundary wire vector sizes: 272, 333, 261
minwire 0: 626
minwire 1: 473
minwire 2: 514
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.872755, 0.0349021, 0.0923424, 
Output 1: 0.0176323, 0.212058, 0.422051, 0.348258, 
Output 2: 0.145772, 0.75343, 0.0978008, 0.00299791, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2202!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6e/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_40_20231202T160523Z_gen_g4_detsim_hitreco__20240508T041702Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5b/83/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_215_20231203T093005Z_gen_g4_detsim_hitreco__20240508T080317Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0e/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_474_20231203T124714Z_gen_g4_detsim_hitreco__20240508T050911Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/57/4b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_30_20231202T004132Z_gen_g4_detsim_hitreco__20240508T053156Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c4/74/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_90_20231202T075132Z_gen_g4_detsim_hitreco__20240507T210613Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/08/9b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_985_20231202T140222Z_gen_g4_detsim_hitreco__20240507T223630Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ad/58/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_393_20231207T010813Z_gen_g4_detsim_hitreco__20240510T040217Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/c2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_376_20231202T054342Z_gen_g4_detsim_hitreco__20240508T034043Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_170_20231207T022517Z_gen_g4_detsim_hitreco__20240510T033409Z_reco2.root
.:
total 5636
-rw-r--r--. 1 dune012 dune 2190047 Sep 15 16:43 flatcaf.root
-rw-r--r--. 1 dune012 dune 1709344 Sep 15 16:43 caf.root
-rw-r--r--. 1 dune012 dune 1709344 Sep 15 16:43 caf_fd_hd_atmo_2501_20250915T143321Z.root
-rw-r--r--. 1 dune012 dune   74449 Sep 15 16:43 caf_20250915T143321Z.log
-rw-r--r--. 1 dune012 dune   34145 Sep 15 16:43 jobscript.log
-rw-r--r--. 1 dune012 dune    1864 Sep 15 16:43 caf_20250915T143321Z.file
-rw-r--r--. 1 dune012 dune    1864 Sep 15 16:43 caf_20250915T143321Z.pfns
-rw-r--r--. 1 dune012 dune    1864 Sep 15 16:33 file.list
-rw-r--r--. 1 dune012 dune    1864 Sep 15 16:43 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune    1374 Sep 15 16:33 all-input-dids.txt
-rw-r--r--. 1 dune012 dune    1374 Sep 15 16:43 caf_20250915T143321Z.did
-rw-r--r--. 1 dune012 dune    1374 Sep 15 16:33 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 15 16:33 debugprod.log