Jobsub ID 40095.40@dunegpschedd02.fnal.gov

Jobsub ID	40095.40@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 15:29:34
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2025-09-15 15:37:58
From worker node	Hostname	wn-sate-048.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 15:29:51
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_503_20231202T060650Z_gen_g4_detsim_hitreco__20240507T203718Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_169_20231201T102953Z_gen_g4_detsim_hitreco__20240507T202934Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_915_20231204T100526Z_gen_g4_detsim_hitreco__20240509T202018Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_65_20231204T055354Z_gen_g4_detsim_hitreco__20240509T215119Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_549_20231202T162406Z_gen_g4_detsim_hitreco__20240508T044637Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_521_20231208T083421Z_gen_g4_detsim_hitreco__20240510T054955Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_858_20231203T235058Z_gen_g4_detsim_hitreco__20240509T211514Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_65_20231207T022350Z_gen_g4_detsim_hitreco__20240510T042353Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50256571_490_20231121T152330Z_gen_g4_detsim_hitreco__20240503T054626Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 15:37:58
Saved logs	justin-logs:40095.40-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

955     0.00330287     0.219777        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000488237    0.0395294      2.38725     0.00328527     0.238231        100    
reco:opdec:Deconvolution                              0.00909113     0.181716      0.340508      0.181545      0.0672934       100    
reco:ophitspe:OpHitFinderDeco                          0.007482      0.0174179     0.057543      0.0137361    0.00950228       100    
reco:opflash:OpFlashFinder                            0.000173235   0.00121727    0.00404412    0.00118063    0.000607269      100    
reco:opslicer:OpSlicer                                0.000152937    0.244377      0.654349      0.209607      0.182507        100    
reco:rns:RandomNumberSaver                            1.9046e-05    2.89433e-05   0.000317746   2.5809e-05    2.91082e-05      100    
reco:anglereconue:NuAngularReco                       0.000155462   0.00128639     0.0325894    0.000383895   0.00361127       100    
reco:anglereconumu:NuAngularReco                      9.4708e-05    0.00116481     0.0320993    0.000264356   0.00357868       100    
reco:anglereconuepfps:NuAngularReco                   0.000136346   0.00132505     0.0323506    0.000360201   0.00364195       100    
reco:anglereconumupfps:NuAngularReco                  0.000135294   0.00124804     0.0324988    0.00033834    0.00362229       100    
reco:anglerecohits:NuAngularReco                      0.000367429   0.00180568     0.0364177    0.000721109   0.00411482       100    
[art]:TriggerResults:TriggerResultInserter             1.094e-05    1.28607e-05   5.8861e-05    1.21075e-05   4.73027e-06      100    
end_path:cafmaker:CAFMaker                            0.00109915     0.0204483     0.498224     0.00412073     0.0637235       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3650.26 MB
  Peak resident set size usage (VmHWM): 1985.91 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 17:37:30 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/33/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_169_20231201T102953Z_gen_g4_detsim_hitreco__20240507T202934Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00118673, 0.974701, 0.0241125, 
Output 1: 0.930574, 0.0677185, 0.0015971, 0.000109944, 
Output 2: 0.965453, 0.0335319, 0.000949009, 6.64096e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2461!
Begin processing the 94th record. run: 6398928 subRun: 1 event: 50394 at 15-Sep-2025 17:37:08 CEST
Boundary wire vector sizes: 73, 102, 93
minwire 0: 2190
minwire 1: 581
minwire 2: 2284
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.351327, 0.0265345, 0.622139, 
Output 1: 0.0105731, 0.0611056, 0.26702, 0.661301, 
Output 2: 0.959026, 0.0391507, 0.00169019, 0.000133549, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1341!
Begin processing the 95th record. run: 6398928 subRun: 1 event: 50395 at 15-Sep-2025 17:37:10 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 655!
Begin processing the 96th record. run: 6398928 subRun: 1 event: 50396 at 15-Sep-2025 17:37:12 CEST
Boundary wire vector sizes: 737, 796, 395
minwire 0: 1496
minwire 1: 999
minwire 2: 1324
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00156118, 0.973957, 0.0244821, 
Output 1: 0.0834897, 0.581757, 0.308061, 0.0266918, 
Output 2: 0.803147, 0.18087, 0.0152024, 0.000780234, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2548!
Begin processing the 97th record. run: 6398928 subRun: 1 event: 50397 at 15-Sep-2025 17:37:16 CEST
Boundary wire vector sizes: 287, 370, 186
minwire 0: 2134
minwire 1: 1143
minwire 2: 1951
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000320817, 0.997601, 0.00207822, 
Output 1: 0.871382, 0.127507, 0.000992034, 0.000119254, 
Output 2: 0.998475, 0.00149468, 2.82675e-05, 2.36182e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2078!
Begin processing the 98th record. run: 6398928 subRun: 1 event: 50398 at 15-Sep-2025 17:37:19 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 234!
Begin processing the 99th record. run: 6398928 subRun: 1 event: 50399 at 15-Sep-2025 17:37:21 CEST
Boundary wire vector sizes: 2249, 3158, 2502
minwire 0: 819
minwire 1: 36
minwire 2: 356
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.575901, 0.133835, 0.290265, 
Output 1: 0.166726, 0.31141, 0.249163, 0.272701, 
Output 2: 0.0129481, 0.0355551, 0.169676, 0.781821, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2419!
Begin processing the 100th record. run: 6398928 subRun: 1 event: 50400 at 15-Sep-2025 17:37:27 CEST
Boundary wire vector sizes: 69, 57, 70
minwire 0: 1755
minwire 1: 1279
minwire 2: 1708
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.993402, 0.00187557, 0.00472231, 
Output 1: 0.025042, 0.972194, 0.0026605, 0.00010394, 
Output 2: 0.994966, 0.00487414, 0.000155107, 4.93323e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1365!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/f3/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_503_20231202T060650Z_gen_g4_detsim_hitreco__20240507T203718Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/33/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_169_20231201T102953Z_gen_g4_detsim_hitreco__20240507T202934Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a8/e7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_915_20231204T100526Z_gen_g4_detsim_hitreco__20240509T202018Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7c/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_65_20231204T055354Z_gen_g4_detsim_hitreco__20240509T215119Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d8/2a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_549_20231202T162406Z_gen_g4_detsim_hitreco__20240508T044637Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/41/f8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_521_20231208T083421Z_gen_g4_detsim_hitreco__20240510T054955Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_858_20231203T235058Z_gen_g4_detsim_hitreco__20240509T211514Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6d/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_65_20231207T022350Z_gen_g4_detsim_hitreco__20240510T042353Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/32/58/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50256571_490_20231121T152330Z_gen_g4_detsim_hitreco__20240503T054626Z_reco2.root
.:
total 5872
-rw-r--r--. 1 dune012 dune 2289359 Sep 15 17:37 flatcaf.root
-rw-r--r--. 1 dune012 dune 1781370 Sep 15 17:37 caf.root
-rw-r--r--. 1 dune012 dune 1781370 Sep 15 17:37 caf_fd_hd_atmo_2501_20250915T153000Z.root
-rw-r--r--. 1 dune012 dune   70724 Sep 15 17:37 caf_20250915T153000Z.log
-rw-r--r--. 1 dune012 dune   37465 Sep 15 17:37 jobscript.log
-rw-r--r--. 1 dune012 dune    1895 Sep 15 17:37 caf_20250915T153000Z.file
-rw-r--r--. 1 dune012 dune    1895 Sep 15 17:37 caf_20250915T153000Z.pfns
-rw-r--r--. 1 dune012 dune    1895 Sep 15 17:30 file.list
-rw-r--r--. 1 dune012 dune    1895 Sep 15 17:37 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune    1373 Sep 15 17:30 all-input-dids.txt
-rw-r--r--. 1 dune012 dune    1373 Sep 15 17:37 caf_20250915T153000Z.did
-rw-r--r--. 1 dune012 dune    1373 Sep 15 17:30 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 15 17:30 debugprod.log