Jobsub ID 40118.6@dunegpschedd02.fnal.gov

Jobsub ID	40118.6@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 16:34:52
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2025-09-15 17:21:25
From worker node	Hostname	wn-snel-025.farm.nikhef.nl
	cpuinfo	AMD EPYC 7H12 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 16:35:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_23_20231118T173307Z_gen_g4_detsim_hitreco__20240503T052009Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_478_20231208T090227Z_gen_g4_detsim_hitreco__20240510T062449Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_687_20231207T102432Z_gen_g4_detsim_hitreco__20240510T051943Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_668_20231201T223602Z_gen_g4_detsim_hitreco__20240507T204428Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_744_20231207T090004Z_gen_g4_detsim_hitreco__20240510T035929Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_208_20231203T031536Z_gen_g4_detsim_hitreco__20240508T042747Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_119_20231208T004521Z_gen_g4_detsim_hitreco__20240510T074023Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_402_20231203T035803Z_gen_g4_detsim_hitreco__20240508T073822Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 17:21:25
Saved logs	justin-logs:40118.6-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000195266    0.0240314      1.44172     0.00232282     0.108923        600    
reco:energyrecnumumcs:EnergyReco                      0.000482777    0.0245689      1.43679     0.00255963      0.11007        600    
reco:energyrecnumumcsllhd:EnergyReco                  0.000352061    0.0242714      1.44871     0.00251236     0.109157        600    
reco:opdec:Deconvolution                               0.0416602     0.215456      0.660558      0.214362      0.0723769       600    
reco:ophitspe:OpHitFinderDeco                          0.0798967     0.125431      0.154492      0.132828      0.0184155       600    
reco:opflash:OpFlashFinder                            0.00020783    0.00119239     0.0129084    0.00112445    0.00070183       600    
reco:opslicer:OpSlicer                                5.4403e-05     0.216914       1.13014      0.172682      0.171843        600    
reco:rns:RandomNumberSaver                            1.8314e-05    2.92659e-05   0.000368903   2.9265e-05    1.42889e-05      600    
reco:anglereconue:NuAngularReco                       0.000157576   0.00109522     0.0647478    0.000355412   0.00362867       600    
reco:anglereconumu:NuAngularReco                      7.5402e-05    0.000930133    0.0673925    0.000182287   0.00372639       600    
reco:anglereconuepfps:NuAngularReco                   0.000118934   0.00101963     0.067117     0.00027657    0.00370244       600    
reco:anglereconumupfps:NuAngularReco                  0.000108143   0.000979975    0.0661532    0.000236459   0.00368696       600    
reco:anglerecohits:NuAngularReco                      0.000102773   0.00149166     0.0727609    0.000623491   0.00413145       600    
[art]:TriggerResults:TriggerResultInserter             8.386e-06    1.22145e-05   4.6207e-05    1.2143e-05    2.2863e-06       600    
end_path:cafmaker:CAFMaker                            0.00128384     0.0143623     0.874661     0.00400215     0.0487895       600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3814.74 MB
  Peak resident set size usage (VmHWM): 2157.32 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 19:21:05 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8d/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_668_20231201T223602Z_gen_g4_detsim_hitreco__20240507T204428Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2095!
Begin processing the 593rd record. run: 6207207 subRun: 1 event: 31193 at 15-Sep-2025 19:20:43 CEST
Boundary wire vector sizes: 717, 605, 771
minwire 0: 1738
minwire 1: 181
minwire 2: 1988
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995817, 0.000443187, 0.00373964, 
Output 1: 0.472444, 0.377461, 0.105597, 0.0444986, 
Output 2: 0.734523, 0.227148, 0.0336128, 0.00471633, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1360!
Begin processing the 594th record. run: 6207207 subRun: 1 event: 31194 at 15-Sep-2025 19:20:46 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1185!
Begin processing the 595th record. run: 6207207 subRun: 1 event: 31195 at 15-Sep-2025 19:20:49 CEST
Boundary wire vector sizes: 37, 36, 28
minwire 0: 1425
minwire 1: 1938
minwire 2: 1089
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00642956, 0.00280224, 0.990768, 
Output 1: 0.818939, 0.178938, 0.00194787, 0.000175104, 
Output 2: 0.997951, 0.00199464, 5.05272e-05, 3.91744e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1311!
Begin processing the 596th record. run: 6207207 subRun: 1 event: 31196 at 15-Sep-2025 19:20:51 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 643!
Begin processing the 597th record. run: 6207207 subRun: 1 event: 31197 at 15-Sep-2025 19:20:54 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 650!
Begin processing the 598th record. run: 6207207 subRun: 1 event: 31198 at 15-Sep-2025 19:20:56 CEST
Boundary wire vector sizes: 106, 176, 130
minwire 0: 610
minwire 1: 2946
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992687, 0.000599144, 0.00671404, 
Output 1: 0.00300699, 0.993955, 0.00296745, 7.04118e-05, 
Output 2: 0.991914, 0.0079141, 0.000168571, 3.42869e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1145!
Begin processing the 599th record. run: 6207207 subRun: 1 event: 31199 at 15-Sep-2025 19:20:59 CEST
Boundary wire vector sizes: 106, 126, 79
minwire 0: 1824
minwire 1: 1762
minwire 2: 1407
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0193565, 0.288538, 0.692106, 
Output 1: 0.0810216, 0.885874, 0.031638, 0.00146631, 
Output 2: 0.91902, 0.078264, 0.00262894, 8.748e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2047!
Begin processing the 600th record. run: 6207207 subRun: 1 event: 31200 at 15-Sep-2025 19:21:02 CEST
Boundary wire vector sizes: 176, 274, 207
minwire 0: 582
minwire 1: 2274
minwire 2: 288
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996618, 0.000447147, 0.00293447, 
Output 1: 0.0210204, 0.944076, 0.034035, 0.000868264, 
Output 2: 0.996516, 0.00339393, 8.80572e-05, 1.73217e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1302!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e4/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_23_20231118T173307Z_gen_g4_detsim_hitreco__20240503T052009Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ce/98/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_478_20231208T090227Z_gen_g4_detsim_hitreco__20240510T062449Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ff/fb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_687_20231207T102432Z_gen_g4_detsim_hitreco__20240510T051943Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/1d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_155_20231120T132906Z_gen_g4_detsim_hitreco__20240503T061912Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8d/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_668_20231201T223602Z_gen_g4_detsim_hitreco__20240507T204428Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/83/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50610358_744_20231207T090004Z_gen_g4_detsim_hitreco__20240510T035929Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f8/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_208_20231203T031536Z_gen_g4_detsim_hitreco__20240508T042747Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/52/ee/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_119_20231208T004521Z_gen_g4_detsim_hitreco__20240510T074023Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/24/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_402_20231203T035803Z_gen_g4_detsim_hitreco__20240508T073822Z_reco2.root
.:
total 25160
-rw-r--r--. 1 dune012 dune 9507877 Sep 15 19:21 flatcaf.root
-rw-r--r--. 1 dune012 dune 7854884 Sep 15 19:21 caf.root
-rw-r--r--. 1 dune012 dune 7854884 Sep 15 19:21 caf_fd_hd_atmo_2501_20250915T163542Z.root
-rw-r--r--. 1 dune012 dune  370479 Sep 15 19:21 caf_20250915T163542Z.log
-rw-r--r--. 1 dune012 dune  125396 Sep 15 19:21 jobscript.log
-rw-r--r--. 1 dune012 dune    1693 Sep 15 19:21 caf_20250915T163542Z.file
-rw-r--r--. 1 dune012 dune    1693 Sep 15 19:21 caf_20250915T163542Z.pfns
-rw-r--r--. 1 dune012 dune    1693 Sep 15 18:35 file.list
-rw-r--r--. 1 dune012 dune    1693 Sep 15 19:21 justin-processed-pfns.txt
-rw-r--r--. 1 dune012 dune    1238 Sep 15 18:35 all-input-dids.txt
-rw-r--r--. 1 dune012 dune    1238 Sep 15 19:21 caf_20250915T163542Z.did
-rw-r--r--. 1 dune012 dune    1238 Sep 15 18:35 did.list
-rw-r--r--. 1 dune012 dune       0 Sep 15 18:35 debugprod.log