Jobsub ID 46713.29@dunegpschedd01.fnal.gov

Jobsub ID	46713.29@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 16:16:51
Site	UK_RAL-Tier1
Entry	LIGO_UK_RAL_arc_ce03
Last heartbeat	2025-09-15 17:27:39
From worker node	Hostname	dune001-8668536.0-lcg2701.gridpp.rl.ac.uk
	cpuinfo	AMD EPYC 9654 96-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 16:17:53
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_565_20231208T083543Z_gen_g4_detsim_hitreco__20240510T055341Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_152_20231204T120346Z_gen_g4_detsim_hitreco__20240509T215822Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_693_20231203T052003Z_gen_g4_detsim_hitreco__20240508T050629Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_627_20231203T120428Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_496_20231207T091040Z_gen_g4_detsim_hitreco__20240510T051813Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_134_20231207T082751Z_gen_g4_detsim_hitreco__20240510T033811Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_689_20231203T120629Z_gen_g4_detsim_hitreco__20240509T200042Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_502_20231207T042239Z_gen_g4_detsim_hitreco__20240510T055633Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 17:27:39
Saved logs	justin-logs:46713.29-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

500    
reco:energyrecnumumcs:EnergyReco                      0.000533062    0.0344296      3.45083      0.0028369     0.193373        500    
reco:energyrecnumumcsllhd:EnergyReco                  0.000335716    0.0335206      3.48014     0.00281746     0.194582        500    
reco:opdec:Deconvolution                               0.084082      0.265306      0.554234      0.263741      0.0744148       500    
reco:ophitspe:OpHitFinderDeco                          0.241924      0.276506      0.482267       0.25373      0.0475044       500    
reco:opflash:OpFlashFinder                            0.000154663   0.00151302     0.0117431    0.00134995    0.000970162      500    
reco:opslicer:OpSlicer                                3.0937e-05     0.231908      0.875514      0.185523      0.187745        500    
reco:rns:RandomNumberSaver                             2.027e-05    5.52898e-05   0.000476647   5.3691e-05    2.60957e-05      500    
reco:anglereconue:NuAngularReco                       0.000181834   0.00128176     0.114399     0.000450168   0.00553532       500    
reco:anglereconumu:NuAngularReco                      5.3211e-05    0.000978695    0.117018     0.000174332   0.00557359       500    
reco:anglereconuepfps:NuAngularReco                   9.3801e-05    0.00109968     0.106471     0.00030234    0.00518039       500    
reco:anglereconumupfps:NuAngularReco                   6.636e-05    0.00104217     0.106604     0.000232915   0.00520811       500    
reco:anglerecohits:NuAngularReco                      6.3646e-05    0.00163659     0.116535     0.000623028   0.00579511       500    
[art]:TriggerResults:TriggerResultInserter             8.613e-06    1.38012e-05   4.9795e-05    1.3055e-05    3.73206e-06      500    
end_path:cafmaker:CAFMaker                            0.00147467     0.0182772      1.44644     0.00680113     0.0703142       500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4041 MB
  Peak resident set size usage (VmHWM): 2245.06 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 17:05:59 UTC ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_134_20231207T082751Z_gen_g4_detsim_hitreco__20240510T033811Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2378!
Begin processing the 495th record. run: 74550731 subRun: 1 event: 49695 at 15-Sep-2025 17:05:37 UTC
Boundary wire vector sizes: 111, 334, 201
minwire 0: 1848
minwire 1: 640
minwire 2: 1815
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996146, 0.000830619, 0.00302297, 
Output 1: 0.00675612, 0.988725, 0.00440855, 0.000110835, 
Output 2: 0.99572, 0.00419273, 8.56527e-05, 1.75662e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1440!
Begin processing the 496th record. run: 74550731 subRun: 1 event: 49696 at 15-Sep-2025 17:05:41 UTC
Boundary wire vector sizes: 412, 332, 344
minwire 0: 1882
minwire 1: 291
minwire 2: 1766
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.459574, 0.210627, 0.329799, 
Output 1: 0.0210626, 0.27884, 0.577713, 0.122385, 
Output 2: 0.157946, 0.503534, 0.298752, 0.0397686, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1834!
Begin processing the 497th record. run: 74550731 subRun: 1 event: 49697 at 15-Sep-2025 17:05:45 UTC
Boundary wire vector sizes: 44, 38, 27
minwire 0: 294
minwire 1: 2129
minwire 2: 347
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0644965, 0.931185, 0.00431889, 
Output 1: 0.019819, 0.898021, 0.0794855, 0.0026747, 
Output 2: 0.99605, 0.00377511, 0.000168177, 7.19885e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 751!
Begin processing the 498th record. run: 74550731 subRun: 1 event: 49698 at 15-Sep-2025 17:05:48 UTC
Boundary wire vector sizes: 52, 80, 65
minwire 0: 1249
minwire 1: 2321
minwire 2: 749
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.948264, 0.0030253, 0.048711, 
Output 1: 0.0200355, 0.925036, 0.0535089, 0.00141996, 
Output 2: 0.95378, 0.0451958, 0.00100692, 1.75176e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1730!
Begin processing the 499th record. run: 74550731 subRun: 1 event: 49699 at 15-Sep-2025 17:05:52 UTC
Boundary wire vector sizes: 55, 46, 48
minwire 0: 813
minwire 1: 2301
minwire 2: 300
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0034912, 0.00195568, 0.994553, 
Output 1: 0.168725, 0.826882, 0.00424778, 0.000144794, 
Output 2: 0.99788, 0.00203012, 8.58975e-05, 4.36919e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1261!
Begin processing the 500th record. run: 74550731 subRun: 1 event: 49700 at 15-Sep-2025 17:05:55 UTC
Boundary wire vector sizes: 166, 81, 139
minwire 0: 2151
minwire 1: 258
minwire 2: 2537
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.986002, 0.00231768, 0.0116802, 
Output 1: 0.165206, 0.821782, 0.0126604, 0.000351409, 
Output 2: 0.988012, 0.0117826, 0.00020159, 4.00809e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 768!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ef/3e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_565_20231208T083543Z_gen_g4_detsim_hitreco__20240510T055341Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/25/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_152_20231204T120346Z_gen_g4_detsim_hitreco__20240509T215822Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d7/64/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_693_20231203T052003Z_gen_g4_detsim_hitreco__20240508T050629Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/04/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_627_20231203T120428Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/4b/a6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_496_20231207T091040Z_gen_g4_detsim_hitreco__20240510T051813Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_134_20231207T082751Z_gen_g4_detsim_hitreco__20240510T033811Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/60/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_689_20231203T120629Z_gen_g4_detsim_hitreco__20240509T200042Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b4/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_502_20231207T042239Z_gen_g4_detsim_hitreco__20240510T055633Z_reco2.root
.:
total 21108
-rw-r--r-- 1 dune001 dune 7983588 Sep 15 17:05 flatcaf.root
-rw-r--r-- 1 dune001 dune 6574826 Sep 15 17:05 caf.root
-rw-r--r-- 1 dune001 dune 6574826 Sep 15 17:05 caf_fd_hd_atmo_2501_20250915T161804Z.root
-rw-r--r-- 1 dune001 dune  307176 Sep 15 17:05 caf_20250915T161804Z.log
-rw-r--r-- 1 dune001 dune  118445 Sep 15 17:05 jobscript.log
-rw-r--r-- 1 dune001 dune    1799 Sep 15 17:05 caf_20250915T161804Z.file
-rw-r--r-- 1 dune001 dune    1799 Sep 15 17:05 caf_20250915T161804Z.pfns
-rw-r--r-- 1 dune001 dune    1799 Sep 15 16:18 file.list
-rw-r--r-- 1 dune001 dune    1799 Sep 15 17:05 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune    1236 Sep 15 16:18 all-input-dids.txt
-rw-r--r-- 1 dune001 dune    1236 Sep 15 17:05 caf_20250915T161804Z.did
-rw-r--r-- 1 dune001 dune    1236 Sep 15 16:18 did.list
-rw-r--r-- 1 dune001 dune       0 Sep 15 16:18 debugprod.log