Jobsub ID 47404.112@dunegpschedd01.fnal.gov

Jobsub ID	47404.112@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 07:52:57
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2025-09-17 08:19:48
From worker node	Hostname	heplnc168.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-17 07:53:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_296_20231120T133410Z_gen_g4_detsim_hitreco__20240503T061703Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_600_20231203T151549Z_gen_g4_detsim_hitreco__20240508T054521Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_989_20231204T140428Z_gen_g4_detsim_hitreco__20240509T204319Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_368_20231120T134927Z_gen_g4_detsim_hitreco__20240503T062923Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-17 08:19:48
Saved logs	justin-logs:47404.112-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

8.509e-05     0.0146429     0.658642     0.00103098     0.0529568       400    
reco:pandorapid:Chi2ParticleID                        3.2571e-05    0.00013915    0.00155777    0.000108839   0.000114814      400    
reco:cvnmap:CVNMapper                                 1.9727e-05    0.00859147     0.0434235    0.00730315    0.00742369       400    
reco:cvneva:CVNEvaluator                               1.548e-05     0.293901       1.86874      0.366762      0.191546        400    
reco:energyrecnumu:EnergyReco                         0.000257473    0.0584712      1.19073      0.0105789     0.118708        400    
reco:energyrecnue:EnergyReco                          0.000120586   0.00198829     0.0440775    0.000588003   0.00410662       400    
reco:energyrecnc:EnergyReco                           0.00152787     0.078615      0.875047      0.0151165     0.113839        400    
reco:energyrecnumurange:EnergyReco                    0.000185678    0.0291703      1.40394     0.00283968     0.103044        400    
reco:energyrecnumumcs:EnergyReco                      0.000525075    0.0274637      1.04794     0.00343507     0.0885574       400    
reco:energyrecnumumcsllhd:EnergyReco                  0.000429506    0.0265546     0.914258      0.0033842     0.0850406       400    
reco:opdec:Deconvolution                              0.00650276      0.21142      0.682646      0.197489       0.10286        400    
reco:ophitspe:OpHitFinderDeco                         0.00296036     0.104524      0.916404      0.0401835      0.11142        400    
reco:opflash:OpFlashFinder                            0.000148318   0.00127761    0.00358851     0.0011813    0.000625214      400    
reco:opslicer:OpSlicer                                4.5806e-05     0.223681      0.807432      0.180887      0.180907        400    
reco:rns:RandomNumberSaver                            1.7834e-05    4.22217e-05   0.000414297   3.7866e-05    2.66881e-05      400    
reco:anglereconue:NuAngularReco                       0.000110657   0.00111298     0.0258348    0.000386164   0.00227663       400    
reco:anglereconumu:NuAngularReco                      7.1655e-05    0.000937583    0.0249458    0.000207084   0.00224652       400    
reco:anglereconuepfps:NuAngularReco                   0.000106329   0.00102407     0.0251307    0.000309741    0.002231        400    
reco:anglereconumupfps:NuAngularReco                  8.9307e-05    0.000971623    0.0252073    0.000274746   0.00219673       400    
reco:anglerecohits:NuAngularReco                      0.000114806   0.00152562     0.029101     0.000658645    0.0026082       400    
[art]:TriggerResults:TriggerResultInserter             8.887e-06    1.30899e-05   5.4682e-05    1.1767e-05    5.56313e-06      400    
end_path:cafmaker:CAFMaker                            0.00111351     0.0142654     0.324732     0.00509378     0.0286769       400    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3654.9 MB
  Peak resident set size usage (VmHWM): 2052.55 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 09:17:45 BST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/25/ba/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_368_20231120T134927Z_gen_g4_detsim_hitreco__20240503T062923Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 74, 147, 110
minwire 0: 2789
minwire 1: 505
minwire 2: 2596
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.230937, 0.00471196, 0.764351, 
Output 1: 0.903125, 0.0623152, 0.0226524, 0.0119074, 
Output 2: 0.216234, 0.773469, 0.0100612, 0.000235171, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1206!
Begin processing the 395th record. run: 74517882 subRun: 1 event: 98995 at 17-Sep-2025 09:17:25 BST
Boundary wire vector sizes: 191, 329, 291
minwire 0: 330
minwire 1: 2038
minwire 2: 217
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000271018, 0.998459, 0.00127047, 
Output 1: 0.00296438, 0.993073, 0.00383314, 0.000129477, 
Output 2: 0.998306, 0.00164866, 4.37438e-05, 1.45164e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 951!
Begin processing the 396th record. run: 74517882 subRun: 1 event: 98996 at 17-Sep-2025 09:17:28 BST
Boundary wire vector sizes: 179, 282, 190
minwire 0: 813
minwire 1: 1678
minwire 2: 330
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0554754, 0.0141132, 0.930411, 
Output 1: 0.0509267, 0.374082, 0.512559, 0.0624325, 
Output 2: 0.0593791, 0.761977, 0.173459, 0.00518468, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2660!
Begin processing the 397th record. run: 74517882 subRun: 1 event: 98997 at 17-Sep-2025 09:17:32 BST
Boundary wire vector sizes: 322, 238, 250
minwire 0: 2023
minwire 1: 54
minwire 2: 1930
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0195694, 0.127016, 0.853414, 
Output 1: 0.596203, 0.388817, 0.013427, 0.00155311, 
Output 2: 0.613686, 0.355648, 0.0283798, 0.00228622, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1015!
Begin processing the 398th record. run: 74517882 subRun: 1 event: 98998 at 17-Sep-2025 09:17:35 BST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 571!
Begin processing the 399th record. run: 74517882 subRun: 1 event: 98999 at 17-Sep-2025 09:17:37 BST
Boundary wire vector sizes: 173, 138, 83
minwire 0: 570
minwire 1: 2805
minwire 2: 27
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000748908, 0.989719, 0.00953184, 
Output 1: 0.0149566, 0.975638, 0.00914205, 0.000263186, 
Output 2: 0.996994, 0.00291273, 8.98942e-05, 3.44277e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1159!
Begin processing the 400th record. run: 74517882 subRun: 1 event: 99000 at 17-Sep-2025 09:17:41 BST
Boundary wire vector sizes: 473, 510, 176
minwire 0: 893
minwire 1: 1394
minwire 2: 1028
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.989869, 0.000718298, 0.00941327, 
Output 1: 0.306693, 0.689431, 0.00365376, 0.000222855, 
Output 2: 0.495836, 0.498298, 0.00581208, 5.38594e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1616!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/37/92/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_296_20231120T133410Z_gen_g4_detsim_hitreco__20240503T061703Z_reco2.root
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/9b/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_600_20231203T151549Z_gen_g4_detsim_hitreco__20240508T054521Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c0/47/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_989_20231204T140428Z_gen_g4_detsim_hitreco__20240509T204319Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/25/ba/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_368_20231120T134927Z_gen_g4_detsim_hitreco__20240503T062923Z_reco2.root
.:
total 18888
-rw-r--r-- 1 pltdune04 pltdune 7142371 Sep 17 09:17 flatcaf.root
-rw-r--r-- 1 pltdune04 pltdune 5891365 Sep 17 09:17 caf.root
-rw-r--r-- 1 pltdune04 pltdune 5891365 Sep 17 09:17 caf_fd_hd_atmo_2501_20250917T075343Z.root
-rw-r--r-- 1 pltdune04 pltdune  247576 Sep 17 09:17 caf_20250917T075343Z.log
-rw-r--r-- 1 pltdune04 pltdune   85303 Sep 17 09:17 jobscript.log
-rw-r--r-- 1 pltdune04 pltdune     771 Sep 17 09:17 caf_20250917T075343Z.file
-rw-r--r-- 1 pltdune04 pltdune     771 Sep 17 09:17 caf_20250917T075343Z.pfns
-rw-r--r-- 1 pltdune04 pltdune     771 Sep 17 08:53 file.list
-rw-r--r-- 1 pltdune04 pltdune     771 Sep 17 09:17 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune04 pltdune     549 Sep 17 08:53 all-input-dids.txt
-rw-r--r-- 1 pltdune04 pltdune     549 Sep 17 09:17 caf_20250917T075343Z.did
-rw-r--r-- 1 pltdune04 pltdune     549 Sep 17 08:53 did.list
-rw-r--r-- 1 pltdune04 pltdune       0 Sep 17 08:53 debugprod.log