Jobsub ID 46603.99@dunegpschedd01.fnal.gov

Jobsub ID	46603.99@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:31:41
Site	UK_Bristol
Entry	CMSHTPC_T2_UK_SGrid_Bristol_lcgce02
Last heartbeat	2025-09-15 13:13:14
From worker node	Hostname	hd85.dice.priv
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	259200 (72 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:33:29
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_157_20231207T123809Z_gen_g4_detsim_hitreco__20240510T042104Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_1014_20231203T154815Z_gen_g4_detsim_hitreco__20240508T072052Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_487_20231208T100110Z_gen_g4_detsim_hitreco__20240510T065017Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_226_20231202T112332Z_gen_g4_detsim_hitreco__20240507T215248Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_351_20231207T134706Z_gen_g4_detsim_hitreco__20240510T062951Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_316_20231207T083332Z_gen_g4_detsim_hitreco__20240510T050015Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_71_20231203T141327Z_gen_g4_detsim_hitreco__20240508T055304Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_758_20231208T072001Z_gen_g4_detsim_hitreco__20240510T060510Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_665_20231207T080916Z_gen_g4_detsim_hitreco__20240510T063925Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_216_20231207T035919Z_gen_g4_detsim_hitreco__20240510T043609Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 13:13:14
Saved logs	justin-logs:46603.99-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000702022    0.0599334     0.949904     0.00446125     0.160387        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000582677    0.0611939      1.07803     0.00446887     0.167545        100    
reco:opdec:Deconvolution                               0.0860337     0.303926      0.613927      0.296904      0.0999621       100    
reco:ophitspe:OpHitFinderDeco                           0.21522      0.224088      0.265317      0.223383     0.00606795       100    
reco:opflash:OpFlashFinder                            0.000250111   0.00151188    0.00520258     0.0014382    0.000769186      100    
reco:opslicer:OpSlicer                                0.000474294    0.290005       1.0374       0.240727      0.241146        100    
reco:rns:RandomNumberSaver                            2.1561e-05    4.1343e-05    0.00049361    3.2441e-05    4.63586e-05      100    
reco:anglereconue:NuAngularReco                       0.000172545   0.00261688     0.0772867    0.000492592   0.00929572       100    
reco:anglereconumu:NuAngularReco                      0.000101411   0.00247673     0.0782001    0.000335211   0.00939654       100    
reco:anglereconuepfps:NuAngularReco                   0.000155303   0.00255538     0.0764061    0.000441451   0.00922043       100    
reco:anglereconumupfps:NuAngularReco                  0.000144833   0.00248509     0.0745211    0.000405458    0.0091154       100    
reco:anglerecohits:NuAngularReco                      0.000403911   0.00335872     0.0817504    0.00101976     0.0100522       100    
[art]:TriggerResults:TriggerResultInserter            1.1742e-05    1.49195e-05   7.3148e-05    1.3496e-05    6.21707e-06      100    
end_path:cafmaker:CAFMaker                            0.00139065     0.0350385      1.02114     0.00648187     0.118672        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3666.16 MB
  Peak resident set size usage (VmHWM): 1999.05 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 12:50:41 UTC ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a7/83/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_1014_20231203T154815Z_gen_g4_detsim_hitreco__20240508T072052Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 611!
Begin processing the 95th record. run: 50624901 subRun: 1 event: 15795 at 15-Sep-2025 12:50:12 UTC
Boundary wire vector sizes: 102, 90, 103
minwire 0: 1894
minwire 1: 890
minwire 2: 1903
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.98851, 0.000752039, 0.0107382, 
Output 1: 0.0186766, 0.972487, 0.00866898, 0.000167535, 
Output 2: 0.994091, 0.00577278, 0.000133586, 2.8944e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1391!
Begin processing the 96th record. run: 50624901 subRun: 1 event: 15796 at 15-Sep-2025 12:50:16 UTC
Boundary wire vector sizes: 448, 755, 675
minwire 0: 2312
minwire 1: 327
minwire 2: 2358
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.632643, 0.00314431, 0.364213, 
Output 1: 0.28003, 0.698709, 0.020053, 0.00120777, 
Output 2: 0.228003, 0.745366, 0.0261999, 0.000430433, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1693!
Begin processing the 97th record. run: 50624901 subRun: 1 event: 15797 at 15-Sep-2025 12:50:21 UTC
Boundary wire vector sizes: 100, 51, 50
minwire 0: 2128
minwire 1: 886
minwire 2: 1949
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00273376, 0.00151324, 0.995753, 
Output 1: 0.0710038, 0.921585, 0.00721435, 0.000196358, 
Output 2: 0.998048, 0.00185993, 8.82211e-05, 4.07602e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1487!
Begin processing the 98th record. run: 50624901 subRun: 1 event: 15798 at 15-Sep-2025 12:50:26 UTC
Boundary wire vector sizes: 342, 307, 134
minwire 0: 1132
minwire 1: 1022
minwire 2: 1304
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996532, 0.000371831, 0.00309601, 
Output 1: 0.0181945, 0.091714, 0.495964, 0.394127, 
Output 2: 0.97438, 0.024337, 0.0011943, 8.91438e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1875!
Begin processing the 99th record. run: 50624901 subRun: 1 event: 15799 at 15-Sep-2025 12:50:31 UTC
Boundary wire vector sizes: 207, 249, 175
minwire 0: 2814
minwire 1: 581
minwire 2: 2484
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00329055, 0.985342, 0.0113672, 
Output 1: 0.955289, 0.0437699, 0.000874854, 6.67146e-05, 
Output 2: 0.223197, 0.765997, 0.0106855, 0.000120447, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1457!
Begin processing the 100th record. run: 50624901 subRun: 1 event: 15800 at 15-Sep-2025 12:50:35 UTC
Boundary wire vector sizes: 470, 678, 505
minwire 0: 1441
minwire 1: 966
minwire 2: 1558
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.65303, 0.0181008, 0.328869, 
Output 1: 0.483523, 0.380109, 0.100184, 0.0361837, 
Output 2: 0.124442, 0.419702, 0.317068, 0.138788, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2593!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cf/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_157_20231207T123809Z_gen_g4_detsim_hitreco__20240510T042104Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a7/83/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_1014_20231203T154815Z_gen_g4_detsim_hitreco__20240508T072052Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/ed/de/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472938_487_20231208T100110Z_gen_g4_detsim_hitreco__20240510T065017Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/06/5e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_226_20231202T112332Z_gen_g4_detsim_hitreco__20240507T215248Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a2/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_351_20231207T134706Z_gen_g4_detsim_hitreco__20240510T062951Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/eb/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_316_20231207T083332Z_gen_g4_detsim_hitreco__20240510T050015Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/54/94/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_71_20231203T141327Z_gen_g4_detsim_hitreco__20240508T055304Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/54/f6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_758_20231208T072001Z_gen_g4_detsim_hitreco__20240510T060510Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/f1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_665_20231207T080916Z_gen_g4_detsim_hitreco__20240510T063925Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/12/fa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_216_20231207T035919Z_gen_g4_detsim_hitreco__20240510T043609Z_reco2.root
.:
total 6044
-rw-r--r-- 1 dnpil001 users 2353032 Sep 15 12:50 flatcaf.root
-rw-r--r-- 1 dnpil001 users 1841393 Sep 15 12:50 caf.root
-rw-r--r-- 1 dnpil001 users 1841393 Sep 15 12:50 caf_fd_hd_atmo_2501_20250915T123340Z.root
-rw-r--r-- 1 dnpil001 users   71781 Sep 15 12:50 caf_20250915T123340Z.log
-rw-r--r-- 1 dnpil001 users   42592 Sep 15 12:50 jobscript.log
-rw-r--r-- 1 dnpil001 users    1904 Sep 15 12:50 caf_20250915T123340Z.file
-rw-r--r-- 1 dnpil001 users    1904 Sep 15 12:50 caf_20250915T123340Z.pfns
-rw-r--r-- 1 dnpil001 users    1904 Sep 15 12:33 file.list
-rw-r--r-- 1 dnpil001 users    1904 Sep 15 12:50 justin-processed-pfns.txt
-rw-r--r-- 1 dnpil001 users    1374 Sep 15 12:33 all-input-dids.txt
-rw-r--r-- 1 dnpil001 users    1374 Sep 15 12:50 caf_20250915T123340Z.did
-rw-r--r-- 1 dnpil001 users    1374 Sep 15 12:33 did.list
-rw-r--r-- 1 dnpil001 users       0 Sep 15 12:33 debugprod.log