Jobsub ID 47404.70@dunegpschedd01.fnal.gov

Jobsub ID	47404.70@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 07:52:57
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_condce_mcore
Last heartbeat	2025-09-17 08:11:03
From worker node	Hostname	mwt2-c009.campuscluster.illinois.edu
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-17 07:53:33
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_327_20231203T110835Z_gen_g4_detsim_hitreco__20240508T042307Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_502_20231202T082910Z_gen_g4_detsim_hitreco__20240507T205121Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_994_20231201T211134Z_gen_g4_detsim_hitreco__20240508T024440Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_296_20231203T035656Z_gen_g4_detsim_hitreco__20240508T073707Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_248_20231203T153645Z_gen_g4_detsim_hitreco__20240508T031823Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_104_20231122T081035Z_gen_g4_detsim_hitreco__20240503T065048Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_52_20231203T090316Z_gen_g4_detsim_hitreco__20240509T195132Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_993_20231202T182947Z_gen_g4_detsim_hitreco__20240508T065422Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_363_20231207T135125Z_gen_g4_detsim_hitreco__20240510T063216Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_27_20231208T072741Z_gen_g4_detsim_hitreco__20240510T061353Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-17 08:11:03
Saved logs	justin-logs:47404.70-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Reco                      0.00065968     0.0228249     0.500362     0.00337977     0.0685382       100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00045751     0.0227599     0.506201     0.00342512     0.0688376       100    
reco:opdec:Deconvolution                               0.325229       0.56104      0.955315      0.576371      0.112134        100    
reco:ophitspe:OpHitFinderDeco                          0.971534      0.978265      0.985378      0.977953     0.00304709       100    
reco:opflash:OpFlashFinder                            0.000236284   0.00151802     0.0056719    0.00140803    0.000765271      100    
reco:opslicer:OpSlicer                                4.4404e-05     0.273235       1.03128      0.205081      0.225738        100    
reco:rns:RandomNumberSaver                             2.684e-05    8.30912e-05   0.00047391    6.44415e-05   7.95757e-05      100    
reco:anglereconue:NuAngularReco                       0.00021269    0.00193815     0.0439043    0.000572431   0.00545184       100    
reco:anglereconumu:NuAngularReco                      8.9037e-05    0.00156897     0.043907     0.000287816   0.00539306       100    
reco:anglereconuepfps:NuAngularReco                   0.000145504   0.00175383     0.0430958    0.000433335   0.00532986       100    
reco:anglereconumupfps:NuAngularReco                  0.000120777   0.00163237     0.0439052    0.000340004   0.00539951       100    
reco:anglerecohits:NuAngularReco                      0.000117281   0.00324762     0.0787296    0.000832865   0.00992361       100    
[art]:TriggerResults:TriggerResultInserter            1.2263e-05    1.92568e-05   7.3578e-05    1.7017e-05    8.58235e-06      100    
end_path:cafmaker:CAFMaker                             0.002491      0.0372526      1.12429     0.00829208     0.131978        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3652.6 MB
  Peak resident set size usage (VmHWM): 1974.85 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 03:10:47 CDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_502_20231202T082910Z_gen_g4_detsim_hitreco__20240507T205121Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.428849, 0.0575644, 0.513587, 
Output 1: 0.120231, 0.27534, 0.299492, 0.304937, 
Output 2: 0.331083, 0.417177, 0.194411, 0.0573297, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2664!
Begin processing the 94th record. run: 6418365 subRun: 1 event: 32794 at 17-Sep-2025 03:09:58 CDT
Boundary wire vector sizes: 65, 99, 84
minwire 0: 542
minwire 1: 1751
minwire 2: 256
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0860417, 0.00855567, 0.905403, 
Output 1: 0.73639, 0.25199, 0.0107799, 0.000839756, 
Output 2: 0.0930242, 0.89837, 0.008525, 8.08299e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1203!
Begin processing the 95th record. run: 6418365 subRun: 1 event: 32795 at 17-Sep-2025 03:10:05 CDT
Boundary wire vector sizes: 2081, 2099, 1360
minwire 0: 446
minwire 1: 2379
minwire 2: 4
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000217771, 0.998759, 0.00102282, 
Output 1: 0.0159831, 0.980011, 0.00386979, 0.000136047, 
Output 2: 0.997412, 0.00254178, 4.50565e-05, 1.2542e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2542!
Begin processing the 96th record. run: 6418365 subRun: 1 event: 32796 at 17-Sep-2025 03:10:15 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1205!
Begin processing the 97th record. run: 6418365 subRun: 1 event: 32797 at 17-Sep-2025 03:10:20 CDT
Boundary wire vector sizes: 786, 608, 157
minwire 0: 1902
minwire 1: 431
minwire 2: 2209
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.98793, 0.00113872, 0.0109312, 
Output 1: 0.00538387, 0.0229584, 0.196096, 0.775562, 
Output 2: 0.99076, 0.00883344, 0.000354895, 5.20496e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1836!
Begin processing the 98th record. run: 6418365 subRun: 1 event: 32798 at 17-Sep-2025 03:10:27 CDT
Boundary wire vector sizes: 46, 109, 75
minwire 0: 123
minwire 1: 2343
minwire 2: 0
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.556925, 0.00897348, 0.434101, 
Output 1: 0.663468, 0.330377, 0.00591854, 0.000236779, 
Output 2: 0.503239, 0.491698, 0.00500323, 6.02379e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 791!
Begin processing the 99th record. run: 6418365 subRun: 1 event: 32799 at 17-Sep-2025 03:10:34 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1004!
Begin processing the 100th record. run: 6418365 subRun: 1 event: 32800 at 17-Sep-2025 03:10:40 CDT
Boundary wire vector sizes: 447, 182, 271
minwire 0: 487
minwire 1: 2780
minwire 2: 8
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000230825, 0.998895, 0.000874428, 
Output 1: 0.00704887, 0.987834, 0.00493191, 0.000184993, 
Output 2: 0.998509, 0.00144892, 4.03224e-05, 1.27829e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1598!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a0/1c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_327_20231203T110835Z_gen_g4_detsim_hitreco__20240508T042307Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_502_20231202T082910Z_gen_g4_detsim_hitreco__20240507T205121Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/23/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_994_20231201T211134Z_gen_g4_detsim_hitreco__20240508T024440Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0e/94/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_296_20231203T035656Z_gen_g4_detsim_hitreco__20240508T073707Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/34/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_248_20231203T153645Z_gen_g4_detsim_hitreco__20240508T031823Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/68/0f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220954_104_20231122T081035Z_gen_g4_detsim_hitreco__20240503T065048Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/aa/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_52_20231203T090316Z_gen_g4_detsim_hitreco__20240509T195132Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/ee/8a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_993_20231202T182947Z_gen_g4_detsim_hitreco__20240508T065422Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3e/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_363_20231207T135125Z_gen_g4_detsim_hitreco__20240510T063216Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f2/88/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_27_20231208T072741Z_gen_g4_detsim_hitreco__20240510T061353Z_reco2.root
.:
total 5540
-rw-r--r-- 1 dune osgvo 2160071 Sep 17 03:10 flatcaf.root
-rw-r--r-- 1 dune osgvo 1674197 Sep 17 03:10 caf.root
-rw-r--r-- 1 dune osgvo 1674197 Sep 17 03:10 caf_fd_hd_atmo_2501_20250917T075339Z.root
-rw-r--r-- 1 dune osgvo   74448 Sep 17 03:10 caf_20250917T075339Z.log
-rw-r--r-- 1 dune osgvo   32978 Sep 17 03:10 jobscript.log
-rw-r--r-- 1 dune osgvo    1884 Sep 17 03:10 caf_20250917T075339Z.file
-rw-r--r-- 1 dune osgvo    1884 Sep 17 03:10 caf_20250917T075339Z.pfns
-rw-r--r-- 1 dune osgvo    1884 Sep 17 02:53 file.list
-rw-r--r-- 1 dune osgvo    1884 Sep 17 03:10 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo    1374 Sep 17 02:53 all-input-dids.txt
-rw-r--r-- 1 dune osgvo    1374 Sep 17 03:10 caf_20250917T075339Z.did
-rw-r--r-- 1 dune osgvo    1374 Sep 17 02:53 did.list
-rw-r--r-- 1 dune osgvo       0 Sep 17 02:53 debugprod.log