Jobsub ID 44848.5@dunegpschedd01.fnal.gov

Jobsub ID	44848.5@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:10:35
Site	US_Wisconsin
Entry	HCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat	2025-09-11 11:49:53
From worker node	Hostname	e2603
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	82800 (23 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-11 10:54:05
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_204_20231202T190818Z_gen_g4_detsim_hitreco__20240508T060736Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_703_20231202T161004Z_gen_g4_detsim_hitreco__20240508T043928Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_411_20231201T115303Z_gen_g4_detsim_hitreco__20240507T185645Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_187_20231208T001422Z_gen_g4_detsim_hitreco__20240510T030125Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_263_20231202T093402Z_gen_g4_detsim_hitreco__20240507T212930Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_798_20231208T084632Z_gen_g4_detsim_hitreco__20240510T061052Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_201_20231122T090957Z_gen_g4_detsim_hitreco__20240503T064345Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_1189_20231203T121814Z_gen_g4_detsim_hitreco__20240509T192918Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_730_20231204T140408Z_gen_g4_detsim_hitreco__20240509T203412Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_558_20231202T181122Z_gen_g4_detsim_hitreco__20240508T064500Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-11 11:49:53
Saved logs	justin-logs:44848.5-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

:energyrecnumumcsllhd:EnergyReco                  0.000313873    0.0193497      2.1659      0.00174051     0.118714        400    
reco:opdec:Deconvolution                               0.328811      0.520211       1.11826       0.53669      0.0895176       400    
reco:ophitspe:OpHitFinderDeco                          0.971763      0.994202       1.19471      0.998428      0.0147258       400    
reco:opflash:OpFlashFinder                            0.000160734   0.000961541   0.00299905    0.000909584   0.00039506       400    
reco:opslicer:OpSlicer                                4.4754e-05     0.159133      0.697387       0.12436      0.131168        400    
reco:rns:RandomNumberSaver                            1.8405e-05    3.55885e-05   0.000293244   3.14595e-05   1.73957e-05      400    
reco:anglereconue:NuAngularReco                       0.000136969   0.000975315    0.0365496    0.000326647   0.00280195       400    
reco:anglereconumu:NuAngularReco                      6.0504e-05    0.000801271    0.0368692    0.00016933    0.00274356       400    
reco:anglereconuepfps:NuAngularReco                   9.4929e-05    0.000882757    0.0363675    0.00025133    0.00273172       400    
reco:anglereconumupfps:NuAngularReco                  8.3889e-05    0.000841794    0.0366812    0.000208609   0.00275011       400    
reco:anglerecohits:NuAngularReco                      7.8359e-05    0.00125838     0.0404471    0.00049001    0.00311297       400    
[art]:TriggerResults:TriggerResultInserter             8.847e-06    1.20046e-05    5.84e-05     1.10655e-05   4.07147e-06      400    
end_path:cafmaker:CAFMaker                            0.00106629     0.0256643      5.3198      0.00413578     0.267457        400    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3827.8 MB
  Peak resident set size usage (VmHWM): 2149.59 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 06:42:52 CDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/84/18/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_263_20231202T093402Z_gen_g4_detsim_hitreco__20240507T212930Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Classifier summary: 
Output 0: 0.997722, 0.00025291, 0.00202468, 
Output 1: 0.0716873, 0.292352, 0.496999, 0.138962, 
Output 2: 0.360203, 0.624836, 0.0147866, 0.000174414, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2165!
Begin processing the 394th record. run: 6491894 subRun: 1 event: 18794 at 11-Sep-2025 06:42:00 CDT
Boundary wire vector sizes: 4358, 6743, 4582
minwire 0: 1733
minwire 1: 261
minwire 2: 1230
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.778282, 0.151596, 0.0701224, 
Output 1: 0.280065, 0.444378, 0.170317, 0.10524, 
Output 2: 0.0129224, 0.0482628, 0.197159, 0.741656, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3098!
Begin processing the 395th record. run: 6491894 subRun: 1 event: 18795 at 11-Sep-2025 06:42:18 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 330!
Begin processing the 396th record. run: 6491894 subRun: 1 event: 18796 at 11-Sep-2025 06:42:23 CDT
Boundary wire vector sizes: 48, 53, 47
minwire 0: 106
minwire 1: 2155
minwire 2: 61
Used alternate method to get min and max wires due to vertex determination failure: 83, 382
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 87, 386
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00248221, 0.00175814, 0.99576, 
Output 1: 0.980646, 0.0190395, 0.000204024, 0.000110028, 
Output 2: 0.998872, 0.00110246, 2.10489e-05, 4.84455e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 693!
Begin processing the 397th record. run: 6491894 subRun: 1 event: 18797 at 11-Sep-2025 06:42:29 CDT
Boundary wire vector sizes: 99, 168, 103
minwire 0: 803
minwire 1: 1882
minwire 2: 600
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0107504, 0.932391, 0.056859, 
Output 1: 0.0069817, 0.0800584, 0.755493, 0.157467, 
Output 2: 0.984607, 0.0147773, 0.000592144, 2.31732e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1725!
Begin processing the 398th record. run: 6491894 subRun: 1 event: 18798 at 11-Sep-2025 06:42:35 CDT
Boundary wire vector sizes: 126, 70, 91
minwire 0: 1418
minwire 1: 1758
minwire 2: 1118
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000626313, 0.997776, 0.0015979, 
Output 1: 0.0104072, 0.983274, 0.00610122, 0.000218043, 
Output 2: 0.998384, 0.00156037, 5.34939e-05, 1.9459e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1458!
Begin processing the 399th record. run: 6491894 subRun: 1 event: 18799 at 11-Sep-2025 06:42:41 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 910!
Begin processing the 400th record. run: 6491894 subRun: 1 event: 18800 at 11-Sep-2025 06:42:46 CDT
Boundary wire vector sizes: 383, 414, 348
minwire 0: 730
minwire 1: 1432
minwire 2: 247
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0810953, 0.0168346, 0.90207, 
Output 1: 0.200346, 0.65831, 0.134519, 0.00682507, 
Output 2: 0.020227, 0.126691, 0.644064, 0.209018, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3009!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/4c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_204_20231202T190818Z_gen_g4_detsim_hitreco__20240508T060736Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d2/7a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_703_20231202T161004Z_gen_g4_detsim_hitreco__20240508T043928Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/fc/11/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_411_20231201T115303Z_gen_g4_detsim_hitreco__20240507T185645Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/10/79/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_187_20231208T001422Z_gen_g4_detsim_hitreco__20240510T030125Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/84/18/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_263_20231202T093402Z_gen_g4_detsim_hitreco__20240507T212930Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f7/45/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_798_20231208T084632Z_gen_g4_detsim_hitreco__20240510T061052Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/33/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_201_20231122T090957Z_gen_g4_detsim_hitreco__20240503T064345Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e9/15/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_1189_20231203T121814Z_gen_g4_detsim_hitreco__20240509T192918Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c8/84/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_730_20231204T140408Z_gen_g4_detsim_hitreco__20240509T203412Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/17/ee/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_558_20231202T181122Z_gen_g4_detsim_hitreco__20240508T064500Z_reco2.root
.:
total 17400
-rw-r--r-- 1 slot1_3 slot1_3 6605237 Sep 11 06:42 flatcaf.root
-rw-r--r-- 1 slot1_3 slot1_3 5405376 Sep 11 06:42 caf.root
-rw-r--r-- 1 slot1_3 slot1_3 5405376 Sep 11 06:42 caf_fd_hd_atmo_2501_20250911T105409Z.root
-rw-r--r-- 1 slot1_3 slot1_3  250302 Sep 11 06:42 caf_20250911T105409Z.log
-rw-r--r-- 1 slot1_3 slot1_3  101856 Sep 11 06:42 jobscript.log
-rw-r--r-- 1 slot1_3 slot1_3    1929 Sep 11 06:42 caf_20250911T105409Z.file
-rw-r--r-- 1 slot1_3 slot1_3    1929 Sep 11 06:42 caf_20250911T105409Z.pfns
-rw-r--r-- 1 slot1_3 slot1_3    1929 Sep 11 05:54 file.list
-rw-r--r-- 1 slot1_3 slot1_3    1929 Sep 11 06:42 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_3 slot1_3    1377 Sep 11 05:54 all-input-dids.txt
-rw-r--r-- 1 slot1_3 slot1_3    1377 Sep 11 06:42 caf_20250911T105409Z.did
-rw-r--r-- 1 slot1_3 slot1_3    1377 Sep 11 05:54 did.list
-rw-r--r-- 1 slot1_3 slot1_3       0 Sep 11 05:54 debugprod.log