Jobsub ID 41725.11@dunegpschedd02.fnal.gov

Jobsub ID	41725.11@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-18 07:28:10
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw6
Last heartbeat	2025-09-18 11:48:53
From worker node	Hostname	mh-7662-12.t2.ucsd.edu
	cpuinfo	AMD EPYC 7662 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-09-18 07:31:32
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_931_20231202T113802Z_gen_g4_detsim_hitreco__20240507T212327Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_602_20231203T141918Z_gen_g4_detsim_hitreco__20240508T080128Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_918_20231202T191330Z_gen_g4_detsim_hitreco__20240508T064428Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_267_20231204T063107Z_gen_g4_detsim_hitreco__20240509T204651Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_56_20231206T230207Z_gen_g4_detsim_hitreco__20240510T031902Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_24_20231208T040205Z_gen_g4_detsim_hitreco__20240510T065328Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_299_20231207T072019Z_gen_g4_detsim_hitreco__20240510T063126Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_77_20231204T080606Z_gen_g4_detsim_hitreco__20240509T220853Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_173_20231204T060235Z_gen_g4_detsim_hitreco__20240509T214745Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_395_20231120T135034Z_gen_g4_detsim_hitreco__20240503T061452Z_reco2.root
Jobscript	Exit code	0
	Real time	4h (15313s)
	CPU time	2h (7908s = 51%)
	Max RSS bytes	2258042880 (2153 MiB)
Outputting started	2025-09-18 11:46:46
Output files
Finished	2025-09-18 11:48:53
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.264611     0.000957671    0.0155576      1100    
reco:energyrecnc:EnergyReco                           0.00256294     0.495233       1.63143      0.618199      0.333078       1100    
reco:energyrecnumurange:EnergyReco                    0.000295247    0.0893075      9.2324      0.00444265     0.501175       1100    
reco:energyrecnumumcs:EnergyReco                      0.000835099    0.0910086      9.66107     0.00515408     0.515789       1100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000577083    0.0898352      9.54049      0.0051532     0.508111       1100    
reco:opdec:Deconvolution                               0.436143      0.805639       2.94323      0.824692      0.186299       1100    
reco:ophitspe:OpHitFinderDeco                           1.24014       1.33513       1.42632       1.35183      0.0536885      1100    
reco:opflash:OpFlashFinder                            0.000268486   0.00235525     0.012091     0.00214003    0.00129271      1100    
reco:opslicer:OpSlicer                                6.0291e-05     0.397654       1.64758      0.329166      0.316732       1100    
reco:rns:RandomNumberSaver                            2.6051e-05    6.7287e-05    0.00331963    5.4811e-05    0.000122548     1100    
reco:anglereconue:NuAngularReco                       0.000184475   0.00321798     0.174714     0.000663215    0.0120948      1100    
reco:anglereconumu:NuAngularReco                      8.6732e-05    0.00284172     0.173156     0.000351473    0.0118858      1100    
reco:anglereconuepfps:NuAngularReco                   0.000136853   0.00304655     0.176218     0.000511686    0.0120246      1100    
reco:anglereconumupfps:NuAngularReco                  0.000123183   0.00298477      0.18504     0.00045142     0.0120734      1100    
reco:anglerecohits:NuAngularReco                      0.000131263   0.00407648      0.1835      0.00115128     0.0130501      1100    
[art]:TriggerResults:TriggerResultInserter             1.122e-05    1.93189e-05   0.000156604   1.7381e-05    1.20821e-05     1100    
end_path:cafmaker:CAFMaker                            0.00160408     0.0413597      2.08111     0.00921197     0.143603       1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3841.02 MB
  Peak resident set size usage (VmHWM): 2258.04 MB
====================================================================================================
lar exit code 0
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 215, 114, 145
minwire 0: 1492
minwire 1: 1315
minwire 2: 1441
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.718616, 0.250489, 0.0308942, 
Output 1: 0.0191696, 0.916045, 0.0627488, 0.00203639, 
Output 2: 0.0529003, 0.786458, 0.156396, 0.00424595, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1707!
Begin processing the 1095th record. run: 6207207 subRun: 1 event: 79195 at 18-Sep-2025 04:45:46 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 455!
Begin processing the 1096th record. run: 6207207 subRun: 1 event: 79196 at 18-Sep-2025 04:45:53 PDT
Boundary wire vector sizes: 88, 83, 78
minwire 0: 732
minwire 1: 2740
minwire 2: 182
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000548996, 0.995957, 0.00349417, 
Output 1: 0.00381169, 0.989125, 0.00687106, 0.000192431, 
Output 2: 0.997962, 0.00197595, 6.01728e-05, 2.10621e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1115!
Begin processing the 1097th record. run: 6207207 subRun: 1 event: 79197 at 18-Sep-2025 04:46:03 PDT
Boundary wire vector sizes: 1324, 1042, 954
minwire 0: 119
minwire 1: 2182
minwire 2: 68
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000962519, 0.9943, 0.00473702, 
Output 1: 0.0343168, 0.340229, 0.547856, 0.0775987, 
Output 2: 0.883722, 0.10761, 0.00830877, 0.00035906, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1686!
Begin processing the 1098th record. run: 6207207 subRun: 1 event: 79198 at 18-Sep-2025 04:46:14 PDT
Boundary wire vector sizes: 808, 844, 518
minwire 0: 2516
minwire 1: 627
minwire 2: 2336
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000511167, 0.992373, 0.00711578, 
Output 1: 0.411639, 0.56694, 0.0201468, 0.0012741, 
Output 2: 0.989243, 0.0103754, 0.000358485, 2.3049e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2108!
Begin processing the 1099th record. run: 6207207 subRun: 1 event: 79199 at 18-Sep-2025 04:46:25 PDT
Boundary wire vector sizes: 45, 20, 38
minwire 0: 1554
minwire 1: 1822
minwire 2: 1280
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00388248, 0.00248545, 0.993632, 
Output 1: 0.01353, 0.981026, 0.00529644, 0.000147632, 
Output 2: 0.998059, 0.00187939, 5.93671e-05, 2.10608e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1532!
Begin processing the 1100th record. run: 6207207 subRun: 1 event: 79200 at 18-Sep-2025 04:46:33 PDT
Boundary wire vector sizes: 735, 1112, 842
minwire 0: 1815
minwire 1: 520
minwire 2: 2358
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00812495, 0.877228, 0.114647, 
Output 1: 0.0334888, 0.895007, 0.0693611, 0.00214354, 
Output 2: 0.846096, 0.149468, 0.0043513, 8.51324e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1944!
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
processed files
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/79/b0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_931_20231202T113802Z_gen_g4_detsim_hitreco__20240507T212327Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/39/c0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_602_20231203T141918Z_gen_g4_detsim_hitreco__20240508T080128Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4b/72/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_918_20231202T191330Z_gen_g4_detsim_hitreco__20240508T064428Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/67/df/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_267_20231204T063107Z_gen_g4_detsim_hitreco__20240509T204651Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ca/8b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_56_20231206T230207Z_gen_g4_detsim_hitreco__20240510T031902Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/30/9d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_24_20231208T040205Z_gen_g4_detsim_hitreco__20240510T065328Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1e/68/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_299_20231207T072019Z_gen_g4_detsim_hitreco__20240510T063126Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/8c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422672_77_20231204T080606Z_gen_g4_detsim_hitreco__20240509T220853Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d1/0e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_173_20231204T060235Z_gen_g4_detsim_hitreco__20240509T214745Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1b/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_395_20231120T135034Z_gen_g4_detsim_hitreco__20240503T061452Z_reco2.root
.:
total 61948
-rw-r--r-- 1 cuser cuser 23175748 Sep 18 04:46 flatcaf.root
-rw-r--r-- 1 cuser cuser 19548191 Sep 18 04:46 caf.root
-rw-r--r-- 1 cuser cuser 19548191 Sep 18 04:46 caf_fd_hd_atmo_2501_20250918T073141Z.root
-rw-r--r-- 1 cuser cuser   652367 Sep 18 04:46 caf_20250918T073141Z.log
-rw-r--r-- 1 cuser cuser   258313 Sep 18 04:46 jobscript.log
-rw-r--r-- 1 cuser cuser     1956 Sep 18 04:46 caf_20250918T073141Z.file
-rw-r--r-- 1 cuser cuser     1956 Sep 18 04:46 caf_20250918T073141Z.pfns
-rw-r--r-- 1 cuser cuser     1956 Sep 18 00:31 file.list
-rw-r--r-- 1 cuser cuser     1956 Sep 18 04:46 justin-processed-pfns.txt
-rw-r--r-- 1 cuser cuser     1372 Sep 18 00:31 all-input-dids.txt
-rw-r--r-- 1 cuser cuser     1372 Sep 18 04:46 caf_20250918T073141Z.did
-rw-r--r-- 1 cuser cuser     1372 Sep 18 00:31 did.list
-rw-r--r-- 1 cuser cuser        0 Sep 18 00:32 debugprod.log