Jobsub ID 107917.40@dunegpschedd01.fnal.gov

Jobsub ID	107917.40@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 10:17:39
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw6
Last heartbeat	2025-09-23 10:45:10
From worker node	Hostname	mh-7662-1.t2.ucsd.edu
	cpuinfo	AMD EPYC 7662 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:20:12
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_527_20231205T033012Z_gen_g4_detsim_hitreco__20240509T221720Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_403_20231204T140343Z_gen_g4_detsim_hitreco__20240509T221437Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_895_20231202T140048Z_gen_g4_detsim_hitreco__20240507T223426Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_82_20231202T042635Z_gen_g4_detsim_hitreco__20240507T202740Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_329_20231201T170206Z_gen_g4_detsim_hitreco__20240507T191436Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_796_20231204T140413Z_gen_g4_detsim_hitreco__20240509T202818Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 10:45:10
Saved logs	justin-logs:107917.40-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

12681       2.78048      0.0628526     0.426981        100    
reco:energyrecnue:EnergyReco                          0.000212475   0.00374928     0.0878154    0.00104898     0.0102545       100    
reco:energyrecnc:EnergyReco                           0.00274662     0.480096       1.26926      0.636209      0.360976        100    
reco:energyrecnumurange:EnergyReco                    0.00043749     0.0771141      1.89475     0.00425397     0.276434        100    
reco:energyrecnumumcs:EnergyReco                      0.00134271     0.0721442      1.82706     0.00524983     0.241646        100    
reco:energyrecnumumcsllhd:EnergyReco                  0.00110044     0.0713351      1.77899     0.00498481     0.233485        100    
reco:opdec:Deconvolution                               0.462608      0.819647       1.36691      0.856988      0.189581        100    
reco:ophitspe:OpHitFinderDeco                           1.36831       1.37794       1.39279       1.37799     0.00496443       100    
reco:opflash:OpFlashFinder                            0.000210485   0.00201846    0.00808631    0.00199709    0.00106948       100    
reco:opslicer:OpSlicer                                5.9422e-05     0.410833       1.23524      0.398524       0.32194        100    
reco:rns:RandomNumberSaver                             3.012e-05    6.92373e-05   0.000650015   5.7371e-05    6.43613e-05      100    
reco:anglereconue:NuAngularReco                       0.000263696   0.00244378     0.0499533    0.00066748    0.00662762       100    
reco:anglereconumu:NuAngularReco                      9.2393e-05    0.00207343     0.0502947    0.000345668   0.00664242       100    
reco:anglereconuepfps:NuAngularReco                   0.000147924   0.00227726     0.0503324    0.000523182   0.00666917       100    
reco:anglereconumupfps:NuAngularReco                  0.000126083   0.00220369     0.0500526    0.000446825   0.00667309       100    
reco:anglerecohits:NuAngularReco                      0.000145423   0.00344241     0.0621786    0.00119441    0.00801097       100    
[art]:TriggerResults:TriggerResultInserter            1.2221e-05    2.22537e-05   0.000119723   1.7866e-05    1.97148e-05      100    
end_path:cafmaker:CAFMaker                            0.00254111     0.0337533     0.601448      0.0103115     0.0814546       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3652.15 MB
  Peak resident set size usage (VmHWM): 1928.68 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 03:44:44 PDT ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/10/46/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_403_20231204T140343Z_gen_g4_detsim_hitreco__20240509T221437Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 2: 0.997666, 0.00224619, 6.50342e-05, 2.32772e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2124!
Begin processing the 93rd record. run: 74521450 subRun: 1 event: 52793 at 23-Sep-2025 03:38:53 PDT
Boundary wire vector sizes: 68, 80, 58
minwire 0: 1896
minwire 1: 381
minwire 2: 2045
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.227943, 0.277376, 0.49468, 
Output 1: 0.0531377, 0.305704, 0.505992, 0.135166, 
Output 2: 0.576909, 0.383362, 0.0378635, 0.00186479, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1194!
Begin processing the 94th record. run: 74521450 subRun: 1 event: 52794 at 23-Sep-2025 03:39:02 PDT
Boundary wire vector sizes: 924, 1245, 1001
minwire 0: 712
minwire 1: 1723
minwire 2: 206
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0037254, 0.962617, 0.0336576, 
Output 1: 0.148608, 0.649346, 0.183106, 0.0189403, 
Output 2: 0.803856, 0.18717, 0.0086566, 0.000317939, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2960!
Begin processing the 95th record. run: 74521450 subRun: 1 event: 52795 at 23-Sep-2025 03:39:13 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 108!
Begin processing the 96th record. run: 74521450 subRun: 1 event: 52796 at 23-Sep-2025 03:39:20 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 547!
Begin processing the 97th record. run: 74521450 subRun: 1 event: 52797 at 23-Sep-2025 03:39:28 PDT
Boundary wire vector sizes: 645, 676, 510
minwire 0: 950
minwire 1: 2058
minwire 2: 375
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00291578, 0.981183, 0.0159011, 
Output 1: 0.0576964, 0.439389, 0.449423, 0.0534913, 
Output 2: 0.17808, 0.545473, 0.254418, 0.0220287, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2470!
Begin processing the 98th record. run: 74521450 subRun: 1 event: 52798 at 23-Sep-2025 03:39:39 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1023!
Begin processing the 99th record. run: 74521450 subRun: 1 event: 52799 at 23-Sep-2025 03:39:47 PDT
Boundary wire vector sizes: 2996, 3639, 2800
minwire 0: 2352
minwire 1: 115
minwire 2: 2111
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.828369, 0.148306, 0.0233247, 
Output 1: 0.287413, 0.607848, 0.084242, 0.0204971, 
Output 2: 0.173311, 0.363704, 0.298991, 0.163994, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2768!
Begin processing the 100th record. run: 74521450 subRun: 1 event: 52800 at 23-Sep-2025 03:40:35 PDT
Boundary wire vector sizes: 5854, 5254, 3438
minwire 0: 268
minwire 1: 1695
minwire 2: 97
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.85399, 0.0889656, 0.0570441, 
Output 1: 0.433856, 0.531647, 0.0248256, 0.00967116, 
Output 2: 0.11158, 0.406686, 0.259921, 0.221814, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2803!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2c/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_527_20231205T033012Z_gen_g4_detsim_hitreco__20240509T221720Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/10/46/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_403_20231204T140343Z_gen_g4_detsim_hitreco__20240509T221437Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/f7/0a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_895_20231202T140048Z_gen_g4_detsim_hitreco__20240507T223426Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e8/0e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_82_20231202T042635Z_gen_g4_detsim_hitreco__20240507T202740Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7d/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_329_20231201T170206Z_gen_g4_detsim_hitreco__20240507T191436Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/95/90/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_796_20231204T140413Z_gen_g4_detsim_hitreco__20240509T202818Z_reco2.root
.:
total 4964
-rw-r--r--. 1 cuser cuser 1945930 Sep 23 03:44 flatcaf.root
-rw-r--r--. 1 cuser cuser 1491821 Sep 23 03:44 caf.root
-rw-r--r--. 1 cuser cuser 1491821 Sep 23 03:44 caf_fd_hd_atmo_3302_20250923T102021Z.root
-rw-r--r--. 1 cuser cuser   72846 Sep 23 03:44 caf_20250923T102021Z.log
-rw-r--r--. 1 cuser cuser   40898 Sep 23 03:44 jobscript.log
-rw-r--r--. 1 cuser cuser    1241 Sep 23 03:44 caf_20250923T102021Z.file
-rw-r--r--. 1 cuser cuser    1241 Sep 23 03:44 caf_20250923T102021Z.pfns
-rw-r--r--. 1 cuser cuser    1241 Sep 23 03:20 file.list
-rw-r--r--. 1 cuser cuser    1241 Sep 23 03:44 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser     825 Sep 23 03:20 all-input-dids.txt
-rw-r--r--. 1 cuser cuser     825 Sep 23 03:44 caf_20250923T102021Z.did
-rw-r--r--. 1 cuser cuser     825 Sep 23 03:20 did.list
-rw-r--r--. 1 cuser cuser       0 Sep 23 03:20 debugprod.log