Jobsub ID 149785.16@dunegpschedd02.fnal.gov

Jobsub ID	149785.16@dunegpschedd02.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 15:41:58
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-09-23 16:15:24
From worker node	Hostname	mh-7763-6.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 15:43:47
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220762_106_20231122T051652Z_gen_g4_detsim_hitreco__20240503T062852Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_49_20231202T162253Z_gen_g4_detsim_hitreco__20240508T043153Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_152_20231205T115557Z_gen_g4_detsim_hitreco__20240510T062657Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_720_20231208T082429Z_gen_g4_detsim_hitreco__20240510T063045Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_639_20231207T080827Z_gen_g4_detsim_hitreco__20240510T062946Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_715_20231203T112336Z_gen_g4_detsim_hitreco__20240508T050700Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_363_20231207T154044Z_gen_g4_detsim_hitreco__20240510T044120Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_791_20231203T232221Z_gen_g4_detsim_hitreco__20240509T200309Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_294_20231207T010719Z_gen_g4_detsim_hitreco__20240510T035415Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_290_20231201T170157Z_gen_g4_detsim_hitreco__20240507T190735Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 16:15:24
Saved logs	justin-logs:149785.16-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.908967     0.00342525     0.126997        100    
reco:opdec:Deconvolution                               0.470515       1.02193       2.06429      0.963987      0.296689        100    
reco:ophitspe:OpHitFinderDeco                           1.36545       1.39953       2.09299       1.37462      0.105028        100    
reco:opflash:OpFlashFinder                            0.000293069   0.00135394    0.00375551    0.00126542    0.000616072      100    
reco:opslicer:OpSlicer                                0.000872436    0.241643      0.782804      0.200522       0.18281        100    
reco:rns:RandomNumberSaver                             2.533e-05    4.97003e-05   0.000377708    3.977e-05    3.7591e-05       100    
reco:anglereconue:NuAngularReco                       0.000147849   0.00117811     0.0325944    0.000404143   0.00346923       100    
reco:anglereconumu:NuAngularReco                       7.142e-05    0.000951943    0.0316778    0.000231864   0.00332437       100    
reco:anglereconuepfps:NuAngularReco                   0.00010239     0.001081      0.0318236    0.000331578   0.00333992       100    
reco:anglereconumupfps:NuAngularReco                  9.5129e-05     0.0010195     0.0323793    0.000298568   0.00338528       100    
reco:anglerecohits:NuAngularReco                      0.000167199   0.00169313     0.0411838    0.000751506   0.00428508       100    
[art]:TriggerResults:TriggerResultInserter             9.13e-06     1.54155e-05    6.148e-05    1.4265e-05    6.69106e-06      100    
end_path:cafmaker:CAFMaker                            0.00142822     0.0174203     0.469188     0.00491485     0.052608        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3660.61 MB
  Peak resident set size usage (VmHWM): 1952.78 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 09:15:04 PDT ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/2c/0b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_49_20231202T162253Z_gen_g4_detsim_hitreco__20240508T043153Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.70332, 0.141499, 0.155181, 
Output 1: 0.0218237, 0.0816565, 0.158618, 0.737901, 
Output 2: 0.0517027, 0.109553, 0.251599, 0.587146, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2373!
Begin processing the 93rd record. run: 6220762 subRun: 1 event: 10693 at 23-Sep-2025 09:06:06 PDT
Boundary wire vector sizes: 414, 391, 123
minwire 0: 703
minwire 1: 2169
minwire 2: 438
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.90769, 0.00392223, 0.0883879, 
Output 1: 0.0215004, 0.974937, 0.00342931, 0.00013365, 
Output 2: 0.917893, 0.0795016, 0.00255792, 4.77047e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1752!
Begin processing the 94th record. run: 6220762 subRun: 1 event: 10694 at 23-Sep-2025 09:06:17 PDT
Boundary wire vector sizes: 259, 357, 277
minwire 0: 2148
minwire 1: 1104
minwire 2: 1892
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0116803, 0.958507, 0.029813, 
Output 1: 0.682699, 0.307165, 0.00945784, 0.000678765, 
Output 2: 0.0844646, 0.9061, 0.009366, 6.98384e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2107!
Begin processing the 95th record. run: 6220762 subRun: 1 event: 10695 at 23-Sep-2025 09:06:31 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 665!
Begin processing the 96th record. run: 6220762 subRun: 1 event: 10696 at 23-Sep-2025 09:06:38 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1364!
Begin processing the 97th record. run: 6220762 subRun: 1 event: 10697 at 23-Sep-2025 09:06:47 PDT
Boundary wire vector sizes: 377, 973, 646
minwire 0: 2021
minwire 1: 486
minwire 2: 1970
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.988733, 0.000946247, 0.0103212, 
Output 1: 0.991447, 0.00828267, 0.000177873, 9.23492e-05, 
Output 2: 0.995124, 0.00477509, 8.37761e-05, 1.73105e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1700!
Begin processing the 98th record. run: 6220762 subRun: 1 event: 10698 at 23-Sep-2025 09:06:58 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 331!
Begin processing the 99th record. run: 6220762 subRun: 1 event: 10699 at 23-Sep-2025 09:07:05 PDT
Boundary wire vector sizes: 348, 577, 532
minwire 0: 127
minwire 1: 1911
minwire 2: 14
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996192, 0.000627744, 0.00318043, 
Output 1: 0.0354586, 0.824988, 0.136151, 0.00340234, 
Output 2: 0.991213, 0.00854567, 0.000236915, 4.34293e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1133!
Begin processing the 100th record. run: 6220762 subRun: 1 event: 10700 at 23-Sep-2025 09:07:16 PDT
Boundary wire vector sizes: 2066, 1642, 1593
minwire 0: 1186
minwire 1: 1345
minwire 2: 656
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000456661, 0.998213, 0.00133049, 
Output 1: 0.00497637, 0.988042, 0.00671454, 0.000267337, 
Output 2: 0.996824, 0.00307511, 9.75407e-05, 3.29594e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3122!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7e/a8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6220762_106_20231122T051652Z_gen_g4_detsim_hitreco__20240503T062852Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/2c/0b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_49_20231202T162253Z_gen_g4_detsim_hitreco__20240508T043153Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6a/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_152_20231205T115557Z_gen_g4_detsim_hitreco__20240510T062657Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/dd/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_720_20231208T082429Z_gen_g4_detsim_hitreco__20240510T063045Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ce/87/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_639_20231207T080827Z_gen_g4_detsim_hitreco__20240510T062946Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/80/44/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6418365_715_20231203T112336Z_gen_g4_detsim_hitreco__20240508T050700Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/47/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_363_20231207T154044Z_gen_g4_detsim_hitreco__20240510T044120Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a2/ce/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_791_20231203T232221Z_gen_g4_detsim_hitreco__20240509T200309Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/37/3b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_294_20231207T010719Z_gen_g4_detsim_hitreco__20240510T035415Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_290_20231201T170157Z_gen_g4_detsim_hitreco__20240507T190735Z_reco2.root
.:
total 4904
-rw-r--r--. 1 cuser cuser 1932485 Sep 23 09:15 flatcaf.root
-rw-r--r--. 1 cuser cuser 1472276 Sep 23 09:15 caf.root
-rw-r--r--. 1 cuser cuser 1472276 Sep 23 09:15 caf_fd_hd_atmo_3302_20250923T154355Z.root
-rw-r--r--. 1 cuser cuser   73318 Sep 23 09:15 caf_20250923T154355Z.log
-rw-r--r--. 1 cuser cuser   34253 Sep 23 09:15 jobscript.log
-rw-r--r--. 1 cuser cuser    1969 Sep 23 09:15 caf_20250923T154355Z.file
-rw-r--r--. 1 cuser cuser    1969 Sep 23 09:15 caf_20250923T154355Z.pfns
-rw-r--r--. 1 cuser cuser    1969 Sep 23 08:43 file.list
-rw-r--r--. 1 cuser cuser    1969 Sep 23 09:15 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser    1375 Sep 23 08:43 all-input-dids.txt
-rw-r--r--. 1 cuser cuser    1375 Sep 23 09:15 caf_20250923T154355Z.did
-rw-r--r--. 1 cuser cuser    1375 Sep 23 08:43 did.list
-rw-r--r--. 1 cuser cuser       0 Sep 23 08:44 debugprod.log