Jobsub ID 105970.53@dunegpschedd01.fnal.gov

Jobsub ID	105970.53@dunegpschedd01.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:51:37
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-23 13:02:32
From worker node	Hostname	wn66
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 09:57:23
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_353_20231203T180402Z_gen_g4_detsim_hitreco__20240509T215025Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_492_20231207T025255Z_gen_g4_detsim_hitreco__20240510T050830Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_769_20231207T001723Z_gen_g4_detsim_hitreco__20240510T043319Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_397_20231201T170319Z_gen_g4_detsim_hitreco__20240507T201101Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_414_20231118T191900Z_gen_g4_detsim_hitreco__20240503T060447Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50264716_55_20231122T125719Z_gen_g4_detsim_hitreco__20240507T180717Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_108_20231204T043309Z_gen_g4_detsim_hitreco__20240509T202832Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_786_20231207T152046Z_gen_g4_detsim_hitreco__20240510T061752Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_713_20231208T084302Z_gen_g4_detsim_hitreco__20240510T060533Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_205_20231205T020636Z_gen_g4_detsim_hitreco__20240509T222712Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 13:02:32
Saved logs	justin-logs:105970.53-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0705      4.59379     0.00467104     0.276252        700    
reco:opdec:Deconvolution                               0.498392      0.980426       1.96758      0.981311      0.210357        700    
reco:ophitspe:OpHitFinderDeco                           1.46178       1.74382       1.86709       1.77573      0.115327        700    
reco:opflash:OpFlashFinder                            0.000142805   0.00176383     0.0153283    0.00167033    0.00104254       700    
reco:opslicer:OpSlicer                                4.8372e-05     0.359679       1.54467      0.279416      0.304898        700    
reco:rns:RandomNumberSaver                            1.9971e-05    3.86146e-05   0.000487338   3.66915e-05   2.08073e-05      700    
reco:anglereconue:NuAngularReco                       0.000158595   0.00263545     0.132133     0.000497468   0.00963186       700    
reco:anglereconumu:NuAngularReco                      8.7283e-05    0.00240744     0.121892     0.000342517    0.0088874       700    
reco:anglereconuepfps:NuAngularReco                   0.000151076   0.00251262     0.128773     0.000456321   0.00888533       700    
reco:anglereconumupfps:NuAngularReco                  0.000116855   0.00249006     0.123751     0.000428516   0.00886545       700    
reco:anglerecohits:NuAngularReco                      8.6824e-05    0.00339627     0.135632     0.000977845    0.0104318       700    
[art]:TriggerResults:TriggerResultInserter             1.047e-05    1.80395e-05   6.1923e-05     1.751e-05    5.28183e-06      700    
end_path:cafmaker:CAFMaker                            0.00120121     0.0305009      1.33728     0.00517616     0.105748        700    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3671.91 MB
  Peak resident set size usage (VmHWM): 1994.05 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 09:58:14 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/1d/4e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_108_20231204T043309Z_gen_g4_detsim_hitreco__20240509T202832Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 389, 381, 340
minwire 0: 997
minwire 1: 1440
minwire 2: 1052
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0421683, 0.810388, 0.147443, 
Output 1: 0.0166443, 0.179417, 0.478981, 0.324957, 
Output 2: 0.449174, 0.427257, 0.109579, 0.0139898, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2001!
Begin processing the 692nd record. run: 50264716 subRun: 1 event: 5592 at 23-Sep-2025 09:49:24 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 418!
Begin processing the 693rd record. run: 50264716 subRun: 1 event: 5593 at 23-Sep-2025 09:49:33 -03
Boundary wire vector sizes: 945, 1578, 1318
minwire 0: 1519
minwire 1: 398
minwire 2: 1665
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.165059, 0.468103, 0.366838, 
Output 1: 0.274858, 0.619178, 0.0893408, 0.0166232, 
Output 2: 0.0852916, 0.215394, 0.378126, 0.321188, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2100!
Begin processing the 694th record. run: 50264716 subRun: 1 event: 5594 at 23-Sep-2025 09:49:46 -03
Boundary wire vector sizes: 101, 205, 124
minwire 0: 1197
minwire 1: 1140
minwire 2: 1332
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99488, 0.000909902, 0.00421025, 
Output 1: 0.0163112, 0.979976, 0.00362932, 8.34227e-05, 
Output 2: 0.99619, 0.0036997, 0.000107846, 2.70594e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1193!
Begin processing the 695th record. run: 50264716 subRun: 1 event: 5595 at 23-Sep-2025 09:49:57 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 27!
Begin processing the 696th record. run: 50264716 subRun: 1 event: 5596 at 23-Sep-2025 09:50:05 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 685!
Begin processing the 697th record. run: 50264716 subRun: 1 event: 5597 at 23-Sep-2025 09:50:15 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 782!
Begin processing the 698th record. run: 50264716 subRun: 1 event: 5598 at 23-Sep-2025 09:50:24 -03
Boundary wire vector sizes: 979, 1252, 1055
minwire 0: 412
minwire 1: 2451
minwire 2: 16
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.158506, 0.483653, 0.357841, 
Output 1: 0.394542, 0.569227, 0.0316916, 0.00453899, 
Output 2: 0.0354532, 0.157163, 0.422559, 0.384825, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2339!
Begin processing the 699th record. run: 50264716 subRun: 1 event: 5599 at 23-Sep-2025 09:50:38 -03
Boundary wire vector sizes: 113, 83, 56
minwire 0: 1158
minwire 1: 2115
minwire 2: 838
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.957788, 0.00124696, 0.040965, 
Output 1: 0.862235, 0.135602, 0.00204119, 0.000121241, 
Output 2: 0.962339, 0.0370023, 0.000634747, 2.40405e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1744!
Begin processing the 700th record. run: 50264716 subRun: 1 event: 5600 at 23-Sep-2025 09:50:48 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 584!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c9/c1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_353_20231203T180402Z_gen_g4_detsim_hitreco__20240509T215025Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2c/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_492_20231207T025255Z_gen_g4_detsim_hitreco__20240510T050830Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/77/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_769_20231207T001723Z_gen_g4_detsim_hitreco__20240510T043319Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/67/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_397_20231201T170319Z_gen_g4_detsim_hitreco__20240507T201101Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f0/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_414_20231118T191900Z_gen_g4_detsim_hitreco__20240503T060447Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d4/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50264716_55_20231122T125719Z_gen_g4_detsim_hitreco__20240507T180717Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/1d/4e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_108_20231204T043309Z_gen_g4_detsim_hitreco__20240509T202832Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b5/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_786_20231207T152046Z_gen_g4_detsim_hitreco__20240510T061752Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/fb/81/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_713_20231208T084302Z_gen_g4_detsim_hitreco__20240510T060533Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7a/89/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_205_20231205T020636Z_gen_g4_detsim_hitreco__20240509T222712Z_reco2.root
.:
total 37504
-rw-r--r-- 1 nobody nobody 14124321 Sep 23 09:58 flatcaf.root
-rw-r--r-- 1 nobody nobody 11806332 Sep 23 09:58 caf.root
-rw-r--r-- 1 nobody nobody 11806332 Sep 23 09:58 caf_fd_hd_atmo_3301_20250923T095737Z.root
-rw-r--r-- 1 nobody nobody   419933 Sep 23 09:58 caf_20250923T095737Z.log
-rw-r--r-- 1 nobody nobody   165579 Sep 23 09:58 jobscript.log
-rw-r--r-- 1 nobody nobody     1950 Sep 23 09:58 caf_20250923T095737Z.file
-rw-r--r-- 1 nobody nobody     1950 Sep 23 09:58 caf_20250923T095737Z.pfns
-rw-r--r-- 1 nobody nobody     1950 Sep 23 06:57 file.list
-rw-r--r-- 1 nobody nobody     1950 Sep 23 09:58 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1376 Sep 23 06:57 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1376 Sep 23 09:58 caf_20250923T095737Z.did
-rw-r--r-- 1 nobody nobody     1376 Sep 23 06:57 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 23 06:58 debugprod.log