Jobsub ID 52233.4@dunegpschedd01.fnal.gov

Jobsub ID	52233.4@dunegpschedd01.fnal.gov
Workflow ID	3298
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-22 12:33:32
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-22 15:10:15
From worker node	Hostname	wn65
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-22 12:34:22
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_113_20231028T031318Z_gen_g4_detsim_hitreco__20240503T045754Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_601_20231208T004816Z_gen_g4_detsim_hitreco__20240510T031628Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_1171_20231204T005724Z_gen_g4_detsim_hitreco__20240509T203704Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_572_20231201T171800Z_gen_g4_detsim_hitreco__20240507T191505Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_158_20231121T151006Z_gen_g4_detsim_hitreco__20240503T065929Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_67_20231202T195940Z_gen_g4_detsim_hitreco__20240508T062630Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_290_20231122T120048Z_gen_g4_detsim_hitreco__20240503T055421Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_410_20231207T032138Z_gen_g4_detsim_hitreco__20240510T024658Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370890_66_20231201T073712Z_gen_g4_detsim_hitreco__20240507T201918Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_235_20231201T122207Z_gen_g4_detsim_hitreco__20240507T192711Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-22 15:10:15
Saved logs	justin-logs:52233.4-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.00435351     0.188825        700    
reco:opdec:Deconvolution                               0.476078      0.919386       2.45379      0.907477       0.21142        700    
reco:ophitspe:OpHitFinderDeco                           1.38859       1.6292        1.8922        1.5196        0.20239        700    
reco:opflash:OpFlashFinder                            0.000220699   0.00167639    0.00620368    0.00160579    0.000776625      700    
reco:opslicer:OpSlicer                                7.1243e-05     0.350049       1.65534      0.278976      0.290287        700    
reco:rns:RandomNumberSaver                            2.3021e-05    3.71681e-05   0.000491621   3.5471e-05     1.912e-05       700    
reco:anglereconue:NuAngularReco                       0.000198008   0.00233519     0.165574     0.000471649    0.0106096       700    
reco:anglereconumu:NuAngularReco                      9.4824e-05    0.00219686     0.164098     0.000331004    0.0106395       700    
reco:anglereconuepfps:NuAngularReco                   0.000161727   0.00229158     0.171563     0.000441819    0.0107151       700    
reco:anglereconumupfps:NuAngularReco                  0.000147126   0.00226247      0.16671     0.000415898    0.0106208       700    
reco:anglerecohits:NuAngularReco                      0.000130995   0.00304775      0.17386     0.000899237    0.0115333       700    
[art]:TriggerResults:TriggerResultInserter             9.581e-06    1.70992e-05   0.000113124    1.639e-05    6.74887e-06      700    
end_path:cafmaker:CAFMaker                            0.00149735     0.0281803      2.34793     0.00508789     0.129436        700    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3778.53 MB
  Peak resident set size usage (VmHWM): 2086.82 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 22-Sep-2025 12:05:57 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/45/a2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_67_20231202T195940Z_gen_g4_detsim_hitreco__20240508T062630Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1411!
Begin processing the 692nd record. run: 65840069 subRun: 1 event: 31792 at 22-Sep-2025 12:02:05 -03
Boundary wire vector sizes: 116, 147, 81
minwire 0: 2449
minwire 1: 439
minwire 2: 2553
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000210025, 0.998753, 0.00103659, 
Output 1: 0.655554, 0.340886, 0.00320575, 0.000354941, 
Output 2: 0.999029, 0.000952694, 1.7416e-05, 9.79707e-07, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1011!
Begin processing the 693rd record. run: 65840069 subRun: 1 event: 31793 at 22-Sep-2025 12:02:16 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 358!
Begin processing the 694th record. run: 65840069 subRun: 1 event: 31794 at 22-Sep-2025 12:02:25 -03
Boundary wire vector sizes: 4581, 3116, 3040
minwire 0: 16
minwire 1: 1414
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.702332, 0.134949, 0.162719, 
Output 1: 0.19145, 0.370409, 0.222511, 0.215629, 
Output 2: 0.0131228, 0.0400759, 0.167316, 0.779485, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2388!
Begin processing the 695th record. run: 65840069 subRun: 1 event: 31795 at 22-Sep-2025 12:02:51 -03
Boundary wire vector sizes: 233, 165, 90
minwire 0: 2159
minwire 1: 320
minwire 2: 2549
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996984, 0.000531206, 0.00248471, 
Output 1: 0.00591942, 0.988301, 0.00564937, 0.00013001, 
Output 2: 0.996538, 0.0033925, 6.85109e-05, 1.23271e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 974!
Begin processing the 696th record. run: 65840069 subRun: 1 event: 31796 at 22-Sep-2025 12:03:02 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 239!
Begin processing the 697th record. run: 65840069 subRun: 1 event: 31797 at 22-Sep-2025 12:03:11 -03
Boundary wire vector sizes: 65, 67, 67
minwire 0: 453
minwire 1: 1964
minwire 2: 340
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.166907, 0.828249, 0.00484433, 
Output 1: 0.370101, 0.62415, 0.00559769, 0.000151459, 
Output 2: 0.996677, 0.00322846, 8.92909e-05, 5.00024e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 810!
Begin processing the 698th record. run: 65840069 subRun: 1 event: 31798 at 22-Sep-2025 12:03:22 -03
Boundary wire vector sizes: 282, 382, 262
minwire 0: 930
minwire 1: 1232
minwire 2: 1003
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.426059, 0.00494413, 0.568996, 
Output 1: 0.0197941, 0.136226, 0.275296, 0.568684, 
Output 2: 0.0988227, 0.43709, 0.348481, 0.115606, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2149!
Begin processing the 699th record. run: 65840069 subRun: 1 event: 31799 at 22-Sep-2025 12:03:33 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 203!
Begin processing the 700th record. run: 65840069 subRun: 1 event: 31800 at 22-Sep-2025 12:03:43 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 636!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/eb/f9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_113_20231028T031318Z_gen_g4_detsim_hitreco__20240503T045754Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5c/dd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_601_20231208T004816Z_gen_g4_detsim_hitreco__20240510T031628Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/98/b3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_1171_20231204T005724Z_gen_g4_detsim_hitreco__20240509T203704Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/9a/24/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_572_20231201T171800Z_gen_g4_detsim_hitreco__20240507T191505Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f5/d6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_158_20231121T151006Z_gen_g4_detsim_hitreco__20240503T065929Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/45/a2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_67_20231202T195940Z_gen_g4_detsim_hitreco__20240508T062630Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b8/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_290_20231122T120048Z_gen_g4_detsim_hitreco__20240503T055421Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4d/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_410_20231207T032138Z_gen_g4_detsim_hitreco__20240510T024658Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/45/d8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370890_66_20231201T073712Z_gen_g4_detsim_hitreco__20240507T201918Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/34/f1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_235_20231201T122207Z_gen_g4_detsim_hitreco__20240507T192711Z_reco2.root
.:
total 32136
-rw-r--r-- 1 nobody nobody 12104492 Sep 22 12:05 flatcaf.root
-rw-r--r-- 1 nobody nobody 10064281 Sep 22 12:05 caf.root
-rw-r--r-- 1 nobody nobody 10064281 Sep 22 12:05 caf_fd_hd_atmo_3298_20250922T123438Z.root
-rw-r--r-- 1 nobody nobody   429441 Sep 22 12:05 caf_20250922T123438Z.log
-rw-r--r-- 1 nobody nobody   144621 Sep 22 12:05 jobscript.log
-rw-r--r-- 1 nobody nobody     1952 Sep 22 12:05 caf_20250922T123438Z.file
-rw-r--r-- 1 nobody nobody     1952 Sep 22 12:05 caf_20250922T123438Z.pfns
-rw-r--r-- 1 nobody nobody     1952 Sep 22 09:34 file.list
-rw-r--r-- 1 nobody nobody     1952 Sep 22 12:05 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1372 Sep 22 09:34 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1372 Sep 22 12:05 caf_20250922T123438Z.did
-rw-r--r-- 1 nobody nobody     1372 Sep 22 09:34 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 22 09:35 debugprod.log