Jobsub ID 106434.30@dunegpschedd01.fnal.gov

Jobsub ID	106434.30@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:57:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce04
Last heartbeat	2025-09-23 11:42:02
From worker node	Hostname	wn123
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:04:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_90_20231204T140258Z_gen_g4_detsim_hitreco__20240509T204647Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_906_20231205T091027Z_gen_g4_detsim_hitreco__20240510T022917Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1793_481_20231027T180638Z_gen_g4_detsim_hitreco__20240503T041542Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_23_20231205T014859Z_gen_g4_detsim_hitreco__20240509T221159Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_152_20231205T115557Z_gen_g4_detsim_hitreco__20240510T062657Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_173_20231122T063721Z_gen_g4_detsim_hitreco__20240503T070257Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_581_20231202T064404Z_gen_g4_detsim_hitreco__20240507T204247Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_440_20231203T231108Z_gen_g4_detsim_hitreco__20240509T204955Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_285_20231203T131853Z_gen_g4_detsim_hitreco__20240508T072547Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_162_20231202T043738Z_gen_g4_detsim_hitreco__20240507T202809Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 11:42:02
Saved logs	justin-logs:106434.30-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nergyReco                  0.000685143    0.0358227      1.68571     0.00448348     0.136431        400    
reco:opdec:Deconvolution                               0.603803       1.07941       4.18972       1.01728      0.296797        400    
reco:ophitspe:OpHitFinderDeco                           1.77193       1.82497       1.88958       1.84825      0.0350045       400    
reco:opflash:OpFlashFinder                            0.000237351   0.00166822     0.0050084    0.00157429    0.000742099      400    
reco:opslicer:OpSlicer                                0.00021991     0.355178       1.27105      0.285015      0.285572        400    
reco:rns:RandomNumberSaver                            2.2011e-05    3.77584e-05   0.000442422   3.5727e-05    2.18771e-05      400    
reco:anglereconue:NuAngularReco                       0.000155668   0.00176735     0.111239     0.000468227    0.0072032       400    
reco:anglereconumu:NuAngularReco                      7.8854e-05    0.00162183     0.110487     0.000331392   0.00713592       400    
reco:anglereconuepfps:NuAngularReco                   0.000150497   0.00174561     0.112381     0.000446242   0.00723957       400    
reco:anglereconumupfps:NuAngularReco                  0.000107656   0.00173107     0.111855     0.000422691   0.00726328       400    
reco:anglerecohits:NuAngularReco                      0.000110275   0.00246194     0.116435     0.000973277   0.00773049       400    
[art]:TriggerResults:TriggerResultInserter             1.033e-05    1.78504e-05   0.00021337    1.69805e-05   1.12467e-05      400    
end_path:cafmaker:CAFMaker                            0.00147344     0.0225411      1.41777      0.0050671     0.0921238       400    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3649.05 MB
  Peak resident set size usage (VmHWM): 1957.52 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 08:37:45 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/61/28/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_23_20231205T014859Z_gen_g4_detsim_hitreco__20240509T221159Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 737!
Begin processing the 392nd record. run: 1793 subRun: 1 event: 96392 at 23-Sep-2025 08:33:10 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 571!
Begin processing the 393rd record. run: 1793 subRun: 1 event: 96393 at 23-Sep-2025 08:33:20 -03
Boundary wire vector sizes: 320, 590, 418
minwire 0: 1286
minwire 1: 1593
minwire 2: 1084
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0692552, 0.0357357, 0.895009, 
Output 1: 0.125815, 0.823218, 0.0479777, 0.0029897, 
Output 2: 0.0524144, 0.408423, 0.467399, 0.071763, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2884!
Begin processing the 394th record. run: 1793 subRun: 1 event: 96394 at 23-Sep-2025 08:33:32 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 721!
Begin processing the 395th record. run: 1793 subRun: 1 event: 96395 at 23-Sep-2025 08:33:41 -03
Boundary wire vector sizes: 276, 263, 104
minwire 0: 465
minwire 1: 1498
minwire 2: 383
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99512, 0.000392157, 0.00448795, 
Output 1: 0.00642119, 0.984411, 0.00900967, 0.00015848, 
Output 2: 0.996415, 0.00349298, 9.02872e-05, 1.90515e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2312!
Begin processing the 396th record. run: 1793 subRun: 1 event: 96396 at 23-Sep-2025 08:33:52 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 615!
Begin processing the 397th record. run: 1793 subRun: 1 event: 96397 at 23-Sep-2025 08:34:02 -03
Boundary wire vector sizes: 307, 78, 210
minwire 0: 1588
minwire 1: 539
minwire 2: 1595
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.914839, 0.00180923, 0.0833515, 
Output 1: 0.897956, 0.0985123, 0.00334436, 0.000187663, 
Output 2: 0.186479, 0.795421, 0.0178552, 0.00024514, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1458!
Begin processing the 398th record. run: 1793 subRun: 1 event: 96398 at 23-Sep-2025 08:34:12 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1376!
Begin processing the 399th record. run: 1793 subRun: 1 event: 96399 at 23-Sep-2025 08:34:22 -03
Boundary wire vector sizes: 493, 536, 481
minwire 0: 537
minwire 1: 1961
minwire 2: 174
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0517385, 0.492616, 0.455645, 
Output 1: 0.0220919, 0.96629, 0.0112223, 0.000395693, 
Output 2: 0.787292, 0.206237, 0.00634899, 0.000121671, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2187!
Begin processing the 400th record. run: 1793 subRun: 1 event: 96400 at 23-Sep-2025 08:34:34 -03
Boundary wire vector sizes: 1956, 1958, 1310
minwire 0: 1757
minwire 1: 524
minwire 2: 1842
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00479344, 0.714962, 0.280244, 
Output 1: 0.0660594, 0.900858, 0.0313436, 0.00173873, 
Output 2: 0.942598, 0.0553721, 0.0019404, 8.89662e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2611!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b2/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_90_20231204T140258Z_gen_g4_detsim_hitreco__20240509T204647Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/28/b6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_906_20231205T091027Z_gen_g4_detsim_hitreco__20240510T022917Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2b/16/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1793_481_20231027T180638Z_gen_g4_detsim_hitreco__20240503T041542Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/61/28/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_23_20231205T014859Z_gen_g4_detsim_hitreco__20240509T221159Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/6a/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_152_20231205T115557Z_gen_g4_detsim_hitreco__20240510T062657Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/51/f2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_173_20231122T063721Z_gen_g4_detsim_hitreco__20240503T070257Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/a9/d8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_581_20231202T064404Z_gen_g4_detsim_hitreco__20240507T204247Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bd/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_440_20231203T231108Z_gen_g4_detsim_hitreco__20240509T204955Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/51/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6414162_285_20231203T131853Z_gen_g4_detsim_hitreco__20240508T072547Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d0/cd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_162_20231202T043738Z_gen_g4_detsim_hitreco__20240507T202809Z_reco2.root
.:
total 17768
-rw-r--r-- 1 nobody nobody 6719160 Sep 23 08:37 flatcaf.root
-rw-r--r-- 1 nobody nobody 5524098 Sep 23 08:37 caf.root
-rw-r--r-- 1 nobody nobody 5524098 Sep 23 08:37 caf_fd_hd_atmo_3302_20250923T100428Z.root
-rw-r--r-- 1 nobody nobody  247305 Sep 23 08:37 caf_20250923T100428Z.log
-rw-r--r-- 1 nobody nobody  101190 Sep 23 08:37 jobscript.log
-rw-r--r-- 1 nobody nobody    1924 Sep 23 08:37 caf_20250923T100428Z.file
-rw-r--r-- 1 nobody nobody    1924 Sep 23 08:37 caf_20250923T100428Z.pfns
-rw-r--r-- 1 nobody nobody    1924 Sep 23 07:04 file.list
-rw-r--r-- 1 nobody nobody    1924 Sep 23 08:37 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1370 Sep 23 07:04 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1370 Sep 23 08:37 caf_20250923T100428Z.did
-rw-r--r-- 1 nobody nobody    1370 Sep 23 07:04 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:04 debugprod.log