Jobsub ID 106434.6@dunegpschedd01.fnal.gov

Jobsub ID	106434.6@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:57:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce04
Last heartbeat	2025-09-23 13:42:08
From worker node	Hostname	wn125
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:01:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_691_20231202T104829Z_gen_g4_detsim_hitreco__20240507T200707Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_104_20231118T173324Z_gen_g4_detsim_hitreco__20240503T051601Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_526_20231207T155400Z_gen_g4_detsim_hitreco__20240510T030745Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_117_20231121T150942Z_gen_g4_detsim_hitreco__20240503T065146Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_339_20231205T012028Z_gen_g4_detsim_hitreco__20240509T222445Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_494_20231203T114526Z_gen_g4_detsim_hitreco__20240509T195346Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_746_20231204T100504Z_gen_g4_detsim_hitreco__20240509T201234Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_898_20231203T030443Z_gen_g4_detsim_hitreco__20240508T072516Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370890_424_20231201T083539Z_gen_g4_detsim_hitreco__20240507T212629Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_287_20231201T170155Z_gen_g4_detsim_hitreco__20240507T190956Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 13:42:08
Saved logs	justin-logs:106434.6-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1.76699       1.81928       1.98761       1.84997      0.0430825       600    
reco:opflash:OpFlashFinder                            0.000224198    0.001801     0.00462417    0.00177141    0.000858193      600    
reco:opslicer:OpSlicer                                6.7803e-05     0.391154       1.59818      0.328149       0.32296        600    
reco:rns:RandomNumberSaver                            2.3941e-05    3.72364e-05   0.000350853   3.58565e-05   1.67386e-05      600    
reco:anglereconue:NuAngularReco                       0.000196757    0.0029301     0.094615     0.000533174   0.00860651       600    
reco:anglereconumu:NuAngularReco                      0.000103333   0.00271211     0.0907042    0.000372303    0.0083518       600    
reco:anglereconuepfps:NuAngularReco                   0.000154766   0.00283925     0.0938874    0.000488133   0.00845549       600    
reco:anglereconumupfps:NuAngularReco                  0.000146235   0.00281118     0.0927622    0.000468892   0.00842375       600    
reco:anglerecohits:NuAngularReco                      0.000157746   0.00375164     0.101984     0.000995547   0.00954358       600    
[art]:TriggerResults:TriggerResultInserter             1.045e-05    1.7439e-05    5.8732e-05    1.69955e-05   5.36612e-06      600    
end_path:cafmaker:CAFMaker                            0.00138354     0.0364777      1.23553     0.00572129     0.112176        600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3803.1 MB
  Peak resident set size usage (VmHWM): 2120.04 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 10:37:52 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/1c/93/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_339_20231205T012028Z_gen_g4_detsim_hitreco__20240509T222445Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.00194881, 0.920668, 0.0773829, 
Output 1: 0.961463, 0.0366258, 0.00172716, 0.000184573, 
Output 2: 0.964887, 0.0343896, 0.000670761, 5.28686e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1136!
Begin processing the 594th record. run: 65840069 subRun: 1 event: 23594 at 23-Sep-2025 10:26:56 -03
Boundary wire vector sizes: 120, 108, 48
minwire 0: 1358
minwire 1: 827
minwire 2: 1631
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.97488, 0.000596246, 0.0245235, 
Output 1: 0.990932, 0.00869276, 0.000263215, 0.000111865, 
Output 2: 0.982243, 0.0173366, 0.000357732, 6.29008e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1163!
Begin processing the 595th record. run: 65840069 subRun: 1 event: 23595 at 23-Sep-2025 10:27:07 -03
Boundary wire vector sizes: 8589, 9545, 5075
minwire 0: 605
minwire 1: 408
minwire 2: 514
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.238148, 0.754447, 0.00740412, 
Output 1: 0.293039, 0.414643, 0.146192, 0.146125, 
Output 2: 0.0195413, 0.0762952, 0.271342, 0.632822, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3627!
Begin processing the 596th record. run: 65840069 subRun: 1 event: 23596 at 23-Sep-2025 10:29:21 -03
Boundary wire vector sizes: 450, 381, 337
minwire 0: 1003
minwire 1: 1254
minwire 2: 625
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00221222, 0.970015, 0.0277732, 
Output 1: 0.623608, 0.340151, 0.0338738, 0.00236706, 
Output 2: 0.0197815, 0.904018, 0.0736915, 0.00250929, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2894!
Begin processing the 597th record. run: 65840069 subRun: 1 event: 23597 at 23-Sep-2025 10:29:37 -03
Boundary wire vector sizes: 199, 175, 145
minwire 0: 1865
minwire 1: 101
minwire 2: 2083
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 213, 512
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0236138, 0.00240421, 0.973982, 
Output 1: 0.0366305, 0.0726656, 0.168096, 0.722608, 
Output 2: 0.840164, 0.147859, 0.0106393, 0.00133681, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1446!
Begin processing the 598th record. run: 65840069 subRun: 1 event: 23598 at 23-Sep-2025 10:29:49 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 33!
Begin processing the 599th record. run: 65840069 subRun: 1 event: 23599 at 23-Sep-2025 10:29:59 -03
Boundary wire vector sizes: 60, 56, 51
minwire 0: 1103
minwire 1: 1127
minwire 2: 926
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 555, 854
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0448808, 0.00374235, 0.951377, 
Output 1: 0.695204, 0.281867, 0.0212341, 0.00169537, 
Output 2: 0.985421, 0.0140581, 0.000486666, 3.3773e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1885!
Begin processing the 600th record. run: 65840069 subRun: 1 event: 23600 at 23-Sep-2025 10:30:11 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 990!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/08/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_691_20231202T104829Z_gen_g4_detsim_hitreco__20240507T200707Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f8/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_104_20231118T173324Z_gen_g4_detsim_hitreco__20240503T051601Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e8/4f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_526_20231207T155400Z_gen_g4_detsim_hitreco__20240510T030745Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d3/0f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_117_20231121T150942Z_gen_g4_detsim_hitreco__20240503T065146Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/1c/93/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_339_20231205T012028Z_gen_g4_detsim_hitreco__20240509T222445Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ec/6d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_494_20231203T114526Z_gen_g4_detsim_hitreco__20240509T195346Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/63/0a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_746_20231204T100504Z_gen_g4_detsim_hitreco__20240509T201234Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/08/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_898_20231203T030443Z_gen_g4_detsim_hitreco__20240508T072516Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/44/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370890_424_20231201T083539Z_gen_g4_detsim_hitreco__20240507T212629Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/ae/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_287_20231201T170155Z_gen_g4_detsim_hitreco__20240507T190956Z_reco2.root
.:
total 34328
-rw-r--r-- 1 nobody nobody 12921114 Sep 23 10:37 flatcaf.root
-rw-r--r-- 1 nobody nobody 10812620 Sep 23 10:37 caf.root
-rw-r--r-- 1 nobody nobody 10812620 Sep 23 10:37 caf_fd_hd_atmo_3302_20250923T100125Z.root
-rw-r--r-- 1 nobody nobody   362242 Sep 23 10:37 caf_20250923T100125Z.log
-rw-r--r-- 1 nobody nobody   141331 Sep 23 10:37 jobscript.log
-rw-r--r-- 1 nobody nobody     1964 Sep 23 10:37 caf_20250923T100125Z.file
-rw-r--r-- 1 nobody nobody     1964 Sep 23 10:37 caf_20250923T100125Z.pfns
-rw-r--r-- 1 nobody nobody     1964 Sep 23 07:01 file.list
-rw-r--r-- 1 nobody nobody     1964 Sep 23 10:37 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1378 Sep 23 07:01 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1378 Sep 23 10:37 caf_20250923T100125Z.did
-rw-r--r-- 1 nobody nobody     1378 Sep 23 07:01 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 23 07:01 debugprod.log