Jobsub ID 106434.31@dunegpschedd01.fnal.gov

Jobsub ID	106434.31@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:57:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-23 12:03:59
From worker node	Hostname	wn77
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:04:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_122_20231202T073144Z_gen_g4_detsim_hitreco__20240507T204700Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_533_20231203T102049Z_gen_g4_detsim_hitreco__20240509T195543Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_820_20231203T030309Z_gen_g4_detsim_hitreco__20240508T072242Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_830_20231205T070800Z_gen_g4_detsim_hitreco__20240509T221859Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_434_20231201T124224Z_gen_g4_detsim_hitreco__20240507T190418Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_40_20231202T064213Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_920_20231203T070325Z_gen_g4_detsim_hitreco__20240508T080816Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_562_20231207T091439Z_gen_g4_detsim_hitreco__20240510T040336Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_292_20231202T124557Z_gen_g4_detsim_hitreco__20240507T220656Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_489_20231201T105751Z_gen_g4_detsim_hitreco__20240507T193202Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 12:03:59
Saved logs	justin-logs:106434.31-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.95249      0.216933        500    
reco:ophitspe:OpHitFinderDeco                           1.41317       1.68274       1.86111       1.77595      0.178823        500    
reco:opflash:OpFlashFinder                            0.000166487   0.00173915    0.00589872    0.00164445    0.00080457       500    
reco:opslicer:OpSlicer                                6.3173e-05     0.371478       1.61937      0.291011      0.311296        500    
reco:rns:RandomNumberSaver                            2.4021e-05    3.88608e-05   0.00047013    3.6732e-05    2.22985e-05      500    
reco:anglereconue:NuAngularReco                       0.000201959   0.00224256     0.100764     0.000526573   0.00704599       500    
reco:anglereconumu:NuAngularReco                      0.000123236   0.00208512     0.105362     0.000362621   0.00715015       500    
reco:anglereconuepfps:NuAngularReco                   0.000175198   0.00219515      0.10083     0.000483206   0.00700277       500    
reco:anglereconumupfps:NuAngularReco                  0.000167468   0.00217067     0.101638     0.00046844     0.007013        500    
reco:anglerecohits:NuAngularReco                      0.000155307   0.00301392     0.108522     0.000988923   0.00780892       500    
[art]:TriggerResults:TriggerResultInserter            1.0661e-05    1.79226e-05   0.000117956   1.7311e-05    6.25654e-06      500    
end_path:cafmaker:CAFMaker                            0.00154554     0.0276121      1.05032     0.00573984     0.0866077       500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3651.02 MB
  Peak resident set size usage (VmHWM): 1971.99 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 08:59:42 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/77/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_40_20231202T064213Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 0: 1077
minwire 1: 708
minwire 2: 1109
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995852, 0.000252623, 0.00389571, 
Output 1: 0.993026, 0.00627676, 0.000445667, 0.000251381, 
Output 2: 0.9902, 0.00949843, 0.000248113, 5.30039e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1802!
Begin processing the 495th record. run: 50515285 subRun: 1 event: 43495 at 23-Sep-2025 08:51:13 -03
Boundary wire vector sizes: 407, 424, 398
minwire 0: 1678
minwire 1: 195
minwire 2: 1786
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.969004, 0.00791635, 0.02308, 
Output 1: 0.0750888, 0.900368, 0.0242285, 0.000315115, 
Output 2: 0.0169127, 0.3373, 0.585987, 0.0598005, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1720!
Begin processing the 496th record. run: 50515285 subRun: 1 event: 43496 at 23-Sep-2025 08:51:24 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 81!
Begin processing the 497th record. run: 50515285 subRun: 1 event: 43497 at 23-Sep-2025 08:51:32 -03
Boundary wire vector sizes: 545, 619, 489
minwire 0: 1378
minwire 1: 1970
minwire 2: 862
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0021415, 0.985836, 0.0120228, 
Output 1: 0.252951, 0.729837, 0.0164637, 0.000748424, 
Output 2: 0.923331, 0.0751624, 0.00147095, 3.53714e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2728!
Begin processing the 498th record. run: 50515285 subRun: 1 event: 43498 at 23-Sep-2025 08:51:42 -03
Boundary wire vector sizes: 268, 375, 276
minwire 0: 2463
minwire 1: 273
minwire 2: 2688
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.683142, 0.00705846, 0.309799, 
Output 1: 0.757442, 0.2354, 0.00682818, 0.00032981, 
Output 2: 0.536597, 0.450419, 0.0126483, 0.000335087, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1267!
Begin processing the 499th record. run: 50515285 subRun: 1 event: 43499 at 23-Sep-2025 08:51:52 -03
Boundary wire vector sizes: 254, 269, 193
minwire 0: 2177
minwire 1: 1488
minwire 2: 1778
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996631, 0.000602395, 0.00276654, 
Output 1: 0.0243493, 0.968779, 0.00672085, 0.000150746, 
Output 2: 0.997755, 0.00218801, 5.56897e-05, 1.17212e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2037!
Begin processing the 500th record. run: 50515285 subRun: 1 event: 43500 at 23-Sep-2025 08:52:02 -03
Boundary wire vector sizes: 88, 28, 65
minwire 0: 2110
minwire 1: 237
minwire 2: 2454
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.355743, 0.00313489, 0.641122, 
Output 1: 0.238216, 0.754074, 0.0073145, 0.000395417, 
Output 2: 0.282632, 0.709786, 0.0075135, 6.80555e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 792!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/4e/a3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400837_122_20231202T073144Z_gen_g4_detsim_hitreco__20240507T204700Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5d/57/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_533_20231203T102049Z_gen_g4_detsim_hitreco__20240509T195543Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/eb/e1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_820_20231203T030309Z_gen_g4_detsim_hitreco__20240508T072242Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bb/e0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_830_20231205T070800Z_gen_g4_detsim_hitreco__20240509T221859Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e4/7b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515285_434_20231201T124224Z_gen_g4_detsim_hitreco__20240507T190418Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/77/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_40_20231202T064213Z_gen_g4_detsim_hitreco__20240507T195405Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c4/cd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_920_20231203T070325Z_gen_g4_detsim_hitreco__20240508T080816Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5b/0c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_562_20231207T091439Z_gen_g4_detsim_hitreco__20240510T040336Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/25/d9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_292_20231202T124557Z_gen_g4_detsim_hitreco__20240507T220656Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bd/4a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_489_20231201T105751Z_gen_g4_detsim_hitreco__20240507T193202Z_reco2.root
.:
total 23776
-rw-r--r-- 1 nobody nobody 8954159 Sep 23 08:59 flatcaf.root
-rw-r--r-- 1 nobody nobody 7417893 Sep 23 08:59 caf.root
-rw-r--r-- 1 nobody nobody 7417893 Sep 23 08:59 caf_fd_hd_atmo_3302_20250923T100421Z.root
-rw-r--r-- 1 nobody nobody  306210 Sep 23 08:59 caf_20250923T100421Z.log
-rw-r--r-- 1 nobody nobody  133573 Sep 23 08:59 jobscript.log
-rw-r--r-- 1 nobody nobody    2010 Sep 23 08:59 caf_20250923T100421Z.file
-rw-r--r-- 1 nobody nobody    2010 Sep 23 08:59 caf_20250923T100421Z.pfns
-rw-r--r-- 1 nobody nobody    2010 Sep 23 07:04 file.list
-rw-r--r-- 1 nobody nobody    2010 Sep 23 08:59 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:04 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1376 Sep 23 08:59 caf_20250923T100421Z.did
-rw-r--r-- 1 nobody nobody    1376 Sep 23 07:04 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:04 debugprod.log