Jobsub ID 46124.24@dunegpschedd02.fnal.gov

Jobsub ID	46124.24@dunegpschedd02.fnal.gov
Workflow ID	3301
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-22 18:22:48
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-22 19:03:32
From worker node	Hostname	wn65
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-22 18:32:41
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_319_20231204T002707Z_gen_g4_detsim_hitreco__20240509T204921Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_40_20231208T072618Z_gen_g4_detsim_hitreco__20240510T054048Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_225_20231120T134053Z_gen_g4_detsim_hitreco__20240503T055557Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_394_20231208T064238Z_gen_g4_detsim_hitreco__20240510T054802Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_8_20231205T071616Z_gen_g4_detsim_hitreco__20240510T042442Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1036_20231203T235418Z_gen_g4_detsim_hitreco__20240509T203147Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_603_20231201T212752Z_gen_g4_detsim_hitreco__20240508T045628Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_396_20231120T135035Z_gen_g4_detsim_hitreco__20240503T061342Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_316_20231205T024901Z_gen_g4_detsim_hitreco__20240509T222847Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_620_20231202T162424Z_gen_g4_detsim_hitreco__20240508T045120Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-22 19:03:32
Saved logs	justin-logs:46124.24-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.74574      0.867956      0.202206        100    
reco:ophitspe:OpHitFinderDeco                           1.50146       1.51078       1.52181       1.51085     0.00435663       100    
reco:opflash:OpFlashFinder                            0.000177277   0.00171451    0.00503518    0.00166528    0.00092088       100    
reco:opslicer:OpSlicer                                5.2382e-05     0.379666       1.43714      0.294522      0.326667        100    
reco:rns:RandomNumberSaver                            2.3611e-05    4.05576e-05   0.000417247   3.6582e-05    3.85013e-05      100    
reco:anglereconue:NuAngularReco                       0.000192948   0.00249775     0.0565437    0.000521856   0.00734215       100    
reco:anglereconumu:NuAngularReco                      0.000107694   0.00238074     0.0591352    0.000388715   0.00757177       100    
reco:anglereconuepfps:NuAngularReco                   0.000158897   0.00250947     0.058003     0.000498601   0.00751601       100    
reco:anglereconumupfps:NuAngularReco                  0.000159157   0.00243813     0.055109     0.000473484   0.00731778       100    
reco:anglerecohits:NuAngularReco                      0.000122506   0.00335451     0.0632361    0.00109052    0.00828332       100    
[art]:TriggerResults:TriggerResultInserter             1.126e-05    1.85667e-05   8.6243e-05    1.7255e-05    8.92337e-06      100    
end_path:cafmaker:CAFMaker                            0.00138446     0.0330091     0.762715     0.00585025     0.0960271       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3703.55 MB
  Peak resident set size usage (VmHWM): 2019.46 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 22-Sep-2025 15:58:30 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/0f/e0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_40_20231208T072618Z_gen_g4_detsim_hitreco__20240510T054048Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 1: 698
minwire 2: 2358
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000575928, 0.992229, 0.00719476, 
Output 1: 0.0631336, 0.896442, 0.0385848, 0.00183948, 
Output 2: 0.992197, 0.00744941, 0.000334742, 1.86199e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1849!
Begin processing the 95th record. run: 6419587 subRun: 1 event: 31995 at 22-Sep-2025 15:55:33 -03
Boundary wire vector sizes: 528, 664, 577
minwire 0: 350
minwire 1: 1934
minwire 2: 401
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00896208, 0.980142, 0.0108956, 
Output 1: 0.0137083, 0.225783, 0.631733, 0.128776, 
Output 2: 0.363877, 0.495236, 0.132267, 0.00861937, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1505!
Begin processing the 96th record. run: 6419587 subRun: 1 event: 31996 at 22-Sep-2025 15:55:43 -03
Boundary wire vector sizes: 567, 402, 371
minwire 0: 1890
minwire 1: 1014
minwire 2: 1837
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000578611, 0.995659, 0.00376248, 
Output 1: 0.977624, 0.0219198, 0.000365911, 9.00741e-05, 
Output 2: 0.996364, 0.00354594, 7.88092e-05, 1.16713e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2312!
Begin processing the 97th record. run: 6419587 subRun: 1 event: 31997 at 22-Sep-2025 15:55:54 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 341!
Begin processing the 98th record. run: 6419587 subRun: 1 event: 31998 at 22-Sep-2025 15:56:02 -03
Boundary wire vector sizes: 62, 60, 40
minwire 0: 1814
minwire 1: 474
minwire 2: 2252
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 503, 802
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0275928, 0.965822, 0.00658528, 
Output 1: 0.957512, 0.042057, 0.000318878, 0.000112398, 
Output 2: 0.998286, 0.00163479, 5.88867e-05, 2.00892e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1201!
Begin processing the 99th record. run: 6419587 subRun: 1 event: 31999 at 22-Sep-2025 15:56:12 -03
Boundary wire vector sizes: 128, 132, 105
minwire 0: 2691
minwire 1: 397
minwire 2: 2717
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000795769, 0.996928, 0.0022762, 
Output 1: 0.0071207, 0.114426, 0.833992, 0.0444611, 
Output 2: 0.99073, 0.00875338, 0.000504771, 1.22305e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1088!
Begin processing the 100th record. run: 6419587 subRun: 1 event: 32000 at 22-Sep-2025 15:56:21 -03
Boundary wire vector sizes: 226, 218, 245
minwire 0: 346
minwire 1: 2328
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00412172, 0.94145, 0.0544279, 
Output 1: 0.0240438, 0.910353, 0.062875, 0.00272829, 
Output 2: 0.579841, 0.407899, 0.012057, 0.000202751, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1426!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e6/04/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_319_20231204T002707Z_gen_g4_detsim_hitreco__20240509T204921Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/0f/e0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_40_20231208T072618Z_gen_g4_detsim_hitreco__20240510T054048Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/61/43/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_225_20231120T134053Z_gen_g4_detsim_hitreco__20240503T055557Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f7/e9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74577407_394_20231208T064238Z_gen_g4_detsim_hitreco__20240510T054802Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/22/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_8_20231205T071616Z_gen_g4_detsim_hitreco__20240510T042442Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b6/ff/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_1036_20231203T235418Z_gen_g4_detsim_hitreco__20240509T203147Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/da/49/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_603_20231201T212752Z_gen_g4_detsim_hitreco__20240508T045628Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/60/b7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_396_20231120T135035Z_gen_g4_detsim_hitreco__20240503T061342Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9c/e7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_316_20231205T024901Z_gen_g4_detsim_hitreco__20240509T222847Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/02/c4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_620_20231202T162424Z_gen_g4_detsim_hitreco__20240508T045120Z_reco2.root
.:
total 5588
-rw-r--r-- 1 nobody nobody 2177327 Sep 22 15:58 flatcaf.root
-rw-r--r-- 1 nobody nobody 1696141 Sep 22 15:58 caf.root
-rw-r--r-- 1 nobody nobody 1696141 Sep 22 15:58 caf_fd_hd_atmo_3301_20250922T183255Z.root
-rw-r--r-- 1 nobody nobody   75007 Sep 22 15:58 caf_20250922T183255Z.log
-rw-r--r-- 1 nobody nobody   33695 Sep 22 15:58 jobscript.log
-rw-r--r-- 1 nobody nobody    1927 Sep 22 15:58 caf_20250922T183255Z.file
-rw-r--r-- 1 nobody nobody    1927 Sep 22 15:58 caf_20250922T183255Z.pfns
-rw-r--r-- 1 nobody nobody    1927 Sep 22 15:32 file.list
-rw-r--r-- 1 nobody nobody    1927 Sep 22 15:58 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 22 15:32 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1375 Sep 22 15:58 caf_20250922T183255Z.did
-rw-r--r-- 1 nobody nobody    1375 Sep 22 15:32 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 22 15:33 debugprod.log