Jobsub ID 106434.3@dunegpschedd01.fnal.gov

Jobsub ID	106434.3@dunegpschedd01.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:57:38
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce01
Last heartbeat	2025-09-23 12:04:44
From worker node	Hostname	wn38
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:01:42
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1176_20231204T081514Z_gen_g4_detsim_hitreco__20240509T214359Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_317_20231205T024901Z_gen_g4_detsim_hitreco__20240510T014717Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_347_20231208T002738Z_gen_g4_detsim_hitreco__20240510T030918Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_487_20231201T105759Z_gen_g4_detsim_hitreco__20240507T193542Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_977_20231204T004637Z_gen_g4_detsim_hitreco__20240509T220512Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_268_20231203T093201Z_gen_g4_detsim_hitreco__20240509T190645Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_265_20231201T170304Z_gen_g4_detsim_hitreco__20240507T200143Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_992_20231203T235401Z_gen_g4_detsim_hitreco__20240509T211825Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_747_20231204T072832Z_gen_g4_detsim_hitreco__20240509T210910Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_169_20231208T061714Z_gen_g4_detsim_hitreco__20240510T043346Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 12:04:44
Saved logs	justin-logs:106434.3-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.0622541       100    
reco:opflash:OpFlashFinder                            0.00225571     0.0118871     0.081756     0.00574473     0.012496        100    
reco:opslicer:OpSlicer                                 0.0196798     0.441786       1.68982       0.33094       0.37817        100    
reco:rns:RandomNumberSaver                            0.000264024   0.00104155     0.0144266    0.000482706   0.00199806       100    
reco:anglereconue:NuAngularReco                       0.00194171     0.009264      0.083125     0.00466829     0.0119552       100    
reco:anglereconumu:NuAngularReco                      0.000239645   0.00257575     0.0508032    0.00113528    0.00585967       100    
reco:anglereconuepfps:NuAngularReco                   0.000852989   0.00495723     0.0523828    0.00294716    0.00727958       100    
reco:anglereconumupfps:NuAngularReco                  0.00033512    0.00248302     0.0540142    0.00123417    0.00575539       100    
reco:anglerecohits:NuAngularReco                      0.00158277    0.00719699     0.0588362    0.00357748     0.0090223       100    
[art]:TriggerResults:TriggerResultInserter            3.4547e-05    0.000183088   0.00305684    7.5224e-05    0.000328342      100    
end_path:cafmaker:CAFMaker                             0.0128624      0.14574       1.57641      0.0480406     0.227095        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3660.48 MB
  Peak resident set size usage (VmHWM): 1991.65 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 08:58:06 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/a2/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_317_20231205T024901Z_gen_g4_detsim_hitreco__20240510T014717Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995487, 0.000504061, 0.00400918, 
Output 1: 0.670099, 0.320883, 0.00873944, 0.000278604, 
Output 2: 0.917228, 0.0814907, 0.00125783, 2.33722e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1614!
Begin processing the 95th record. run: 50584512 subRun: 1 event: 117695 at 23-Sep-2025 08:47:46 -03
Boundary wire vector sizes: 89, 93, 59
minwire 0: 2161
minwire 1: 656
minwire 2: 2289
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0018127, 0.976931, 0.0212566, 
Output 1: 0.932332, 0.0666143, 0.000824258, 0.000229504, 
Output 2: 0.997334, 0.00257305, 7.73567e-05, 1.58016e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1131!
Begin processing the 96th record. run: 50584512 subRun: 1 event: 117696 at 23-Sep-2025 08:48:01 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 802!
Begin processing the 97th record. run: 50584512 subRun: 1 event: 117697 at 23-Sep-2025 08:48:14 -03
Boundary wire vector sizes: 4857, 5863, 4969
minwire 0: 397
minwire 1: 511
minwire 2: 340
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.241927, 0.333413, 0.42466, 
Output 1: 0.398724, 0.529068, 0.0488049, 0.023403, 
Output 2: 0.0276748, 0.0932326, 0.26012, 0.618973, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3135!
Begin processing the 98th record. run: 50584512 subRun: 1 event: 117698 at 23-Sep-2025 08:49:38 -03
Boundary wire vector sizes: 366, 249, 280
minwire 0: 1684
minwire 1: 605
minwire 2: 1965
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000582403, 0.995994, 0.00342393, 
Output 1: 0.984725, 0.0149574, 0.00020017, 0.000117875, 
Output 2: 0.998613, 0.00133267, 4.08381e-05, 1.35077e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1260!
Begin processing the 99th record. run: 50584512 subRun: 1 event: 117699 at 23-Sep-2025 08:49:55 -03
Boundary wire vector sizes: 1287, 1304, 726
minwire 0: 1547
minwire 1: 1335
minwire 2: 1296
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992113, 0.00219969, 0.00568701, 
Output 1: 0.560345, 0.347485, 0.0705785, 0.0215922, 
Output 2: 0.0625532, 0.580869, 0.294247, 0.0623302, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2850!
Begin processing the 100th record. run: 50584512 subRun: 1 event: 117700 at 23-Sep-2025 08:50:19 -03
Boundary wire vector sizes: 41, 40, 35
minwire 0: 2305
minwire 1: 402
minwire 2: 2547
Used alternate method to get min and max wires due to vertex determination failure: 103, 402
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 50, 349
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00235228, 0.00176384, 0.995884, 
Output 1: 0.93587, 0.0636623, 0.000395308, 7.23834e-05, 
Output 2: 0.999164, 0.000815302, 1.82061e-05, 2.38206e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 625!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1176_20231204T081514Z_gen_g4_detsim_hitreco__20240509T214359Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/a2/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_317_20231205T024901Z_gen_g4_detsim_hitreco__20240510T014717Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/90/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_347_20231208T002738Z_gen_g4_detsim_hitreco__20240510T030918Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/49/b3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_487_20231201T105759Z_gen_g4_detsim_hitreco__20240507T193542Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/dc/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419766_977_20231204T004637Z_gen_g4_detsim_hitreco__20240509T220512Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f4/c3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74510902_268_20231203T093201Z_gen_g4_detsim_hitreco__20240509T190645Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/28/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_265_20231201T170304Z_gen_g4_detsim_hitreco__20240507T200143Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/82/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_992_20231203T235401Z_gen_g4_detsim_hitreco__20240509T211825Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/aa/eb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_747_20231204T072832Z_gen_g4_detsim_hitreco__20240509T210910Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/95/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_169_20231208T061714Z_gen_g4_detsim_hitreco__20240510T043346Z_reco2.root
.:
total 3852
-rw-r--r-- 1 nobody nobody 1526900 Sep 23 08:58 flatcaf.root
-rw-r--r-- 1 nobody nobody 1129094 Sep 23 08:58 caf.root
-rw-r--r-- 1 nobody nobody 1129094 Sep 23 08:58 caf_fd_hd_atmo_3302_20250923T100251Z.root
-rw-r--r-- 1 nobody nobody   74309 Sep 23 08:58 caf_20250923T100251Z.log
-rw-r--r-- 1 nobody nobody   36491 Sep 23 08:58 jobscript.log
-rw-r--r-- 1 nobody nobody    1996 Sep 23 08:58 caf_20250923T100251Z.file
-rw-r--r-- 1 nobody nobody    1996 Sep 23 08:58 caf_20250923T100251Z.pfns
-rw-r--r-- 1 nobody nobody    1996 Sep 23 07:02 file.list
-rw-r--r-- 1 nobody nobody    1996 Sep 23 08:58 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1378 Sep 23 07:02 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1378 Sep 23 08:58 caf_20250923T100251Z.did
-rw-r--r-- 1 nobody nobody    1378 Sep 23 07:02 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 07:11 debugprod.log