Jobsub ID 103192.3@dunegpschedd02.fnal.gov

Jobsub ID	103192.3@dunegpschedd02.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 10:29:39
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-23 12:57:42
From worker node	Hostname	wn75
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 10:37:23
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_592_20231204T035350Z_gen_g4_detsim_hitreco__20240509T210250Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_563_20231204T041429Z_gen_g4_detsim_hitreco__20240509T195715Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_1178_20231204T000537Z_gen_g4_detsim_hitreco__20240509T213007Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_622_20231207T062028Z_gen_g4_detsim_hitreco__20240510T045614Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_677_20231205T072410Z_gen_g4_detsim_hitreco__20240510T021507Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_474_20231203T041418Z_gen_g4_detsim_hitreco__20240508T071929Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_142_20231204T202226Z_gen_g4_detsim_hitreco__20240510T024428Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_595_20231207T112659Z_gen_g4_detsim_hitreco__20240510T053444Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_903_20231202T065408Z_gen_g4_detsim_hitreco__20240508T054005Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_446_20231205T025956Z_gen_g4_detsim_hitreco__20240510T020956Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 12:57:42
Saved logs	justin-logs:103192.3-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.593749       1.02834       2.09101       1.00085      0.226739        500    
reco:ophitspe:OpHitFinderDeco                           1.76587       1.80658       1.89265       1.7797       0.0379059       500    
reco:opflash:OpFlashFinder                            0.000180388   0.00181654     0.0132593    0.00171589    0.000971269      500    
reco:opslicer:OpSlicer                                6.6823e-05     0.383239       1.60292      0.293154       0.3213         500    
reco:rns:RandomNumberSaver                            2.5331e-05    4.03508e-05   0.000549621   3.6931e-05    2.68874e-05      500    
reco:anglereconue:NuAngularReco                       0.000198128   0.00300623     0.101214     0.00053871     0.0107605       500    
reco:anglereconumu:NuAngularReco                      9.6464e-05    0.00280921     0.102071     0.000380926    0.0105138       500    
reco:anglereconuepfps:NuAngularReco                   0.000167457   0.00294872     0.107776     0.000487949    0.0107103       500    
reco:anglereconumupfps:NuAngularReco                  0.000162287   0.00290746     0.105502     0.000466913    0.0105784       500    
reco:anglerecohits:NuAngularReco                      0.000149666   0.00385295     0.114976     0.00100752     0.0118882       500    
[art]:TriggerResults:TriggerResultInserter            1.1231e-05    1.85221e-05   9.2934e-05    1.7641e-05    6.81053e-06      500    
end_path:cafmaker:CAFMaker                            0.00143176     0.0361358      1.18868     0.00560979     0.126255        500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3775.62 MB
  Peak resident set size usage (VmHWM): 2107.58 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 09:53:31 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/61/ba/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_474_20231203T041418Z_gen_g4_detsim_hitreco__20240508T071929Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1900!
Begin processing the 493rd record. run: 6430442 subRun: 1 event: 67793 at 23-Sep-2025 09:44:49 -03
Boundary wire vector sizes: 255, 130, 217
minwire 0: 2154
minwire 1: 883
minwire 2: 2154
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.993313, 0.000344847, 0.00634217, 
Output 1: 0.00215002, 0.0612989, 0.91761, 0.0189414, 
Output 2: 0.954264, 0.0437042, 0.00200536, 2.67528e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1761!
Begin processing the 494th record. run: 6430442 subRun: 1 event: 67794 at 23-Sep-2025 09:45:00 -03
Boundary wire vector sizes: 473, 284, 129
minwire 0: 882
minwire 1: 1536
minwire 2: 1003
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996223, 0.000396822, 0.0033803, 
Output 1: 0.89945, 0.0950658, 0.00458748, 0.000896865, 
Output 2: 0.989973, 0.00972906, 0.000272366, 2.51684e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1727!
Begin processing the 495th record. run: 6430442 subRun: 1 event: 67795 at 23-Sep-2025 09:45:12 -03
Boundary wire vector sizes: 349, 456, 385
minwire 0: 774
minwire 1: 2153
minwire 2: 533
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000494049, 0.997191, 0.0023148, 
Output 1: 0.992581, 0.00725811, 7.37798e-05, 8.66406e-05, 
Output 2: 0.99916, 0.000803643, 2.3845e-05, 1.26682e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2144!
Begin processing the 496th record. run: 6430442 subRun: 1 event: 67796 at 23-Sep-2025 09:45:24 -03
Boundary wire vector sizes: 91, 92, 88
minwire 0: 1212
minwire 1: 818
minwire 2: 1298
Used alternate method to get min and max wires due to vertex determination failure: 953, 1252
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 678, 977
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00244572, 0.00165352, 0.995901, 
Output 1: 0.278229, 0.716412, 0.00518172, 0.000177467, 
Output 2: 0.998518, 0.00140826, 7.01029e-05, 3.97016e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1692!
Begin processing the 497th record. run: 6430442 subRun: 1 event: 67797 at 23-Sep-2025 09:45:35 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 49!
Begin processing the 498th record. run: 6430442 subRun: 1 event: 67798 at 23-Sep-2025 09:45:44 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 597!
Begin processing the 499th record. run: 6430442 subRun: 1 event: 67799 at 23-Sep-2025 09:45:54 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1602!
Begin processing the 500th record. run: 6430442 subRun: 1 event: 67800 at 23-Sep-2025 09:46:04 -03
Boundary wire vector sizes: 169, 158, 115
minwire 0: 1295
minwire 1: 1827
minwire 2: 1060
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000215646, 0.998939, 0.000845715, 
Output 1: 0.00384171, 0.991942, 0.00407922, 0.000137173, 
Output 2: 0.998468, 0.00149175, 3.94336e-05, 1.16802e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1919!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_592_20231204T035350Z_gen_g4_detsim_hitreco__20240509T210250Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f8/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_563_20231204T041429Z_gen_g4_detsim_hitreco__20240509T195715Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c7/29/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_1178_20231204T000537Z_gen_g4_detsim_hitreco__20240509T213007Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/73/07/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_622_20231207T062028Z_gen_g4_detsim_hitreco__20240510T045614Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cc/39/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_677_20231205T072410Z_gen_g4_detsim_hitreco__20240510T021507Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/61/ba/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_474_20231203T041418Z_gen_g4_detsim_hitreco__20240508T071929Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cf/56/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_142_20231204T202226Z_gen_g4_detsim_hitreco__20240510T024428Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8c/69/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_595_20231207T112659Z_gen_g4_detsim_hitreco__20240510T053444Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d2/bf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_903_20231202T065408Z_gen_g4_detsim_hitreco__20240508T054005Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/6d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_446_20231205T025956Z_gen_g4_detsim_hitreco__20240510T020956Z_reco2.root
.:
total 27712
-rw-r--r-- 1 nobody nobody 10441855 Sep 23 09:53 flatcaf.root
-rw-r--r-- 1 nobody nobody  8720837 Sep 23 09:53 caf.root
-rw-r--r-- 1 nobody nobody  8720837 Sep 23 09:53 caf_fd_hd_atmo_3302_20250923T103736Z.root
-rw-r--r-- 1 nobody nobody   303590 Sep 23 09:53 caf_20250923T103736Z.log
-rw-r--r-- 1 nobody nobody   123889 Sep 23 09:53 jobscript.log
-rw-r--r-- 1 nobody nobody     1898 Sep 23 09:53 caf_20250923T103736Z.file
-rw-r--r-- 1 nobody nobody     1898 Sep 23 09:53 caf_20250923T103736Z.pfns
-rw-r--r-- 1 nobody nobody     1898 Sep 23 07:37 file.list
-rw-r--r-- 1 nobody nobody     1898 Sep 23 09:53 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1376 Sep 23 07:37 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1376 Sep 23 09:53 caf_20250923T103736Z.did
-rw-r--r-- 1 nobody nobody     1376 Sep 23 07:37 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 23 07:38 debugprod.log