Jobsub ID 39988.135@dunegpschedd02.fnal.gov

Jobsub ID	39988.135@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:33:41
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2025-09-15 15:07:05
From worker node	Hostname	mwt2-c061.campuscluster.illinois.edu
	cpuinfo	AMD EPYC 7443 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:35:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_1133_20231202T192620Z_gen_g4_detsim_hitreco__20240508T060206Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_904_20231202T144427Z_gen_g4_detsim_hitreco__20240508T060133Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_320_20231207T153504Z_gen_g4_detsim_hitreco__20240510T030326Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_392_20231121T154438Z_gen_g4_detsim_hitreco__20240507T173731Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_186_20231208T001422Z_gen_g4_detsim_hitreco__20240510T030126Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_314_20231122T053735Z_gen_g4_detsim_hitreco__20240507T182133Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_683_20231204T100454Z_gen_g4_detsim_hitreco__20240509T194143Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_446_20231120T135452Z_gen_g4_detsim_hitreco__20240507T172959Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_133_20231121T150230Z_gen_g4_detsim_hitreco__20240503T070619Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_216_20231208T061734Z_gen_g4_detsim_hitreco__20240510T045043Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 15:07:05
Saved logs	justin-logs:39988.135-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0    
reco:opdec:Deconvolution                               0.304922      0.568073       1.09897      0.583416      0.113358       1100    
reco:ophitspe:OpHitFinderDeco                          0.884513      0.983924       1.09629      0.982721      0.0557704      1100    
reco:opflash:OpFlashFinder                            0.000146344   0.00160444     0.0108672    0.00152824    0.000736107     1100    
reco:opslicer:OpSlicer                                4.3942e-05     0.265837       1.06518      0.203313      0.221815       1100    
reco:rns:RandomNumberSaver                            2.3404e-05    7.20156e-05   0.00101781    6.4266e-05    5.39808e-05     1100    
reco:anglereconue:NuAngularReco                       0.000179618   0.00157894     0.058704     0.000542024   0.00406144      1100    
reco:anglereconumu:NuAngularReco                       9.089e-05    0.00124642     0.0563771    0.000286648   0.00388974      1100    
reco:anglereconuepfps:NuAngularReco                   0.000134532   0.00140008     0.0567227    0.000422639   0.00393099      1100    
reco:anglereconumupfps:NuAngularReco                  0.000110688    0.001325      0.0562847    0.000350935   0.00393084      1100    
reco:anglerecohits:NuAngularReco                      0.000101791   0.00208158     0.0624202    0.000864895   0.00459316      1100    
[art]:TriggerResults:TriggerResultInserter             8.336e-06    1.69699e-05   0.000294954   1.5594e-05    9.66052e-06     1100    
end_path:cafmaker:CAFMaker                            0.00134483     0.0210733     0.810667     0.00763511     0.0525754      1100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3689.35 MB
  Peak resident set size usage (VmHWM): 1994.01 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 10:06:42 CDT ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/98/86/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_133_20231121T150230Z_gen_g4_detsim_hitreco__20240503T070619Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.815238, 0.00109296, 0.183669, 
Output 1: 0.797845, 0.187995, 0.0132442, 0.000916327, 
Output 2: 0.781934, 0.212795, 0.00508567, 0.000185893, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2321!
Begin processing the 1094th record. run: 74251126 subRun: 1 event: 89394 at 15-Sep-2025 09:58:16 CDT
Boundary wire vector sizes: 85, 143, 82
minwire 0: 2685
minwire 1: 471
minwire 2: 2706
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00060111, 0.997226, 0.00217247, 
Output 1: 0.00409218, 0.991825, 0.00396381, 0.000118658, 
Output 2: 0.997828, 0.0021116, 5.83329e-05, 1.83175e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1125!
Begin processing the 1095th record. run: 74251126 subRun: 1 event: 89395 at 15-Sep-2025 09:58:23 CDT
Boundary wire vector sizes: 442, 274, 354
minwire 0: 566
minwire 1: 2385
minwire 2: 221
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00313, 0.884843, 0.112027, 
Output 1: 0.617719, 0.362933, 0.0180872, 0.00126142, 
Output 2: 0.97899, 0.0201244, 0.000814005, 7.16185e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2087!
Begin processing the 1096th record. run: 74251126 subRun: 1 event: 89396 at 15-Sep-2025 09:58:30 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 715!
Begin processing the 1097th record. run: 74251126 subRun: 1 event: 89397 at 15-Sep-2025 09:58:35 CDT
Boundary wire vector sizes: 102, 283, 148
minwire 0: 1930
minwire 1: 488
minwire 2: 2183
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995861, 0.000545045, 0.00359416, 
Output 1: 0.0838264, 0.912068, 0.0040039, 0.000102136, 
Output 2: 0.996184, 0.0037092, 0.00010324, 3.18939e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1096!
Begin processing the 1098th record. run: 74251126 subRun: 1 event: 89398 at 15-Sep-2025 09:58:42 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 234!
Begin processing the 1099th record. run: 74251126 subRun: 1 event: 89399 at 15-Sep-2025 09:58:47 CDT
Boundary wire vector sizes: 117, 150, 126
minwire 0: 595
minwire 1: 2002
minwire 2: 401
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00615147, 0.00673103, 0.987117, 
Output 1: 0.011805, 0.983162, 0.00481277, 0.000220126, 
Output 2: 0.979293, 0.0201724, 0.000521611, 1.28617e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1349!
Begin processing the 1100th record. run: 74251126 subRun: 1 event: 89400 at 15-Sep-2025 09:58:53 CDT
Boundary wire vector sizes: 248, 206, 190
minwire 0: 1662
minwire 1: 1029
minwire 2: 1410
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0466067, 0.0345412, 0.918852, 
Output 1: 0.0243176, 0.13959, 0.414751, 0.421342, 
Output 2: 0.955822, 0.0418984, 0.00212322, 0.000156858, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2650!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/f0/b6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571149_1133_20231202T192620Z_gen_g4_detsim_hitreco__20240508T060206Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/27/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_904_20231202T144427Z_gen_g4_detsim_hitreco__20240508T060133Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f0/39/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_320_20231207T153504Z_gen_g4_detsim_hitreco__20240510T030326Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/27/68/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_392_20231121T154438Z_gen_g4_detsim_hitreco__20240507T173731Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/3a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_186_20231208T001422Z_gen_g4_detsim_hitreco__20240510T030126Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d6/14/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_314_20231122T053735Z_gen_g4_detsim_hitreco__20240507T182133Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3e/09/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_683_20231204T100454Z_gen_g4_detsim_hitreco__20240509T194143Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1b/64/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74251126_446_20231120T135452Z_gen_g4_detsim_hitreco__20240507T172959Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/98/86/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74254738_133_20231121T150230Z_gen_g4_detsim_hitreco__20240503T070619Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cd/6f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472619_216_20231208T061734Z_gen_g4_detsim_hitreco__20240510T045043Z_reco2.root
.:
total 46464
-rw-r--r-- 1 dune osgvo 17432047 Sep 15 10:06 flatcaf.root
-rw-r--r-- 1 dune osgvo 14593657 Sep 15 10:06 caf.root
-rw-r--r-- 1 dune osgvo 14593657 Sep 15 10:06 caf_fd_hd_atmo_2501_20250915T123544Z.root
-rw-r--r-- 1 dune osgvo   658325 Sep 15 10:06 caf_20250915T123544Z.log
-rw-r--r-- 1 dune osgvo   251071 Sep 15 10:06 jobscript.log
-rw-r--r-- 1 dune osgvo     1922 Sep 15 10:06 caf_20250915T123544Z.file
-rw-r--r-- 1 dune osgvo     1922 Sep 15 10:06 caf_20250915T123544Z.pfns
-rw-r--r-- 1 dune osgvo     1922 Sep 15 07:35 file.list
-rw-r--r-- 1 dune osgvo     1922 Sep 15 10:06 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo     1378 Sep 15 07:35 all-input-dids.txt
-rw-r--r-- 1 dune osgvo     1378 Sep 15 10:06 caf_20250915T123544Z.did
-rw-r--r-- 1 dune osgvo     1378 Sep 15 07:35 did.list
-rw-r--r-- 1 dune osgvo        0 Sep 15 07:35 debugprod.log