Jobsub ID 97035.117@dunegpschedd02.fnal.gov

Jobsub ID	97035.117@dunegpschedd02.fnal.gov
Workflow ID	3302
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-23 09:07:36
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce02
Last heartbeat	2025-09-23 11:21:08
From worker node	Hostname	wn33.cat.cbpf.br
	cpuinfo	Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-23 09:11:40
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_178_20231122T103704Z_gen_g4_detsim_hitreco__20240503T055606Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_850_20231203T062725Z_gen_g4_detsim_hitreco__20240508T051639Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_240_20231203T232034Z_gen_g4_detsim_hitreco__20240509T204144Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_318_20231207T011914Z_gen_g4_detsim_hitreco__20240510T044151Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1_20231202T160435Z_gen_g4_detsim_hitreco__20240508T035907Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_746_20231208T050801Z_gen_g4_detsim_hitreco__20240510T041821Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_246_20231201T104719Z_gen_g4_detsim_hitreco__20240507T193134Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_472_20231208T004704Z_gen_g4_detsim_hitreco__20240510T030843Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_162_20231204T061417Z_gen_g4_detsim_hitreco__20240509T204557Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_47_20231202T195850Z_gen_g4_detsim_hitreco__20240508T062050Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-23 11:21:08
Saved logs	justin-logs:97035.117-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

reco:opflash:OpFlashFinder                            0.000221244   0.00239879    0.00545206    0.00224352    0.00106795       400    
reco:opslicer:OpSlicer                                9.3971e-05     0.425125       1.59827      0.315249      0.357964        400    
reco:rns:RandomNumberSaver                            3.0294e-05    6.73532e-05   0.000748747   5.8557e-05    5.59345e-05      400    
reco:anglereconue:NuAngularReco                       0.000256721   0.00346356     0.0832231    0.000697803   0.00951861       400    
reco:anglereconumu:NuAngularReco                      0.000183635   0.00311642     0.0784856    0.000476528   0.00918491       400    
reco:anglereconuepfps:NuAngularReco                   0.000258147   0.00332181     0.0792589    0.000682621   0.00922749       400    
reco:anglereconumupfps:NuAngularReco                  0.000254694   0.00329186     0.0801097    0.000630708    0.0092488       400    
reco:anglerecohits:NuAngularReco                      0.000240883   0.00454855     0.0902343    0.00144683     0.0103988       400    
[art]:TriggerResults:TriggerResultInserter            1.5415e-05    2.94591e-05    9.157e-05    2.7182e-05    7.56411e-06      400    
end_path:cafmaker:CAFMaker                            0.00185376     0.0479929      1.16623      0.0117654     0.124397        400    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3922.69 MB
  Peak resident set size usage (VmHWM): 2245.79 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Sep-2025 08:16:43 -03 ModuleEndJob
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/a8/6a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1_20231202T160435Z_gen_g4_detsim_hitreco__20240508T035907Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.801675, 0.000930882, 0.197394, 
Output 1: 0.995445, 0.00426456, 0.000188334, 0.000102372, 
Output 2: 0.864926, 0.131879, 0.00279504, 0.000400548, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1211!
Begin processing the 395th record. run: 50604274 subRun: 1 event: 31895 at 23-Sep-2025 08:11:52 -03
Boundary wire vector sizes: 140, 171, 156
minwire 0: 1106
minwire 1: 1192
minwire 2: 1297
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000448323, 0.994701, 0.00485036, 
Output 1: 0.216743, 0.774463, 0.00850089, 0.00029295, 
Output 2: 0.997403, 0.00254792, 4.72015e-05, 1.59318e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1068!
Begin processing the 396th record. run: 50604274 subRun: 1 event: 31896 at 23-Sep-2025 08:12:07 -03
Boundary wire vector sizes: 926, 1061, 926
minwire 0: 2064
minwire 1: 257
minwire 2: 2193
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00714801, 0.052968, 0.939884, 
Output 1: 0.480253, 0.517572, 0.00184785, 0.000327505, 
Output 2: 0.562193, 0.429718, 0.00787581, 0.000213005, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1852!
Begin processing the 397th record. run: 50604274 subRun: 1 event: 31897 at 23-Sep-2025 08:12:29 -03
Boundary wire vector sizes: 102, 98, 92
minwire 0: 1255
minwire 1: 2337
minwire 2: 626
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00386787, 0.124808, 0.871324, 
Output 1: 0.574876, 0.417824, 0.00706751, 0.000232332, 
Output 2: 0.995541, 0.00431756, 0.000131968, 9.3768e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1420!
Begin processing the 398th record. run: 50604274 subRun: 1 event: 31898 at 23-Sep-2025 08:12:44 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 972!
Begin processing the 399th record. run: 50604274 subRun: 1 event: 31899 at 23-Sep-2025 08:12:56 -03
Boundary wire vector sizes: 57, 13, 31
minwire 0: 1386
minwire 1: 2317
minwire 2: 893
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0121527, 0.00349261, 0.984355, 
Output 1: 0.00613167, 0.988474, 0.00524827, 0.000146044, 
Output 2: 0.996892, 0.00302983, 7.56839e-05, 2.20699e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1260!
Begin processing the 400th record. run: 50604274 subRun: 1 event: 31900 at 23-Sep-2025 08:13:09 -03
Boundary wire vector sizes: 88, 86, 84
minwire 0: 1368
minwire 1: 1562
minwire 2: 1121
Used alternate method to get min and max wires due to vertex determination failure: 344, 643
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 230, 529
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 189, 488
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00348021, 0.00290088, 0.993619, 
Output 1: 0.961342, 0.0374839, 0.000911412, 0.000262296, 
Output 2: 0.997338, 0.00259419, 5.90601e-05, 8.75045e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1936!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/43/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_178_20231122T103704Z_gen_g4_detsim_hitreco__20240503T055606Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/73/2a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_850_20231203T062725Z_gen_g4_detsim_hitreco__20240508T051639Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9a/c2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66009532_240_20231203T232034Z_gen_g4_detsim_hitreco__20240509T204144Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/88/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_318_20231207T011914Z_gen_g4_detsim_hitreco__20240510T044151Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/a8/6a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1_20231202T160435Z_gen_g4_detsim_hitreco__20240508T035907Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e9/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_746_20231208T050801Z_gen_g4_detsim_hitreco__20240510T041821Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/60/44/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_246_20231201T104719Z_gen_g4_detsim_hitreco__20240507T193134Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b3/51/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6491894_472_20231208T004704Z_gen_g4_detsim_hitreco__20240510T030843Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ae/17/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517585_162_20231204T061417Z_gen_g4_detsim_hitreco__20240509T204557Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e9/4b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_47_20231202T195850Z_gen_g4_detsim_hitreco__20240508T062050Z_reco2.root
.:
total 18164
-rw-r--r-- 1 nobody nobody 6873274 Sep 23 08:16 flatcaf.root
-rw-r--r-- 1 nobody nobody 5668517 Sep 23 08:16 caf.root
-rw-r--r-- 1 nobody nobody 5668517 Sep 23 08:16 caf_fd_hd_atmo_3302_20250923T091154Z.root
-rw-r--r-- 1 nobody nobody  241127 Sep 23 08:16 caf_20250923T091154Z.log
-rw-r--r-- 1 nobody nobody   98765 Sep 23 08:16 jobscript.log
-rw-r--r-- 1 nobody nobody    1925 Sep 23 08:16 caf_20250923T091154Z.file
-rw-r--r-- 1 nobody nobody    1925 Sep 23 08:16 caf_20250923T091154Z.pfns
-rw-r--r-- 1 nobody nobody    1925 Sep 23 06:11 file.list
-rw-r--r-- 1 nobody nobody    1925 Sep 23 08:16 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1373 Sep 23 06:11 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1373 Sep 23 08:16 caf_20250923T091154Z.did
-rw-r--r-- 1 nobody nobody    1373 Sep 23 06:11 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 23 06:12 debugprod.log