Jobsub ID 40082.72@dunegpschedd02.fnal.gov

Jobsub ID	40082.72@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 15:12:48
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce03
Last heartbeat	2025-09-15 19:25:30
From worker node	Hostname	wn77
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 15:14:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_439_20231204T220459Z_gen_g4_detsim_hitreco__20240509T221259Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_511_20231202T102837Z_gen_g4_detsim_hitreco__20240507T200237Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_12_20231204T000549Z_gen_g4_detsim_hitreco__20240509T203952Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_477_20231027T231400Z_gen_g4_detsim_hitreco__20240503T045205Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_713_20231208T082449Z_gen_g4_detsim_hitreco__20240510T062743Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_119_20231027T230240Z_gen_g4_detsim_hitreco__20240503T043155Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_635_20231203T080206Z_gen_g4_detsim_hitreco__20240508T060521Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1150_20231203T070704Z_gen_g4_detsim_hitreco__20240508T042220Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_985_20231203T065406Z_gen_g4_detsim_hitreco__20240508T075008Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_625_20231207T115346Z_gen_g4_detsim_hitreco__20240510T053744Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 19:25:30
Saved logs	justin-logs:40082.72-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

900    
reco:energyrecnumumcsllhd:EnergyReco                  0.000582106    0.0500404      3.11201     0.00464905      0.21773        900    
reco:opdec:Deconvolution                               0.511148       1.00768       7.74776      0.994661      0.321202        900    
reco:ophitspe:OpHitFinderDeco                           1.4769        1.75641       1.94087       1.84426       0.15993        900    
reco:opflash:OpFlashFinder                            0.00021803    0.00174643     0.0080532    0.00168624    0.000825177      900    
reco:opslicer:OpSlicer                                6.8313e-05     0.368268       1.55196      0.288715       0.30988        900    
reco:rns:RandomNumberSaver                            2.3881e-05    3.75068e-05   0.00046451    3.64415e-05   1.68451e-05      900    
reco:anglereconue:NuAngularReco                       0.000174367   0.00210393     0.131756     0.000482491   0.00897906       900    
reco:anglereconumu:NuAngularReco                      8.6144e-05    0.00194626     0.132619     0.000324755   0.00896044       900    
reco:anglereconuepfps:NuAngularReco                   0.000126716   0.00206843     0.140071     0.000444165    0.0090283       900    
reco:anglereconumupfps:NuAngularReco                  0.000113645   0.00205362      0.13465     0.000422438   0.00907365       900    
reco:anglerecohits:NuAngularReco                      0.000122125   0.00283397     0.147601     0.000959402   0.00983047       900    
[art]:TriggerResults:TriggerResultInserter             1.034e-05    1.75787e-05   8.1284e-05    1.72105e-05   4.81003e-06      900    
end_path:cafmaker:CAFMaker                            0.00146014     0.025396       1.58365     0.00514419     0.104314        900    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3839.25 MB
  Peak resident set size usage (VmHWM): 2169.58 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 16:21:24 -03 ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c1/b2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1150_20231203T070704Z_gen_g4_detsim_hitreco__20240508T042220Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.9646, 0.0127739, 0.022626, 
Output 1: 0.0195491, 0.949637, 0.0285468, 0.00226701, 
Output 2: 0.0368724, 0.951686, 0.01141, 3.14041e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1069!
Begin processing the 894th record. run: 6410210 subRun: 1 event: 63594 at 15-Sep-2025 16:20:02 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 716!
Begin processing the 895th record. run: 6410210 subRun: 1 event: 63595 at 15-Sep-2025 16:20:12 -03
Boundary wire vector sizes: 221, 461, 289
minwire 0: 898
minwire 1: 2064
minwire 2: 525
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0767362, 0.728545, 0.194719, 
Output 1: 0.0996376, 0.720446, 0.16354, 0.0163761, 
Output 2: 0.117721, 0.824669, 0.0564486, 0.0011616, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2085!
Begin processing the 896th record. run: 6410210 subRun: 1 event: 63596 at 15-Sep-2025 16:20:23 -03
Boundary wire vector sizes: 490, 492, 472
minwire 0: 1987
minwire 1: 974
minwire 2: 1772
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0247007, 0.232048, 0.743252, 
Output 1: 0.466079, 0.468054, 0.0583745, 0.00749277, 
Output 2: 0.487984, 0.376046, 0.115044, 0.0209268, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2522!
Begin processing the 897th record. run: 6410210 subRun: 1 event: 63597 at 15-Sep-2025 16:20:36 -03
Boundary wire vector sizes: 406, 180, 275
minwire 0: 921
minwire 1: 1388
minwire 2: 644
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.952083, 0.00284984, 0.0450673, 
Output 1: 0.152932, 0.443215, 0.302061, 0.101792, 
Output 2: 0.590517, 0.390853, 0.0180285, 0.000602157, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2073!
Begin processing the 898th record. run: 6410210 subRun: 1 event: 63598 at 15-Sep-2025 16:20:48 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 822!
Begin processing the 899th record. run: 6410210 subRun: 1 event: 63599 at 15-Sep-2025 16:20:58 -03
Boundary wire vector sizes: 365, 519, 516
minwire 0: 390
minwire 1: 1851
minwire 2: 214
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.994379, 0.000451879, 0.00516865, 
Output 1: 0.0755144, 0.777903, 0.142604, 0.00397871, 
Output 2: 0.992296, 0.00740353, 0.000293169, 7.84998e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1656!
Begin processing the 900th record. run: 6410210 subRun: 1 event: 63600 at 15-Sep-2025 16:21:10 -03
Boundary wire vector sizes: 49, 103, 61
minwire 0: 1930
minwire 1: 722
minwire 2: 2124
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000691586, 0.99806, 0.001248, 
Output 1: 0.00646898, 0.929752, 0.0614652, 0.00231354, 
Output 2: 0.998052, 0.00186467, 8.09939e-05, 2.48169e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1134!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/b6/ee/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_439_20231204T220459Z_gen_g4_detsim_hitreco__20240509T221259Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_511_20231202T102837Z_gen_g4_detsim_hitreco__20240507T200237Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/57/43/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419587_12_20231204T000549Z_gen_g4_detsim_hitreco__20240509T203952Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5d/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_477_20231027T231400Z_gen_g4_detsim_hitreco__20240503T045205Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/07/c8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_713_20231208T082449Z_gen_g4_detsim_hitreco__20240510T062743Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_119_20231027T230240Z_gen_g4_detsim_hitreco__20240503T043155Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c6/3a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_635_20231203T080206Z_gen_g4_detsim_hitreco__20240508T060521Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c1/b2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1150_20231203T070704Z_gen_g4_detsim_hitreco__20240508T042220Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ab/0e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_985_20231203T065406Z_gen_g4_detsim_hitreco__20240508T075008Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d7/01/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_625_20231207T115346Z_gen_g4_detsim_hitreco__20240510T053744Z_reco2.root
.:
total 40016
-rw-r--r-- 1 nobody nobody 15011889 Sep 15 16:21 flatcaf.root
-rw-r--r-- 1 nobody nobody 12557523 Sep 15 16:21 caf.root
-rw-r--r-- 1 nobody nobody 12557523 Sep 15 16:21 caf_fd_hd_atmo_2501_20250915T151429Z.root
-rw-r--r-- 1 nobody nobody   535500 Sep 15 16:21 caf_20250915T151429Z.log
-rw-r--r-- 1 nobody nobody   206173 Sep 15 16:21 jobscript.log
-rw-r--r-- 1 nobody nobody     1936 Sep 15 16:21 caf_20250915T151429Z.file
-rw-r--r-- 1 nobody nobody     1936 Sep 15 16:21 caf_20250915T151429Z.pfns
-rw-r--r-- 1 nobody nobody     1936 Sep 15 12:14 file.list
-rw-r--r-- 1 nobody nobody     1936 Sep 15 16:21 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody     1368 Sep 15 12:14 all-input-dids.txt
-rw-r--r-- 1 nobody nobody     1368 Sep 15 16:21 caf_20250915T151429Z.did
-rw-r--r-- 1 nobody nobody     1368 Sep 15 12:14 did.list
-rw-r--r-- 1 nobody nobody        0 Sep 15 12:14 debugprod.log