Jobsub ID 38122.33@dunegpschedd02.fnal.gov

Jobsub ID	38122.33@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:02:34
Site	BR_CBPF
Entry	DUNE_BR_CBPF_ce04
Last heartbeat	2025-09-11 11:54:06
From worker node	Hostname	wn121.cat.cbpf.br
	cpuinfo	AMD EPYC 7713P 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-11 10:47:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_592_20231202T083531Z_gen_g4_detsim_hitreco__20240507T210043Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_870_20231203T050845Z_gen_g4_detsim_hitreco__20240508T072631Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_389_20231208T000819Z_gen_g4_detsim_hitreco__20240510T030814Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_641_20231207T033027Z_gen_g4_detsim_hitreco__20240510T024954Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_67_20231122T061042Z_gen_g4_detsim_hitreco__20240507T172445Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1088_20231204T120519Z_gen_g4_detsim_hitreco__20240509T215339Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_340_20231208T040032Z_gen_g4_detsim_hitreco__20240510T045800Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_654_20231208T062038Z_gen_g4_detsim_hitreco__20240510T051505Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_646_20231201T210420Z_gen_g4_detsim_hitreco__20240507T222536Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_148_20231207T163054Z_gen_g4_detsim_hitreco__20240510T064106Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-11 11:54:06
Saved logs	justin-logs:38122.33-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.424713        100    
reco:opdec:Deconvolution                                0.48648      0.979333       2.16141      0.874348      0.309045        100    
reco:ophitspe:OpHitFinderDeco                           1.40968       1.44201       1.64831       1.42623      0.0428261       100    
reco:opflash:OpFlashFinder                            0.000948807   0.00454449     0.0261012    0.00301094    0.00442548       100    
reco:opslicer:OpSlicer                                0.000326053    0.449745       5.14419      0.248693      0.637588        100    
reco:rns:RandomNumberSaver                            6.2392e-05    0.000238951   0.00221684    0.000110024   0.000356331      100    
reco:anglereconue:NuAngularReco                       0.000506089   0.00773845      0.16529      0.0014359     0.0233188       100    
reco:anglereconumu:NuAngularReco                      0.000135725   0.00715682     0.179751     0.000570398    0.0248708       100    
reco:anglereconuepfps:NuAngularReco                   0.000242679   0.00771046     0.189855     0.000871299    0.0260127       100    
reco:anglereconumupfps:NuAngularReco                  0.000165677   0.00770203     0.175553     0.000727428    0.0252871       100    
reco:anglerecohits:NuAngularReco                      0.000617194    0.0106801     0.191775     0.00196757     0.0282227       100    
[art]:TriggerResults:TriggerResultInserter            2.0171e-05    0.000113415    0.0013313    4.34465e-05   0.000182377      100    
end_path:cafmaker:CAFMaker                            0.00524104     0.164417       6.73589      0.0190062     0.718283        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3665.85 MB
  Peak resident set size usage (VmHWM): 1937.63 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 08:42:54 -03 ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_870_20231203T050845Z_gen_g4_detsim_hitreco__20240508T072631Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 2: 0.998106, 0.00182679, 6.45331e-05, 2.26771e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1803!
Begin processing the 94th record. run: 6400858 subRun: 1 event: 59294 at 11-Sep-2025 08:31:42 -03
Boundary wire vector sizes: 6571, 7726, 4593
minwire 0: 2140
minwire 1: 145
minwire 2: 1798
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0517627, 0.940637, 0.00759991, 
Output 1: 0.303292, 0.448218, 0.142149, 0.106341, 
Output 2: 0.0406171, 0.187972, 0.38406, 0.387351, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3003!
Begin processing the 95th record. run: 6400858 subRun: 1 event: 59295 at 11-Sep-2025 08:41:05 -03
Boundary wire vector sizes: 1793, 1991, 1923
minwire 0: 1069
minwire 1: 465
minwire 2: 1255
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.992454, 0.00096888, 0.00657677, 
Output 1: 0.363506, 0.396526, 0.143862, 0.0961057, 
Output 2: 0.375132, 0.358684, 0.193678, 0.0725054, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2343!
Begin processing the 96th record. run: 6400858 subRun: 1 event: 59296 at 11-Sep-2025 08:41:40 -03
Boundary wire vector sizes: 469, 1248, 682
minwire 0: 800
minwire 1: 2201
minwire 2: 212
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99784, 0.000320092, 0.00183977, 
Output 1: 0.294562, 0.487382, 0.140122, 0.0779339, 
Output 2: 0.735717, 0.23048, 0.0305251, 0.0032784, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2038!
Begin processing the 97th record. run: 6400858 subRun: 1 event: 59297 at 11-Sep-2025 08:41:59 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 894!
Begin processing the 98th record. run: 6400858 subRun: 1 event: 59298 at 11-Sep-2025 08:42:09 -03

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 729!
Begin processing the 99th record. run: 6400858 subRun: 1 event: 59299 at 11-Sep-2025 08:42:18 -03
Boundary wire vector sizes: 39, 41, 35
minwire 0: 985
minwire 1: 1402
minwire 2: 547
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 793, 1092
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 11, 310
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00345161, 0.0020541, 0.994494, 
Output 1: 0.968388, 0.0310224, 0.00038997, 0.000199898, 
Output 2: 0.999, 0.000971684, 2.27151e-05, 5.16516e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1534!
Begin processing the 100th record. run: 6400858 subRun: 1 event: 59300 at 11-Sep-2025 08:42:36 -03
Boundary wire vector sizes: 42, 42, 32
minwire 0: 1909
minwire 1: 438
minwire 2: 2233
Used alternate method to get min and max wires due to vertex determination failure: 79, 378
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 29, 328
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.071872, 0.00475742, 0.923371, 
Output 1: 0.723749, 0.263449, 0.0122467, 0.000555463, 
Output 2: 0.980569, 0.0188272, 0.000576503, 2.69164e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 794!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e9/55/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_592_20231202T083531Z_gen_g4_detsim_hitreco__20240507T210043Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/a4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_870_20231203T050845Z_gen_g4_detsim_hitreco__20240508T072631Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e2/71/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_389_20231208T000819Z_gen_g4_detsim_hitreco__20240510T030814Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/38/34/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_641_20231207T033027Z_gen_g4_detsim_hitreco__20240510T024954Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b3/e0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_67_20231122T061042Z_gen_g4_detsim_hitreco__20240507T172445Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b7/71/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_1088_20231204T120519Z_gen_g4_detsim_hitreco__20240509T215339Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/19/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74576447_340_20231208T040032Z_gen_g4_detsim_hitreco__20240510T045800Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8f/5a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_654_20231208T062038Z_gen_g4_detsim_hitreco__20240510T051505Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/e1/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_646_20231201T210420Z_gen_g4_detsim_hitreco__20240507T222536Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/6d/7b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_148_20231207T163054Z_gen_g4_detsim_hitreco__20240510T064106Z_reco2.root
.:
total 5612
-rw-r--r-- 1 nobody nobody 2192742 Sep 11 08:42 flatcaf.root
-rw-r--r-- 1 nobody nobody 1697957 Sep 11 08:42 caf.root
-rw-r--r-- 1 nobody nobody 1697957 Sep 11 08:42 caf_fd_hd_atmo_2501_20250911T104839Z.root
-rw-r--r-- 1 nobody nobody   72482 Sep 11 08:42 caf_20250911T104839Z.log
-rw-r--r-- 1 nobody nobody   40969 Sep 11 08:42 jobscript.log
-rw-r--r-- 1 nobody nobody    1906 Sep 11 08:42 caf_20250911T104839Z.file
-rw-r--r-- 1 nobody nobody    1906 Sep 11 08:42 caf_20250911T104839Z.pfns
-rw-r--r-- 1 nobody nobody    1906 Sep 11 07:48 file.list
-rw-r--r-- 1 nobody nobody    1906 Sep 11 08:42 justin-processed-pfns.txt
-rw-r--r-- 1 nobody nobody    1374 Sep 11 07:48 all-input-dids.txt
-rw-r--r-- 1 nobody nobody    1374 Sep 11 08:42 caf_20250911T104839Z.did
-rw-r--r-- 1 nobody nobody    1374 Sep 11 07:48 did.list
-rw-r--r-- 1 nobody nobody       0 Sep 11 07:50 debugprod.log