Jobsub ID 47513.14@dunegpschedd01.fnal.gov

Jobsub ID	47513.14@dunegpschedd01.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-17 13:01:13
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce16-multicore
Last heartbeat	2025-09-17 13:19:30
From worker node	Hostname	hnode77.pic.es
	cpuinfo	AMD EPYC 7443P 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-17 13:01:36
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_646_20231208T050617Z_gen_g4_detsim_hitreco__20240510T041510Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_480_20231122T071607Z_gen_g4_detsim_hitreco__20240503T064414Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_820_20231202T082119Z_gen_g4_detsim_hitreco__20240507T200526Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_974_20231201T213844Z_gen_g4_detsim_hitreco__20240508T053019Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_280_20231203T075242Z_gen_g4_detsim_hitreco__20240509T204542Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_171_20231201T103734Z_gen_g4_detsim_hitreco__20240507T184344Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-17 13:19:30
Saved logs	justin-logs:47513.14-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

NEvaluator                               1.16e-05      0.196035       1.38734      0.253287      0.131679        300    
reco:energyrecnumu:EnergyReco                         0.000179641    0.0563039     0.498907     0.00724078     0.0900739       300    
reco:energyrecnue:EnergyReco                           9.155e-05    0.00123779     0.030447     0.000290989   0.00289307       300    
reco:energyrecnc:EnergyReco                           0.00107638     0.117686       0.42561      0.0226091     0.119692        300    
reco:energyrecnumurange:EnergyReco                    0.000127272    0.0150278     0.469185     0.00124883     0.0470324       300    
reco:energyrecnumumcs:EnergyReco                      0.000335775    0.0154638     0.475941     0.00150819     0.0471391       300    
reco:energyrecnumumcsllhd:EnergyReco                  0.000257364    0.0154266     0.476229     0.00146488     0.0471735       300    
reco:opdec:Deconvolution                               0.0195928     0.220151      0.506783      0.256074      0.103097        300    
reco:ophitspe:OpHitFinderDeco                         0.00292674     0.309939      0.478934       0.46122      0.215503        300    
reco:opflash:OpFlashFinder                            0.000118731   0.000738397    0.003359     0.000696521   0.000344389      300    
reco:opslicer:OpSlicer                                3.5201e-05     0.138511      0.435728      0.106926      0.108193        300    
reco:rns:RandomNumberSaver                             1.231e-05    2.02384e-05   0.000250463   1.9086e-05    1.38244e-05      300    
reco:anglereconue:NuAngularReco                       8.0801e-05    0.000771568    0.0190466    0.000238257   0.00180747       300    
reco:anglereconumu:NuAngularReco                      4.8551e-05    0.000646939    0.0184252    0.000125388   0.00177202       300    
reco:anglereconuepfps:NuAngularReco                   7.7631e-05    0.00071106     0.0180582    0.000188543   0.00178211       300    
reco:anglereconumupfps:NuAngularReco                   6.679e-05    0.000688727    0.0183794    0.000161413   0.00179074       300    
reco:anglerecohits:NuAngularReco                      6.1761e-05    0.00101609     0.0212914    0.000368271    0.0020456       300    
[art]:TriggerResults:TriggerResultInserter             6.38e-06     8.23668e-06   3.3291e-05     7.795e-06    2.17745e-06      300    
end_path:cafmaker:CAFMaker                            0.000819878   0.00943158     0.247373     0.00245705     0.0234962       300    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3692.83 MB
  Peak resident set size usage (VmHWM): 2023.89 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 17-Sep-2025 15:19:14 CEST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ea/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_820_20231202T082119Z_gen_g4_detsim_hitreco__20240507T200526Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.857253, 0.0717371, 0.0710104, 
Output 1: 0.169264, 0.308889, 0.243926, 0.277922, 
Output 2: 0.012878, 0.0377486, 0.176544, 0.772829, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3268!
Begin processing the 294th record. run: 6208838 subRun: 1 event: 96194 at 17-Sep-2025 15:17:47 CEST
Boundary wire vector sizes: 136, 157, 98
minwire 0: 798
minwire 1: 1716
minwire 2: 687
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.982851, 0.000868933, 0.0162799, 
Output 1: 0.930431, 0.0690994, 0.000410689, 5.90897e-05, 
Output 2: 0.988832, 0.0109333, 0.000217655, 1.67871e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1656!
Begin processing the 295th record. run: 6208838 subRun: 1 event: 96195 at 17-Sep-2025 15:17:50 CEST
Boundary wire vector sizes: 83, 44, 57
minwire 0: 1526
minwire 1: 1356
minwire 2: 1351
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.185593, 0.0133163, 0.801091, 
Output 1: 0.0441637, 0.216234, 0.498476, 0.241127, 
Output 2: 0.708548, 0.277242, 0.0137049, 0.000505371, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1975!
Begin processing the 296th record. run: 6208838 subRun: 1 event: 96196 at 17-Sep-2025 15:17:53 CEST
Boundary wire vector sizes: 1899, 2857, 2672
minwire 0: 506
minwire 1: 873
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.99585, 0.00125706, 0.00289318, 
Output 1: 0.0518028, 0.936012, 0.0117271, 0.000457853, 
Output 2: 0.684831, 0.300369, 0.0145025, 0.000298344, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2627!
Begin processing the 297th record. run: 6208838 subRun: 1 event: 96197 at 17-Sep-2025 15:17:59 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 651!
Begin processing the 298th record. run: 6208838 subRun: 1 event: 96198 at 17-Sep-2025 15:18:01 CEST
Boundary wire vector sizes: 182, 164, 151
minwire 0: 761
minwire 1: 528
minwire 2: 896
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000456968, 0.997675, 0.00186844, 
Output 1: 0.253407, 0.678061, 0.0620739, 0.0064579, 
Output 2: 0.995641, 0.0042367, 0.000116875, 5.49933e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2253!
Begin processing the 299th record. run: 6208838 subRun: 1 event: 96199 at 17-Sep-2025 15:18:05 CEST
Boundary wire vector sizes: 67, 42, 42
minwire 0: 1932
minwire 1: 1438
minwire 2: 1601
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00319072, 0.994287, 0.00252266, 
Output 1: 0.967054, 0.0326108, 0.000225556, 0.000109328, 
Output 2: 0.999019, 0.000948444, 2.47747e-05, 7.83795e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1464!
Begin processing the 300th record. run: 6208838 subRun: 1 event: 96200 at 17-Sep-2025 15:18:08 CEST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 472!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/60/ce/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_646_20231208T050617Z_gen_g4_detsim_hitreco__20240510T041510Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/85/4b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_480_20231122T071607Z_gen_g4_detsim_hitreco__20240503T064414Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ea/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_820_20231202T082119Z_gen_g4_detsim_hitreco__20240507T200526Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/b8/9b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_974_20231201T213844Z_gen_g4_detsim_hitreco__20240508T053019Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/62/5d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_280_20231203T075242Z_gen_g4_detsim_hitreco__20240509T204542Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c3/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515283_171_20231201T103734Z_gen_g4_detsim_hitreco__20240507T184344Z_reco2.root
.:
total 13732
-rw-r--r-- 1 dune001 dune 5227753 Sep 17 15:19 flatcaf.root
-rw-r--r-- 1 dune001 dune 4252435 Sep 17 15:19 caf.root
-rw-r--r-- 1 dune001 dune 4252435 Sep 17 15:19 caf_fd_hd_atmo_2501_20250917T130144Z.root
-rw-r--r-- 1 dune001 dune  191792 Sep 17 15:19 caf_20250917T130144Z.log
-rw-r--r-- 1 dune001 dune   85469 Sep 17 15:19 jobscript.log
-rw-r--r-- 1 dune001 dune    1148 Sep 17 15:19 caf_20250917T130144Z.file
-rw-r--r-- 1 dune001 dune    1148 Sep 17 15:19 caf_20250917T130144Z.pfns
-rw-r--r-- 1 dune001 dune    1148 Sep 17 15:01 file.list
-rw-r--r-- 1 dune001 dune    1148 Sep 17 15:19 justin-processed-pfns.txt
-rw-r--r-- 1 dune001 dune     825 Sep 17 15:01 all-input-dids.txt
-rw-r--r-- 1 dune001 dune     825 Sep 17 15:19 caf_20250917T130144Z.did
-rw-r--r-- 1 dune001 dune     825 Sep 17 15:01 did.list
-rw-r--r-- 1 dune001 dune       0 Sep 17 15:01 debugprod.log