Jobsub ID 39988.120@dunegpschedd02.fnal.gov

Jobsub ID	39988.120@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 12:33:41
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-09-15 12:56:42
From worker node	Hostname	mh-7763-3.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 12:35:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_973_20231203T134557Z_gen_g4_detsim_hitreco__20240508T053549Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_667_20231208T011240Z_gen_g4_detsim_hitreco__20240510T031720Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_251_20231202T100322Z_gen_g4_detsim_hitreco__20240508T070543Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_118_20231121T153309Z_gen_g4_detsim_hitreco__20240503T054053Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_507_20231204T214820Z_gen_g4_detsim_hitreco__20240510T030605Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_616_20231203T042107Z_gen_g4_detsim_hitreco__20240508T071949Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_751_20231207T074715Z_gen_g4_detsim_hitreco__20240510T022414Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_64_20231207T045928Z_gen_g4_detsim_hitreco__20240510T023629Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_300_20231203T035705Z_gen_g4_detsim_hitreco__20240508T074331Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_856_20231202T082345Z_gen_g4_detsim_hitreco__20240507T200410Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 12:56:42
Saved logs	justin-logs:39988.120-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.507453      0.0031508     0.0927913       100    
reco:energyrecnumumcs:EnergyReco                      0.000732096    0.0286292     0.487622     0.00373534     0.0873176       100    
reco:energyrecnumumcsllhd:EnergyReco                  0.000435788    0.0282971     0.490043     0.00369919     0.0862267       100    
reco:opdec:Deconvolution                               0.491384      0.763726       1.83833       0.76258       0.21439        100    
reco:ophitspe:OpHitFinderDeco                           1.35908       1.36974       1.72703        1.365       0.0363146       100    
reco:opflash:OpFlashFinder                            0.000357348    0.0014494    0.00370515    0.00129991    0.000702636      100    
reco:opslicer:OpSlicer                                0.00228383     0.272419      0.888914      0.191275      0.240949        100    
reco:rns:RandomNumberSaver                             2.378e-05    5.30724e-05   0.000471307   4.46845e-05   4.77986e-05      100    
reco:anglereconue:NuAngularReco                       0.000177868   0.00148577     0.0181658    0.000478547   0.00326444       100    
reco:anglereconumu:NuAngularReco                       9.257e-05    0.00126807     0.0167237    0.000273693   0.00321158       100    
reco:anglereconuepfps:NuAngularReco                   0.000136259   0.00141398     0.0172159    0.000389228   0.00322691       100    
reco:anglereconumupfps:NuAngularReco                  0.00010846    0.00136474     0.0168648    0.000342678   0.00325663       100    
reco:anglerecohits:NuAngularReco                      0.000357647   0.00205815     0.0200735    0.000829365   0.00382994       100    
[art]:TriggerResults:TriggerResultInserter             9.41e-06     1.66618e-05   0.000104809   1.4985e-05    1.04677e-05      100    
end_path:cafmaker:CAFMaker                            0.00143978     0.160467       14.1303     0.00663869      1.40459        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3648.89 MB
  Peak resident set size usage (VmHWM): 1838.55 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 05:56:16 PDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/bf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_667_20231208T011240Z_gen_g4_detsim_hitreco__20240510T031720Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Output 0: 0.211958, 0.0166523, 0.77139, 
Output 1: 0.614867, 0.377076, 0.0073488, 0.000708021, 
Output 2: 0.160289, 0.426733, 0.337482, 0.075496, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2249!
Begin processing the 93rd record. run: 50574832 subRun: 1 event: 97393 at 15-Sep-2025 05:54:50 PDT
Boundary wire vector sizes: 263, 266, 84
minwire 0: 42
minwire 1: 1924
minwire 2: 8
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.991481, 0.000497652, 0.00802109, 
Output 1: 0.00668179, 0.105765, 0.84123, 0.0463228, 
Output 2: 0.898985, 0.0958534, 0.00506884, 9.2717e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1050!
Begin processing the 94th record. run: 50574832 subRun: 1 event: 97394 at 15-Sep-2025 05:54:58 PDT
Boundary wire vector sizes: 4112, 4815, 3066
minwire 0: 835
minwire 1: 1422
minwire 2: 643
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.199277, 0.527241, 0.273482, 
Output 1: 0.267909, 0.401248, 0.18686, 0.143983, 
Output 2: 0.0148724, 0.0483426, 0.189928, 0.746857, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3374!
Begin processing the 95th record. run: 50574832 subRun: 1 event: 97395 at 15-Sep-2025 05:55:29 PDT
Boundary wire vector sizes: 424, 348, 290
minwire 0: 1232
minwire 1: 789
minwire 2: 1376
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995595, 0.00031567, 0.00408948, 
Output 1: 0.992328, 0.00737504, 0.000166752, 0.000130063, 
Output 2: 0.997001, 0.00291396, 6.56808e-05, 1.91038e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1507!
Begin processing the 96th record. run: 50574832 subRun: 1 event: 97396 at 15-Sep-2025 05:55:37 PDT
Boundary wire vector sizes: 171, 248, 137
minwire 0: 527
minwire 1: 1873
minwire 2: 412
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000626051, 0.996916, 0.00245796, 
Output 1: 0.00650004, 0.989412, 0.00395392, 0.00013415, 
Output 2: 0.997529, 0.00240206, 6.69579e-05, 2.25409e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1156!
Begin processing the 97th record. run: 50574832 subRun: 1 event: 97397 at 15-Sep-2025 05:55:45 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 436!
Begin processing the 98th record. run: 50574832 subRun: 1 event: 97398 at 15-Sep-2025 05:55:52 PDT
Boundary wire vector sizes: 346, 329, 163
minwire 0: 1449
minwire 1: 1520
minwire 2: 1350
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00024453, 0.998836, 0.000919798, 
Output 1: 0.00401, 0.99182, 0.00402293, 0.000147224, 
Output 2: 0.998097, 0.00185393, 4.76594e-05, 1.46551e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2213!
Begin processing the 99th record. run: 50574832 subRun: 1 event: 97399 at 15-Sep-2025 05:56:01 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 806!
Begin processing the 100th record. run: 50574832 subRun: 1 event: 97400 at 15-Sep-2025 05:56:08 PDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1380!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/72/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_973_20231203T134557Z_gen_g4_detsim_hitreco__20240508T053549Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/bf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_667_20231208T011240Z_gen_g4_detsim_hitreco__20240510T031720Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/25/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_251_20231202T100322Z_gen_g4_detsim_hitreco__20240508T070543Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/04/41/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_118_20231121T153309Z_gen_g4_detsim_hitreco__20240503T054053Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/14/34/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_507_20231204T214820Z_gen_g4_detsim_hitreco__20240510T030605Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fa/6c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_616_20231203T042107Z_gen_g4_detsim_hitreco__20240508T071949Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/e9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74548636_751_20231207T074715Z_gen_g4_detsim_hitreco__20240510T022414Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/90/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_64_20231207T045928Z_gen_g4_detsim_hitreco__20240510T023629Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b1/dd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_300_20231203T035705Z_gen_g4_detsim_hitreco__20240508T074331Z_reco2.root
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/88/a1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_856_20231202T082345Z_gen_g4_detsim_hitreco__20240507T200410Z_reco2.root
.:
total 4240
-rw-r--r--. 1 cuser cuser 1679257 Sep 15 05:56 flatcaf.root
-rw-r--r--. 1 cuser cuser 1254387 Sep 15 05:56 caf.root
-rw-r--r--. 1 cuser cuser 1254387 Sep 15 05:56 caf_fd_hd_atmo_2501_20250915T123529Z.root
-rw-r--r--. 1 cuser cuser   74040 Sep 15 05:56 caf_20250915T123529Z.log
-rw-r--r--. 1 cuser cuser   37706 Sep 15 05:56 jobscript.log
-rw-r--r--. 1 cuser cuser    1882 Sep 15 05:56 caf_20250915T123529Z.file
-rw-r--r--. 1 cuser cuser    1882 Sep 15 05:56 caf_20250915T123529Z.pfns
-rw-r--r--. 1 cuser cuser    1882 Sep 15 05:35 file.list
-rw-r--r--. 1 cuser cuser    1882 Sep 15 05:56 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser    1378 Sep 15 05:35 all-input-dids.txt
-rw-r--r--. 1 cuser cuser    1378 Sep 15 05:56 caf_20250915T123529Z.did
-rw-r--r--. 1 cuser cuser    1378 Sep 15 05:35 did.list
-rw-r--r--. 1 cuser cuser       0 Sep 15 05:35 debugprod.log