Jobsub ID 40043.80@dunegpschedd02.fnal.gov

Jobsub ID	40043.80@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-15 14:25:46
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-09-15 16:17:53
From worker node	Hostname	mh-7763-3.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-15 14:27:25
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_108_20231206T230245Z_gen_g4_detsim_hitreco__20240510T024251Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_565_20231208T083543Z_gen_g4_detsim_hitreco__20240510T055341Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_152_20231204T120346Z_gen_g4_detsim_hitreco__20240509T215822Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_693_20231203T052003Z_gen_g4_detsim_hitreco__20240508T050629Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_627_20231203T120428Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_496_20231207T091040Z_gen_g4_detsim_hitreco__20240510T051813Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_134_20231207T082751Z_gen_g4_detsim_hitreco__20240510T033811Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_689_20231203T120629Z_gen_g4_detsim_hitreco__20240509T200042Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_502_20231207T042239Z_gen_g4_detsim_hitreco__20240510T055633Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-15 16:17:53
Saved logs	justin-logs:40043.80-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.000456027    0.0336514      2.78794     0.00350637     0.164375        600    
reco:opdec:Deconvolution                               0.416326      0.748484       2.09223      0.739884      0.188266        600    
reco:ophitspe:OpHitFinderDeco                           1.23615       1.30042       1.71446       1.33805      0.0582141       600    
reco:opflash:OpFlashFinder                            0.000145059   0.00136909     0.0043489    0.00130298    0.000593303      600    
reco:opslicer:OpSlicer                                 4.106e-05     0.278878       1.26028      0.215927      0.231865        600    
reco:rns:RandomNumberSaver                            2.2879e-05    3.88787e-05   0.000527287    3.55e-05     2.32467e-05      600    
reco:anglereconue:NuAngularReco                       0.000176228   0.00143456     0.153296     0.000411817    0.0067926       600    
reco:anglereconumu:NuAngularReco                       7.53e-05     0.00125683     0.151063     0.000237099   0.00671263       600    
reco:anglereconuepfps:NuAngularReco                   0.000112909   0.00137329     0.151925     0.000352833   0.00674303       600    
reco:anglereconumupfps:NuAngularReco                  0.000100579   0.00132962     0.150651     0.000309163    0.0067005       600    
reco:anglerecohits:NuAngularReco                      8.4949e-05    0.00196686      0.16257     0.000778486   0.00732935       600    
[art]:TriggerResults:TriggerResultInserter             9.11e-06     1.58226e-05   0.000250098    1.41e-05     1.25127e-05      600    
end_path:cafmaker:CAFMaker                            0.00119234     0.0173762      1.91848     0.00400773     0.084813        600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3774.21 MB
  Peak resident set size usage (VmHWM): 2079.9 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 15-Sep-2025 09:17:33 PDT ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_134_20231207T082751Z_gen_g4_detsim_hitreco__20240510T033811Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2371!
Begin processing the 595th record. run: 74550731 subRun: 1 event: 49695 at 15-Sep-2025 09:16:47 PDT
Boundary wire vector sizes: 111, 334, 201
minwire 0: 1848
minwire 1: 640
minwire 2: 1815
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.996146, 0.000830618, 0.00302297, 
Output 1: 0.00675612, 0.988725, 0.00440856, 0.000110835, 
Output 2: 0.99572, 0.00419273, 8.56527e-05, 1.75662e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1429!
Begin processing the 596th record. run: 74550731 subRun: 1 event: 49696 at 15-Sep-2025 09:16:55 PDT
Boundary wire vector sizes: 412, 332, 344
minwire 0: 1882
minwire 1: 291
minwire 2: 1766
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.459574, 0.210627, 0.329799, 
Output 1: 0.0210626, 0.27884, 0.577713, 0.122385, 
Output 2: 0.157946, 0.503534, 0.298752, 0.0397686, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1838!
Begin processing the 597th record. run: 74550731 subRun: 1 event: 49697 at 15-Sep-2025 09:17:02 PDT
Boundary wire vector sizes: 44, 38, 27
minwire 0: 294
minwire 1: 2129
minwire 2: 347
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.064497, 0.931184, 0.00431889, 
Output 1: 0.0198192, 0.898022, 0.0794841, 0.00267465, 
Output 2: 0.99605, 0.0037751, 0.000168176, 7.19882e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 747!
Begin processing the 598th record. run: 74550731 subRun: 1 event: 49698 at 15-Sep-2025 09:17:09 PDT
Boundary wire vector sizes: 52, 80, 65
minwire 0: 1249
minwire 1: 2321
minwire 2: 749
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.948264, 0.00302528, 0.0487108, 
Output 1: 0.0200356, 0.925035, 0.0535089, 0.00141996, 
Output 2: 0.95378, 0.0451957, 0.00100692, 1.75176e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1748!
Begin processing the 599th record. run: 74550731 subRun: 1 event: 49699 at 15-Sep-2025 09:17:17 PDT
Boundary wire vector sizes: 55, 46, 48
minwire 0: 813
minwire 1: 2301
minwire 2: 300
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0034912, 0.00195568, 0.994553, 
Output 1: 0.168725, 0.826882, 0.00424777, 0.000144794, 
Output 2: 0.99788, 0.00203011, 8.58975e-05, 4.36919e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1253!
Begin processing the 600th record. run: 74550731 subRun: 1 event: 49700 at 15-Sep-2025 09:17:24 PDT
Boundary wire vector sizes: 166, 81, 139
minwire 0: 2151
minwire 1: 258
minwire 2: 2537
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.986002, 0.00231768, 0.0116802, 
Output 1: 0.165207, 0.821782, 0.0126604, 0.00035141, 
Output 2: 0.988012, 0.0117826, 0.00020159, 4.00809e-06, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 768!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ef/82/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6453326_108_20231206T230245Z_gen_g4_detsim_hitreco__20240510T024251Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ef/3e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472706_565_20231208T083543Z_gen_g4_detsim_hitreco__20240510T055341Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/25/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_152_20231204T120346Z_gen_g4_detsim_hitreco__20240509T215822Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d7/64/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_693_20231203T052003Z_gen_g4_detsim_hitreco__20240508T050629Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/04/c7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_627_20231203T120428Z_gen_g4_detsim_hitreco__20240509T195933Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/4b/a6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74550731_496_20231207T091040Z_gen_g4_detsim_hitreco__20240510T051813Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6483936_134_20231207T082751Z_gen_g4_detsim_hitreco__20240510T033811Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/60/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_689_20231203T120629Z_gen_g4_detsim_hitreco__20240509T200042Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b4/fc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50607041_502_20231207T042239Z_gen_g4_detsim_hitreco__20240510T055633Z_reco2.root
.:
total 24576
-rw-r--r--. 1 cuser cuser 9265319 Sep 15 09:17 flatcaf.root
-rw-r--r--. 1 cuser cuser 7650215 Sep 15 09:17 caf.root
-rw-r--r--. 1 cuser cuser 7650215 Sep 15 09:17 caf_fd_hd_atmo_2501_20250915T142732Z.root
-rw-r--r--. 1 cuser cuser  362828 Sep 15 09:17 caf_20250915T142732Z.log
-rw-r--r--. 1 cuser cuser  136446 Sep 15 09:17 jobscript.log
-rw-r--r--. 1 cuser cuser    1985 Sep 15 09:17 caf_20250915T142732Z.file
-rw-r--r--. 1 cuser cuser    1985 Sep 15 09:17 caf_20250915T142732Z.pfns
-rw-r--r--. 1 cuser cuser    1985 Sep 15 07:27 file.list
-rw-r--r--. 1 cuser cuser    1985 Sep 15 09:17 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser    1373 Sep 15 07:27 all-input-dids.txt
-rw-r--r--. 1 cuser cuser    1373 Sep 15 09:17 caf_20250915T142732Z.did
-rw-r--r--. 1 cuser cuser    1373 Sep 15 07:27 did.list
-rw-r--r--. 1 cuser cuser       0 Sep 15 07:27 debugprod.log