Jobsub ID 38122.89@dunegpschedd02.fnal.gov

Jobsub ID	38122.89@dunegpschedd02.fnal.gov
Workflow ID	2501
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-09-11 10:02:34
Site	US_PuertoRico
Entry	CMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat	2025-09-11 13:34:52
From worker node	Hostname	compute-0-2.local
	cpuinfo	Intel(R) Xeon(R) CPU W3550 @ 3.07GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-09-11 10:53:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_378_20231202T054342Z_gen_g4_detsim_hitreco__20240508T033056Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_593_20231203T081600Z_gen_g4_detsim_hitreco__20240509T210155Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_851_20231204T120455Z_gen_g4_detsim_hitreco__20240509T202448Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_881_20231204T225940Z_gen_g4_detsim_hitreco__20240509T221826Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_1073_20231205T010724Z_gen_g4_detsim_hitreco__20240510T022533Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_270_20231207T071935Z_gen_g4_detsim_hitreco__20240510T063115Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_268_20231208T041613Z_gen_g4_detsim_hitreco__20240510T034743Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_743_20231201T170351Z_gen_g4_detsim_hitreco__20240507T201523Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_737_20231208T094019Z_gen_g4_detsim_hitreco__20240510T064253Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_193_20231201T121428Z_gen_g4_detsim_hitreco__20240507T194400Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-09-11 13:34:52
Saved logs	justin-logs:38122.89-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1.79562      0.835651      0.182859        600    
reco:ophitspe:OpHitFinderDeco                           1.34095       1.43731       1.65527       1.47244      0.0606946       600    
reco:opflash:OpFlashFinder                            0.000160581    0.0018314    0.00575773    0.00168508    0.000924249      600    
reco:opslicer:OpSlicer                                5.1818e-05     0.295104       1.22477      0.240613      0.250101        600    
reco:rns:RandomNumberSaver                            2.2252e-05    3.54259e-05   0.000348329   3.24115e-05   1.53276e-05      600    
reco:anglereconue:NuAngularReco                       0.000181164   0.00266762     0.331072     0.000451671    0.0174386       600    
reco:anglereconumu:NuAngularReco                      0.000112465    0.0025746     0.329225     0.000316433    0.0174946       600    
reco:anglereconuepfps:NuAngularReco                   0.000160728   0.00270133     0.331354     0.000443389    0.0175279       600    
reco:anglereconumupfps:NuAngularReco                  0.000157785    0.0026904     0.330063     0.000440167    0.0175155       600    
reco:anglerecohits:NuAngularReco                      0.000131027   0.00337643      0.33752     0.00100886     0.0179915       600    
[art]:TriggerResults:TriggerResultInserter            1.3645e-05    1.98912e-05   6.0768e-05    1.7868e-05    6.06879e-06      600    
end_path:cafmaker:CAFMaker                             0.001472      0.0590607      17.7538     0.00552059     0.742828        600    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3803.64 MB
  Peak resident set size usage (VmHWM): 1884.51 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 11-Sep-2025 09:34:28 AST ModuleEndJob
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a5/a1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_268_20231208T041613Z_gen_g4_detsim_hitreco__20240510T034743Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 337, 148, 245
minwire 0: 1310
minwire 1: 985
minwire 2: 1217
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.995092, 0.000409673, 0.00449856, 
Output 1: 0.0138657, 0.343323, 0.621859, 0.0209527, 
Output 2: 0.987064, 0.0123905, 0.000535274, 1.06028e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1790!
Begin processing the 596th record. run: 6452759 subRun: 1 event: 27096 at 11-Sep-2025 09:33:27 AST
Boundary wire vector sizes: 1245, 1418, 1034
minwire 0: 521
minwire 1: 1694
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.000565999, 0.996397, 0.00303657, 
Output 1: 0.418319, 0.556302, 0.0239373, 0.00144224, 
Output 2: 0.883958, 0.112899, 0.00305566, 8.74742e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2143!
Begin processing the 597th record. run: 6452759 subRun: 1 event: 27097 at 11-Sep-2025 09:33:41 AST
Boundary wire vector sizes: 316, 305, 199
minwire 0: 1911
minwire 1: 364
minwire 2: 2243
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.0129848, 0.696555, 0.29046, 
Output 1: 0.644669, 0.334264, 0.0196306, 0.00143633, 
Output 2: 0.880493, 0.114355, 0.00483921, 0.000312378, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1266!
Begin processing the 598th record. run: 6452759 subRun: 1 event: 27098 at 11-Sep-2025 09:33:52 AST
Boundary wire vector sizes: 94, 68, 86
minwire 0: 2459
minwire 1: 971
minwire 2: 2094
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.277966, 0.706999, 0.0150354, 
Output 1: 0.00787979, 0.0371479, 0.392861, 0.562111, 
Output 2: 0.980936, 0.0179979, 0.000924665, 0.000141699, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1620!
Begin processing the 599th record. run: 6452759 subRun: 1 event: 27099 at 11-Sep-2025 09:34:03 AST
Boundary wire vector sizes: 64, 70, 52
minwire 0: 1740
minwire 1: 653
minwire 2: 1783
Used alternate method to get min and max wires due to vertex determination failure: 101, 400
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.217242, 0.0183985, 0.76436, 
Output 1: 0.947718, 0.0504565, 0.00175234, 7.30291e-05, 
Output 2: 0.317183, 0.673687, 0.00903672, 9.35597e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1337!
Begin processing the 600th record. run: 6452759 subRun: 1 event: 27100 at 11-Sep-2025 09:34:15 AST
Boundary wire vector sizes: 50, 51, 46
minwire 0: 848
minwire 1: 2108
minwire 2: 254
Used alternate method to get min and max wires due to vertex determination failure: 66, 365
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max wires due to vertex determination failure: 308, 607
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary: 
Output 0: 0.00674023, 0.00198559, 0.991274, 
Output 1: 0.105866, 0.726136, 0.162484, 0.00551385, 
Output 2: 0.991978, 0.00759197, 0.000415814, 1.46846e-05, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1572!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/15/de/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_378_20231202T054342Z_gen_g4_detsim_hitreco__20240508T033056Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0a/fa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_593_20231203T081600Z_gen_g4_detsim_hitreco__20240509T210155Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0a/a2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_851_20231204T120455Z_gen_g4_detsim_hitreco__20240509T202448Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/23/59/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_881_20231204T225940Z_gen_g4_detsim_hitreco__20240509T221826Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/32/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_1073_20231205T010724Z_gen_g4_detsim_hitreco__20240510T022533Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4e/f2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_270_20231207T071935Z_gen_g4_detsim_hitreco__20240510T063115Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a5/a1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6471483_268_20231208T041613Z_gen_g4_detsim_hitreco__20240510T034743Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/61/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_743_20231201T170351Z_gen_g4_detsim_hitreco__20240507T201523Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bc/a0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472782_737_20231208T094019Z_gen_g4_detsim_hitreco__20240510T064253Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/6c/89/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_193_20231201T121428Z_gen_g4_detsim_hitreco__20240507T194400Z_reco2.root
.:
total 24348
-rw-r--r-- 1 dunepilot gridusers 9184602 Sep 11 09:34 flatcaf.root
-rw-r--r-- 1 dunepilot gridusers 7594634 Sep 11 09:34 caf.root
-rw-r--r-- 1 dunepilot gridusers 7594634 Sep 11 09:34 caf_fd_hd_atmo_2501_20250911T105331Z.root
-rw-r--r-- 1 dunepilot gridusers  359401 Sep 11 09:34 caf_20250911T105331Z.log
-rw-r--r-- 1 dunepilot gridusers  139876 Sep 11 09:34 jobscript.log
-rw-r--r-- 1 dunepilot gridusers    1960 Sep 11 09:34 caf_20250911T105331Z.file
-rw-r--r-- 1 dunepilot gridusers    1960 Sep 11 09:34 caf_20250911T105331Z.pfns
-rw-r--r-- 1 dunepilot gridusers    1960 Sep 11 06:53 file.list
-rw-r--r-- 1 dunepilot gridusers    1960 Sep 11 09:34 justin-processed-pfns.txt
-rw-r--r-- 1 dunepilot gridusers    1376 Sep 11 06:53 all-input-dids.txt
-rw-r--r-- 1 dunepilot gridusers    1376 Sep 11 09:34 caf_20250911T105331Z.did
-rw-r--r-- 1 dunepilot gridusers    1376 Sep 11 06:53 did.list
-rw-r--r-- 1 dunepilot gridusers       0 Sep 11 06:54 debugprod.log