Jobsub ID 40000.99@dunegpschedd02.fnal.gov
| Jobsub ID | 40000.99@dunegpschedd02.fnal.gov | |
| Workflow ID | 2501 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-15 13:11:43 | |
| Site | UK_Lancaster | |
| Entry | UBoone_UK_Lancaster_HEC_grendel_ce02 | |
| Last heartbeat | 2025-09-15 14:04:19 | |
| From worker node | Hostname | comp23-23 |
| cpuinfo | Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-09-15 13:13:20 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_47_20231207T004846Z_gen_g4_detsim_hitreco__20240510T041118Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_263_20231202T162322Z_gen_g4_detsim_hitreco__20240508T035610Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_37_20231207T161145Z_gen_g4_detsim_hitreco__20240510T030915Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_427_20231207T112218Z_gen_g4_detsim_hitreco__20240510T053333Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-09-15 14:04:19 | |
| Saved logs | justin-logs:40000.99-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
000934162 0.390099 100
reco:pandorapid:Chi2ParticleID 3.1678e-05 0.000146919 0.00164271 0.000105752 0.000200594 100
reco:cvnmap:CVNMapper 2.134e-05 0.0145407 0.0877632 0.0176601 0.0136071 100
reco:cvneva:CVNEvaluator 1.4329e-05 0.184574 3.25775 0.220905 0.328308 100
reco:energyrecnumu:EnergyReco 0.000214872 0.137189 3.12308 0.0359198 0.340948 100
reco:energyrecnue:EnergyReco 0.000124783 0.00426215 0.218204 0.000662062 0.0224494 100
reco:energyrecnc:EnergyReco 0.00177308 0.150294 0.655309 0.191721 0.117497 100
reco:energyrecnumurange:EnergyReco 0.000277823 0.0553132 1.75591 0.0025751 0.217014 100
reco:energyrecnumumcs:EnergyReco 0.000612246 0.0566704 1.85939 0.0027268 0.22487 100
reco:energyrecnumumcsllhd:EnergyReco 0.000506606 0.0576143 1.97298 0.00261999 0.233879 100
reco:opdec:Deconvolution 0.104681 0.276898 0.545948 0.272503 0.0819264 100
reco:ophitspe:OpHitFinderDeco 0.263328 0.276541 0.31287 0.276637 0.0060175 100
reco:opflash:OpFlashFinder 0.00058946 0.00187747 0.020132 0.00163729 0.00190682 100
reco:opslicer:OpSlicer 0.000422463 0.195629 0.840883 0.140241 0.167598 100
reco:rns:RandomNumberSaver 2.1509e-05 3.43255e-05 0.000345715 2.81685e-05 3.24157e-05 100
reco:anglereconue:NuAngularReco 0.000152636 0.00245488 0.122828 0.000318878 0.0127859 100
reco:anglereconumu:NuAngularReco 6.6235e-05 0.0022111 0.114279 0.000184027 0.0119724 100
reco:anglereconuepfps:NuAngularReco 0.000100772 0.00231533 0.114802 0.000265711 0.0120197 100
reco:anglereconumupfps:NuAngularReco 8.6297e-05 0.0022744 0.115273 0.000247089 0.0120644 100
reco:anglerecohits:NuAngularReco 0.000219872 0.00276353 0.118185 0.000522225 0.0125292 100
[art]:TriggerResults:TriggerResultInserter 9.5e-06 1.22923e-05 7.1215e-05 1.105e-05 6.32344e-06 100
end_path:cafmaker:CAFMaker 0.00126852 0.0452816 1.73118 0.00448639 0.217459 100
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4059.8 MB
Peak resident set size usage (VmHWM): 2385.77 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 15-Sep-2025 14:42:34 BST ModuleEndJob
---- FileOpenError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Open
[FATAL] Socket timeout
ROOT severity: 3000
---- FatalRootError END
RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/a9/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_263_20231202T162322Z_gen_g4_detsim_hitreco__20240508T035610Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 474!
Begin processing the 92nd record. run: 74530544 subRun: 1 event: 4792 at 15-Sep-2025 14:34:18 BST
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 239!
Begin processing the 93rd record. run: 74530544 subRun: 1 event: 4793 at 15-Sep-2025 14:34:21 BST
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 762!
Begin processing the 94th record. run: 74530544 subRun: 1 event: 4794 at 15-Sep-2025 14:34:23 BST
Boundary wire vector sizes: 253, 504, 446
minwire 0: 1701
minwire 1: 444
minwire 2: 1884
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.980872, 0.00103144, 0.0180961,
Output 1: 0.0452036, 0.872384, 0.0800929, 0.00231912,
Output 2: 0.869437, 0.12744, 0.00307421, 4.83155e-05,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1239!
Begin processing the 95th record. run: 74530544 subRun: 1 event: 4795 at 15-Sep-2025 14:34:27 BST
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 769!
Begin processing the 96th record. run: 74530544 subRun: 1 event: 4796 at 15-Sep-2025 14:34:29 BST
Boundary wire vector sizes: 5803, 5602, 4223
minwire 0: 189
minwire 1: 1826
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.000439897, 0.995874, 0.00368579,
Output 1: 0.780064, 0.21215, 0.00616136, 0.00162427,
Output 2: 0.140821, 0.704251, 0.137177, 0.0177511,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3035!
Begin processing the 97th record. run: 74530544 subRun: 1 event: 4797 at 15-Sep-2025 14:34:50 BST
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 454!
Begin processing the 98th record. run: 74530544 subRun: 1 event: 4798 at 15-Sep-2025 14:34:53 BST
Boundary wire vector sizes: 56, 38, 29
minwire 0: 1802
minwire 1: 711
minwire 2: 2024
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.0509434, 0.00640139, 0.942655,
Output 1: 0.0117423, 0.0510715, 0.377842, 0.559345,
Output 2: 0.987175, 0.0122242, 0.000546917, 5.42946e-05,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1318!
Begin processing the 99th record. run: 74530544 subRun: 1 event: 4799 at 15-Sep-2025 14:34:56 BST
Boundary wire vector sizes: 81, 58, 74
minwire 0: 720
minwire 1: 1912
minwire 2: 563
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.0197565, 0.0022446, 0.977999,
Output 1: 0.0163517, 0.0585491, 0.296087, 0.629012,
Output 2: 0.973966, 0.0244984, 0.00136812, 0.000167277,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1415!
Begin processing the 100th record. run: 74530544 subRun: 1 event: 4800 at 15-Sep-2025 14:34:59 BST
Boundary wire vector sizes: 1100, 1074, 1005
minwire 0: 1212
minwire 1: 763
minwire 2: 1829
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.853745, 0.0731716, 0.0730836,
Output 1: 0.155413, 0.419934, 0.259396, 0.165257,
Output 2: 0.0476983, 0.13092, 0.342361, 0.479021,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2638!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4f/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74530544_47_20231207T004846Z_gen_g4_detsim_hitreco__20240510T041118Z_reco2.root
root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/a9/35/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_263_20231202T162322Z_gen_g4_detsim_hitreco__20240508T035610Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/28/4f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_37_20231207T161145Z_gen_g4_detsim_hitreco__20240510T030915Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/65/09/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_427_20231207T112218Z_gen_g4_detsim_hitreco__20240510T053333Z_reco2.root
.:
total 5720
-rw-r--r-- 1 pltdune002 pltdune 2243768 Sep 15 14:42 flatcaf.root
-rw-r--r-- 1 pltdune002 pltdune 1733493 Sep 15 14:42 caf.root
-rw-r--r-- 1 pltdune002 pltdune 1733493 Sep 15 14:42 caf_fd_hd_atmo_2501_20250915T131800Z.root
-rw-r--r-- 1 pltdune002 pltdune 69917 Sep 15 14:42 caf_20250915T131800Z.log
-rw-r--r-- 1 pltdune002 pltdune 34981 Sep 15 14:42 jobscript.log
-rw-r--r-- 1 pltdune002 pltdune 747 Sep 15 14:42 caf_20250915T131800Z.file
-rw-r--r-- 1 pltdune002 pltdune 747 Sep 15 14:42 caf_20250915T131800Z.pfns
-rw-r--r-- 1 pltdune002 pltdune 747 Sep 15 14:17 file.list
-rw-r--r-- 1 pltdune002 pltdune 747 Sep 15 14:42 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune002 pltdune 549 Sep 15 14:17 all-input-dids.txt
-rw-r--r-- 1 pltdune002 pltdune 549 Sep 15 14:42 caf_20250915T131800Z.did
-rw-r--r-- 1 pltdune002 pltdune 549 Sep 15 14:17 did.list
-rw-r--r-- 1 pltdune002 pltdune 0 Sep 15 14:18 debugprod.log