Jobsub ID 149785.190@dunegpschedd02.fnal.gov
| Jobsub ID | 149785.190@dunegpschedd02.fnal.gov | |
| Workflow ID | 3302 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-09-23 15:41:58 | |
| Site | UK_RAL-PPD | |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep208 | |
| Last heartbeat | 2025-09-23 16:09:58 | |
| From worker node | Hostname | heplnc161.pp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-09-23 15:44:11 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_310_20231201T170020Z_gen_g4_detsim_hitreco__20240507T195012Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_305_20231207T124355Z_gen_g4_detsim_hitreco__20240510T043145Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_768_20231202T181919Z_gen_g4_detsim_hitreco__20240508T064502Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-09-23 16:09:58 | |
| Saved logs | justin-logs:149785.190-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
lorimetry 9.9006e-05 0.0192474 0.722516 0.00137839 0.0749905 200
reco:pandorapid:Chi2ParticleID 3.4865e-05 0.000188622 0.00311381 0.000141225 0.000239202 200
reco:cvnmap:CVNMapper 2.1501e-05 0.00941782 0.052149 0.00924517 0.00734577 200
reco:cvneva:CVNEvaluator 1.7783e-05 0.355757 2.3442 0.445828 0.240498 200
reco:energyrecnumu:EnergyReco 0.00027701 0.0651409 0.962352 0.021825 0.106269 200
reco:energyrecnue:EnergyReco 0.000125105 0.00225538 0.0347602 0.000786915 0.00438383 200
reco:energyrecnc:EnergyReco 0.00230176 0.0850549 0.276722 0.100685 0.0625514 200
reco:energyrecnumurange:EnergyReco 0.000327153 0.0266293 0.796538 0.00324074 0.0888675 200
reco:energyrecnumumcs:EnergyReco 0.000620754 0.0261222 0.655012 0.00366639 0.0781664 200
reco:energyrecnumumcsllhd:EnergyReco 0.000476994 0.0270845 0.747204 0.00363253 0.0868447 200
reco:opdec:Deconvolution 0.0712676 0.246262 0.452332 0.250437 0.0741881 200
reco:ophitspe:OpHitFinderDeco 0.197691 0.206298 0.214263 0.206277 0.00303321 200
reco:opflash:OpFlashFinder 0.000140283 0.00144142 0.00590876 0.0013656 0.000682816 200
reco:opslicer:OpSlicer 4.5015e-05 0.273015 1.05481 0.225521 0.217813 200
reco:rns:RandomNumberSaver 2.3755e-05 4.657e-05 0.00071977 3.97345e-05 4.95522e-05 200
reco:anglereconue:NuAngularReco 0.000192552 0.00137245 0.0247946 0.000488316 0.00283585 200
reco:anglereconumu:NuAngularReco 8.8135e-05 0.00116201 0.0244923 0.00028775 0.00278918 200
reco:anglereconuepfps:NuAngularReco 0.000126177 0.00130045 0.0247204 0.000402685 0.00283815 200
reco:anglereconumupfps:NuAngularReco 0.000104596 0.00122961 0.0247084 0.000360586 0.00281367 200
reco:anglerecohits:NuAngularReco 0.00010637 0.00189726 0.0284287 0.000854171 0.00328725 200
[art]:TriggerResults:TriggerResultInserter 9.959e-06 1.60207e-05 8.0431e-05 1.4477e-05 8.45887e-06 200
end_path:cafmaker:CAFMaker 0.00142024 0.0177568 0.279786 0.00567596 0.0370297 200
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 3654.78 MB
Peak resident set size usage (VmHWM): 1943.63 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 23-Sep-2025 17:09:33 BST ModuleEndJob
---- FileOpenError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Open
[FATAL] Socket timeout
ROOT severity: 3000
---- FatalRootError END
RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c9/8b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_768_20231202T181919Z_gen_g4_detsim_hitreco__20240508T064502Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
lar exit code 1
=== Start last 100 lines of lar log file ===
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 188!
Begin processing the 195th record. run: 50624901 subRun: 1 event: 30595 at 23-Sep-2025 17:01:39 BST
Boundary wire vector sizes: 185, 224, 117
minwire 0: 2318
minwire 1: 1040
minwire 2: 2161
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.242861, 0.641669, 0.115469,
Output 1: 0.0116684, 0.978961, 0.00885783, 0.000513087,
Output 2: 0.355509, 0.636517, 0.0079262, 4.78575e-05,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1779!
Begin processing the 196th record. run: 50624901 subRun: 1 event: 30596 at 23-Sep-2025 17:01:42 BST
Boundary wire vector sizes: 934, 995, 687
minwire 0: 38
minwire 1: 1802
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.470187, 0.100022, 0.429791,
Output 1: 0.0288118, 0.165639, 0.32163, 0.483919,
Output 2: 0.104793, 0.405404, 0.362413, 0.12739,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1701!
Begin processing the 197th record. run: 50624901 subRun: 1 event: 30597 at 23-Sep-2025 17:01:46 BST
Boundary wire vector sizes: 758, 353, 594
minwire 0: 1412
minwire 1: 874
minwire 2: 1603
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.994894, 0.00072767, 0.00437849,
Output 1: 0.0290403, 0.723599, 0.223791, 0.0235696,
Output 2: 0.0778789, 0.897428, 0.0245597, 0.000133125,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1947!
Begin processing the 198th record. run: 50624901 subRun: 1 event: 30598 at 23-Sep-2025 17:01:51 BST
Boundary wire vector sizes: 725, 743, 619
minwire 0: 2142
minwire 1: 429
minwire 2: 2003
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.0126095, 0.0404193, 0.946971,
Output 1: 0.190876, 0.357465, 0.257993, 0.193666,
Output 2: 0.0591799, 0.303194, 0.419986, 0.21764,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2232!
Begin processing the 199th record. run: 50624901 subRun: 1 event: 30599 at 23-Sep-2025 17:01:55 BST
Boundary wire vector sizes: 115, 111, 98
minwire 0: 1758
minwire 1: 1007
minwire 2: 1858
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.992149, 0.00112549, 0.00672517,
Output 1: 0.00229106, 0.99491, 0.00272316, 7.59819e-05,
Output 2: 0.98965, 0.0101295, 0.000216527, 3.91707e-06,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1346!
Begin processing the 200th record. run: 50624901 subRun: 1 event: 30600 at 23-Sep-2025 17:01:58 BST
Boundary wire vector sizes: 222, 177, 207
minwire 0: 1130
minwire 1: 1537
minwire 2: 1071
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299
Classifier summary:
Output 0: 0.000775735, 0.993963, 0.00526081,
Output 1: 0.976993, 0.0226418, 0.000275688, 8.97359e-05,
Output 2: 0.995417, 0.00444877, 0.000114213, 2.02397e-05,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1405!
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/23/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494002_310_20231201T170020Z_gen_g4_detsim_hitreco__20240507T195012Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e4/52/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_305_20231207T124355Z_gen_g4_detsim_hitreco__20240510T043145Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c9/8b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_768_20231202T181919Z_gen_g4_detsim_hitreco__20240508T064502Z_reco2.root
.:
total 9432
-rw-r--r-- 1 pltdune04 pltdune 3620179 Sep 23 17:09 flatcaf.root
-rw-r--r-- 1 pltdune04 pltdune 2902750 Sep 23 17:09 caf.root
-rw-r--r-- 1 pltdune04 pltdune 2902750 Sep 23 17:09 caf_fd_hd_atmo_3302_20250923T154422Z.root
-rw-r--r-- 1 pltdune04 pltdune 136568 Sep 23 17:09 caf_20250923T154422Z.log
-rw-r--r-- 1 pltdune04 pltdune 60488 Sep 23 17:09 jobscript.log
-rw-r--r-- 1 pltdune04 pltdune 593 Sep 23 17:09 caf_20250923T154422Z.file
-rw-r--r-- 1 pltdune04 pltdune 593 Sep 23 17:09 caf_20250923T154422Z.pfns
-rw-r--r-- 1 pltdune04 pltdune 593 Sep 23 16:44 file.list
-rw-r--r-- 1 pltdune04 pltdune 593 Sep 23 17:09 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune04 pltdune 414 Sep 23 16:44 all-input-dids.txt
-rw-r--r-- 1 pltdune04 pltdune 414 Sep 23 17:09 caf_20250923T154422Z.did
-rw-r--r-- 1 pltdune04 pltdune 414 Sep 23 16:44 did.list
-rw-r--r-- 1 pltdune04 pltdune 0 Sep 23 16:44 debugprod.log