Jobsub ID 226590.38@dunegpschedd02.fnal.gov
| Jobsub ID | 226590.38@dunegpschedd02.fnal.gov |
| Workflow ID | 8772 |
| Stage ID | 1 |
| User name | lwhite86@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 3600 (1 hours) |
| Submitted time | 2025-10-09 10:22:37 |
| Site | US_UCSD |
| Entry | CMSHTPC_T2_US_UCSD_gw7 |
| Last heartbeat | 2025-10-09 10:54:12 |
| From worker node | Hostname | mh-7763-13.t2.ucsd.edu |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2025-10-09 10:25:17 |
| Input files | fardet-hd:nue_dune10kt_1x2x6_1418_972_20230827T014216Z_gen_g4_detsim_hitreco__20240221T051923Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 28m (1707s) |
| CPU time | 5m (307s = 17%) |
| Max RSS bytes | 1195130880 (1139 MiB) |
| Outputting started | 2025-10-09 10:53:45 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/08772/1/001/trainingFile_nue_dune10kt_1x2x6_1418_972_20230827T014216Z_gen_g4_detsim_hitreco__20240221T051923Z_reco2.root
|
| Finished | 2025-10-09 10:54:12 |
| Saved logs | justin-logs:226590.38-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
Oct-2025 03:35:23 PDT
221
178
929
453
144
Begin processing the 36th record. run: 1418 subRun: 1 event: 97236 at 09-Oct-2025 03:36:05 PDT
160
36
204
Begin processing the 37th record. run: 1418 subRun: 1 event: 97237 at 09-Oct-2025 03:36:26 PDT
Begin processing the 38th record. run: 1418 subRun: 1 event: 97238 at 09-Oct-2025 03:36:42 PDT
408
97
Begin processing the 39th record. run: 1418 subRun: 1 event: 97239 at 09-Oct-2025 03:36:55 PDT
572
747
225
43
185
37
Begin processing the 40th record. run: 1418 subRun: 1 event: 97240 at 09-Oct-2025 03:37:24 PDT
27
Begin processing the 41st record. run: 1418 subRun: 1 event: 97241 at 09-Oct-2025 03:37:40 PDT
382
117
Begin processing the 42nd record. run: 1418 subRun: 1 event: 97242 at 09-Oct-2025 03:37:53 PDT
Begin processing the 43rd record. run: 1418 subRun: 1 event: 97243 at 09-Oct-2025 03:37:55 PDT
Begin processing the 44th record. run: 1418 subRun: 1 event: 97244 at 09-Oct-2025 03:37:58 PDT
120
63
Begin processing the 45th record. run: 1418 subRun: 1 event: 97245 at 09-Oct-2025 03:38:02 PDT
44
119
Begin processing the 46th record. run: 1418 subRun: 1 event: 97246 at 09-Oct-2025 03:38:16 PDT
Begin processing the 47th record. run: 1418 subRun: 1 event: 97247 at 09-Oct-2025 03:38:18 PDT
422
Begin processing the 48th record. run: 1418 subRun: 1 event: 97248 at 09-Oct-2025 03:38:25 PDT
990
Begin processing the 49th record. run: 1418 subRun: 1 event: 97249 at 09-Oct-2025 03:38:52 PDT
714
365
384
205
239
31
Begin processing the 50th record. run: 1418 subRun: 1 event: 97250 at 09-Oct-2025 03:39:39 PDT
716
956
74
76
38
60
Begin processing the 51st record. run: 1418 subRun: 1 event: 97251 at 09-Oct-2025 03:39:55 PDT
122
797
716
36
Begin processing the 52nd record. run: 1418 subRun: 1 event: 97252 at 09-Oct-2025 03:40:46 PDT
490
55
Begin processing the 53rd record. run: 1418 subRun: 1 event: 97253 at 09-Oct-2025 03:41:15 PDT
189
70
Begin processing the 54th record. run: 1418 subRun: 1 event: 97254 at 09-Oct-2025 03:41:24 PDT
127
98
36
446
Begin processing the 55th record. run: 1418 subRun: 1 event: 97255 at 09-Oct-2025 03:41:37 PDT
47
Begin processing the 56th record. run: 1418 subRun: 1 event: 97256 at 09-Oct-2025 03:41:50 PDT
133
170
Begin processing the 57th record. run: 1418 subRun: 1 event: 97257 at 09-Oct-2025 03:41:55 PDT
25
Begin processing the 58th record. run: 1418 subRun: 1 event: 97258 at 09-Oct-2025 03:42:08 PDT
214
42
347
Begin processing the 59th record. run: 1418 subRun: 1 event: 97259 at 09-Oct-2025 03:42:24 PDT
132
391
170
79
61
134
126
113
147
46
Begin processing the 60th record. run: 1418 subRun: 1 event: 97260 at 09-Oct-2025 03:43:36 PDT
477
30
Begin processing the 61st record. run: 1418 subRun: 1 event: 97261 at 09-Oct-2025 03:44:08 PDT
328
44
Begin processing the 62nd record. run: 1418 subRun: 1 event: 97262 at 09-Oct-2025 03:44:25 PDT
203
360
206
159
Begin processing the 63rd record. run: 1418 subRun: 1 event: 97263 at 09-Oct-2025 03:44:37 PDT
300
205
Begin processing the 64th record. run: 1418 subRun: 1 event: 97264 at 09-Oct-2025 03:44:52 PDT
97
Begin processing the 65th record. run: 1418 subRun: 1 event: 97265 at 09-Oct-2025 03:45:08 PDT
60
83
31
Begin processing the 66th record. run: 1418 subRun: 1 event: 97266 at 09-Oct-2025 03:45:25 PDT
Begin processing the 67th record. run: 1418 subRun: 1 event: 97267 at 09-Oct-2025 03:45:31 PDT
277
Begin processing the 68th record. run: 1418 subRun: 1 event: 97268 at 09-Oct-2025 03:45:46 PDT
61
123
37
Begin processing the 69th record. run: 1418 subRun: 1 event: 97269 at 09-Oct-2025 03:45:51 PDT
169
Begin processing the 70th record. run: 1418 subRun: 1 event: 97270 at 09-Oct-2025 03:46:00 PDT
397
103
Begin processing the 71st record. run: 1418 subRun: 1 event: 97271 at 09-Oct-2025 03:46:34 PDT
89
Begin processing the 72nd record. run: 1418 subRun: 1 event: 97272 at 09-Oct-2025 03:46:44 PDT
36
Begin processing the 73rd record. run: 1418 subRun: 1 event: 97273 at 09-Oct-2025 03:47:00 PDT
151
101
Begin processing the 74th record. run: 1418 subRun: 1 event: 97274 at 09-Oct-2025 03:47:10 PDT
107
107
163
54
26
139
Begin processing the 75th record. run: 1418 subRun: 1 event: 97275 at 09-Oct-2025 03:47:21 PDT
45
Begin processing the 76th record. run: 1418 subRun: 1 event: 97276 at 09-Oct-2025 03:47:24 PDT
528
Begin processing the 77th record. run: 1418 subRun: 1 event: 97277 at 09-Oct-2025 03:47:35 PDT
Begin processing the 78th record. run: 1418 subRun: 1 event: 97278 at 09-Oct-2025 03:47:43 PDT
172
760
216
184
Begin processing the 79th record. run: 1418 subRun: 1 event: 97279 at 09-Oct-2025 03:47:56 PDT
Begin processing the 80th record. run: 1418 subRun: 1 event: 97280 at 09-Oct-2025 03:48:05 PDT
Begin processing the 81st record. run: 1418 subRun: 1 event: 97281 at 09-Oct-2025 03:48:16 PDT
Begin processing the 82nd record. run: 1418 subRun: 1 event: 97282 at 09-Oct-2025 03:48:19 PDT
137
103
26
Begin processing the 83rd record. run: 1418 subRun: 1 event: 97283 at 09-Oct-2025 03:48:31 PDT
28
163
49
Begin processing the 84th record. run: 1418 subRun: 1 event: 97284 at 09-Oct-2025 03:48:40 PDT
53
Begin processing the 85th record. run: 1418 subRun: 1 event: 97285 at 09-Oct-2025 03:48:59 PDT
28
308
85
Begin processing the 86th record. run: 1418 subRun: 1 event: 97286 at 09-Oct-2025 03:49:05 PDT
187
173
Begin processing the 87th record. run: 1418 subRun: 1 event: 97287 at 09-Oct-2025 03:49:17 PDT
514
Begin processing the 88th record. run: 1418 subRun: 1 event: 97288 at 09-Oct-2025 03:49:24 PDT
291
178
189
25
Begin processing the 89th record. run: 1418 subRun: 1 event: 97289 at 09-Oct-2025 03:49:29 PDT
161
2157
Begin processing the 90th record. run: 1418 subRun: 1 event: 97290 at 09-Oct-2025 03:49:38 PDT
390
245
172
83
215
42
25
37
Begin processing the 91st record. run: 1418 subRun: 1 event: 97291 at 09-Oct-2025 03:49:59 PDT
715
509
28
Begin processing the 92nd record. run: 1418 subRun: 1 event: 97292 at 09-Oct-2025 03:50:25 PDT
111
54
Begin processing the 93rd record. run: 1418 subRun: 1 event: 97293 at 09-Oct-2025 03:50:43 PDT
28
105
53
208
35
Begin processing the 94th record. run: 1418 subRun: 1 event: 97294 at 09-Oct-2025 03:50:49 PDT
69
69
142
170
Begin processing the 95th record. run: 1418 subRun: 1 event: 97295 at 09-Oct-2025 03:52:27 PDT
496
106
Begin processing the 96th record. run: 1418 subRun: 1 event: 97296 at 09-Oct-2025 03:52:43 PDT
iter->second->Run() throw STATUS_CODE_FAILURE
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 97th record. run: 1418 subRun: 1 event: 97297 at 09-Oct-2025 03:53:05 PDT
Begin processing the 98th record. run: 1418 subRun: 1 event: 97298 at 09-Oct-2025 03:53:09 PDT
326
Begin processing the 99th record. run: 1418 subRun: 1 event: 97299 at 09-Oct-2025 03:53:13 PDT
52
Begin processing the 100th record. run: 1418 subRun: 1 event: 97300 at 09-Oct-2025 03:53:35 PDT
09-Oct-2025 03:53:44 PDT Closed output file "nue_dune10kt_1x2x6_1418_972_20230827T014216Z_gen_g4_detsim_hitreco__20240221T051923Z_reco2_reco2.root"
09-Oct-2025 03:53:44 PDT Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/8e/62/nue_dune10kt_1x2x6_1418_972_20230827T014216Z_gen_g4_detsim_hitreco__20240221T051923Z_reco2.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 1.46182 3.19176 45.9857 2.34111 4.52684 100
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0654196 0.136121 0.727541 0.114223 0.0812303 100
reco:pandora2:StandardPandora 1.34394 3.05477 45.8748 2.19528 4.52892 100
[art]:TriggerResults:TriggerResultInserter 1.148e-05 2.66446e-05 7.6759e-05 2.2794e-05 1.38347e-05 100
end_path:out1:RootOutput 3.01e-06 5.23286e-06 1.799e-05 4.15e-06 2.51921e-06 100
end_path:out1:RootOutput(write) 0.00015787 0.000577542 0.00329718 0.000312821 0.000593845 100
================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 2211.09 MB
Peak resident set size usage (VmHWM): 1195.13 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 848
-rw-r--r--. 1 cuser cuser 212 Oct 9 03:25 all-input-dids.txt
-rw-r--r--. 1 cuser cuser 0 Oct 9 03:26 debugprod.log
-rw-r--r--. 1 cuser cuser 43437 Oct 9 03:53 jobscript.log
-rw-r--r--. 1 cuser cuser 182 Oct 9 03:53 justin-processed-pfns.txt
drwxr-xr-x. 4 cuser cuser 60 Oct 9 03:26 larpandoracontent
-rw-r--r--. 1 cuser cuser 355030 Oct 9 03:53 nue_dune10kt_1x2x6_1418_972_20230827T014216Z_gen_g4_detsim_hitreco__20240221T051923Z_reco2_reco2.root
-rw-r--r--. 1 cuser cuser 519 Oct 9 03:53 reco2_hist.root
-rw-r--r--. 1 cuser cuser 451103 Oct 9 03:53 trainingFile_nue_dune10kt_1x2x6_1418_972_20230827T014216Z_gen_g4_detsim_hitreco__20240221T051923Z_reco2.root 