justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 226591.45@dunegpschedd02.fnal.gov

Jobsub ID226591.45@dunegpschedd02.fnal.gov
Workflow ID8772
Stage ID1
User namelwhite86@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-10-09 10:24:38
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw7
Last heartbeat2025-10-09 10:55:06
From worker nodeHostnamemh-7763-16.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-09 10:27:47
Input filesfardet-hd:nue_dune10kt_1x2x6_1423_547_20230827T134204Z_gen_g4_detsim_hitreco__20240221T023504Z_reco2.root
JobscriptExit code0
Real time26m (1610s)
CPU time5m (336s = 20%)
Max RSS bytes1220431872 (1163 MiB)
Outputting started2025-10-09 10:54:37
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/08772/1/001/trainingFile_nue_dune10kt_1x2x6_1423_547_20230827T134204Z_gen_g4_detsim_hitreco__20240221T023504Z_reco2.root
Finished2025-10-09 10:55:06
Saved logsjustin-logs:226591.45-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

the 51st record. run: 1423 subRun: 1 event: 54751 at 09-Oct-2025 03:42:49 PDT
Begin processing the 52nd record. run: 1423 subRun: 1 event: 54752 at 09-Oct-2025 03:42:55 PDT
72
26
Begin processing the 53rd record. run: 1423 subRun: 1 event: 54753 at 09-Oct-2025 03:43:03 PDT
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 54th record. run: 1423 subRun: 1 event: 54754 at 09-Oct-2025 03:43:10 PDT
Begin processing the 55th record. run: 1423 subRun: 1 event: 54755 at 09-Oct-2025 03:43:19 PDT
350
56
87
55
Begin processing the 56th record. run: 1423 subRun: 1 event: 54756 at 09-Oct-2025 03:43:28 PDT
Begin processing the 57th record. run: 1423 subRun: 1 event: 54757 at 09-Oct-2025 03:43:43 PDT
153
296
Begin processing the 58th record. run: 1423 subRun: 1 event: 54758 at 09-Oct-2025 03:43:48 PDT
342
Begin processing the 59th record. run: 1423 subRun: 1 event: 54759 at 09-Oct-2025 03:44:04 PDT
157
42
Begin processing the 60th record. run: 1423 subRun: 1 event: 54760 at 09-Oct-2025 03:44:20 PDT
533
63
203
Begin processing the 61st record. run: 1423 subRun: 1 event: 54761 at 09-Oct-2025 03:44:46 PDT
73
61
Begin processing the 62nd record. run: 1423 subRun: 1 event: 54762 at 09-Oct-2025 03:44:54 PDT
Begin processing the 63rd record. run: 1423 subRun: 1 event: 54763 at 09-Oct-2025 03:45:05 PDT
69
Begin processing the 64th record. run: 1423 subRun: 1 event: 54764 at 09-Oct-2025 03:45:07 PDT
421
116
Begin processing the 65th record. run: 1423 subRun: 1 event: 54765 at 09-Oct-2025 03:45:14 PDT
128
511
62
Begin processing the 66th record. run: 1423 subRun: 1 event: 54766 at 09-Oct-2025 03:45:39 PDT
Begin processing the 67th record. run: 1423 subRun: 1 event: 54767 at 09-Oct-2025 03:45:55 PDT
204
83
234
Begin processing the 68th record. run: 1423 subRun: 1 event: 54768 at 09-Oct-2025 03:46:14 PDT
157
312
29
30
Begin processing the 69th record. run: 1423 subRun: 1 event: 54769 at 09-Oct-2025 03:46:33 PDT
151
42
Begin processing the 70th record. run: 1423 subRun: 1 event: 54770 at 09-Oct-2025 03:46:39 PDT
140
Begin processing the 71st record. run: 1423 subRun: 1 event: 54771 at 09-Oct-2025 03:46:44 PDT
148
Begin processing the 72nd record. run: 1423 subRun: 1 event: 54772 at 09-Oct-2025 03:46:57 PDT
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetAllTrackFeatures
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 155
Begin processing the 73rd record. run: 1423 subRun: 1 event: 54773 at 09-Oct-2025 03:46:59 PDT
28
680
260
51
26
39
Begin processing the 74th record. run: 1423 subRun: 1 event: 54774 at 09-Oct-2025 03:47:14 PDT
81
56
Begin processing the 75th record. run: 1423 subRun: 1 event: 54775 at 09-Oct-2025 03:47:22 PDT
Begin processing the 76th record. run: 1423 subRun: 1 event: 54776 at 09-Oct-2025 03:47:25 PDT
431
511
172
57
55
Begin processing the 77th record. run: 1423 subRun: 1 event: 54777 at 09-Oct-2025 03:48:37 PDT
1064
987
50
75
Begin processing the 78th record. run: 1423 subRun: 1 event: 54778 at 09-Oct-2025 03:49:42 PDT
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 79th record. run: 1423 subRun: 1 event: 54779 at 09-Oct-2025 03:49:56 PDT
285
Begin processing the 80th record. run: 1423 subRun: 1 event: 54780 at 09-Oct-2025 03:50:11 PDT
29
Begin processing the 81st record. run: 1423 subRun: 1 event: 54781 at 09-Oct-2025 03:50:18 PDT
131
1151
Begin processing the 82nd record. run: 1423 subRun: 1 event: 54782 at 09-Oct-2025 03:50:34 PDT
153
30
Begin processing the 83rd record. run: 1423 subRun: 1 event: 54783 at 09-Oct-2025 03:50:47 PDT
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 84th record. run: 1423 subRun: 1 event: 54784 at 09-Oct-2025 03:50:53 PDT
101
Begin processing the 85th record. run: 1423 subRun: 1 event: 54785 at 09-Oct-2025 03:50:59 PDT
36
135
Begin processing the 86th record. run: 1423 subRun: 1 event: 54786 at 09-Oct-2025 03:51:15 PDT
33
Begin processing the 87th record. run: 1423 subRun: 1 event: 54787 at 09-Oct-2025 03:51:30 PDT
27
Begin processing the 88th record. run: 1423 subRun: 1 event: 54788 at 09-Oct-2025 03:51:40 PDT
633
58
Begin processing the 89th record. run: 1423 subRun: 1 event: 54789 at 09-Oct-2025 03:51:47 PDT
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 90th record. run: 1423 subRun: 1 event: 54790 at 09-Oct-2025 03:51:52 PDT
Begin processing the 91st record. run: 1423 subRun: 1 event: 54791 at 09-Oct-2025 03:51:57 PDT
53
60
269
164
Begin processing the 92nd record. run: 1423 subRun: 1 event: 54792 at 09-Oct-2025 03:52:10 PDT
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetAllTrackFeatures
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 155
Begin processing the 93rd record. run: 1423 subRun: 1 event: 54793 at 09-Oct-2025 03:52:20 PDT
79
50
Begin processing the 94th record. run: 1423 subRun: 1 event: 54794 at 09-Oct-2025 03:52:46 PDT
47
26
Begin processing the 95th record. run: 1423 subRun: 1 event: 54795 at 09-Oct-2025 03:53:22 PDT
Begin processing the 96th record. run: 1423 subRun: 1 event: 54796 at 09-Oct-2025 03:53:29 PDT
Begin processing the 97th record. run: 1423 subRun: 1 event: 54797 at 09-Oct-2025 03:53:54 PDT
39
32
Begin processing the 98th record. run: 1423 subRun: 1 event: 54798 at 09-Oct-2025 03:54:09 PDT
71
Begin processing the 99th record. run: 1423 subRun: 1 event: 54799 at 09-Oct-2025 03:54:15 PDT
210
Begin processing the 100th record. run: 1423 subRun: 1 event: 54800 at 09-Oct-2025 03:54:32 PDT
117
09-Oct-2025 03:54:36 PDT  Closed output file "nue_dune10kt_1x2x6_1423_547_20230827T134204Z_gen_g4_detsim_hitreco__20240221T023504Z_reco2_reco2.root"
09-Oct-2025 03:54:37 PDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/0e/7e/nue_dune10kt_1x2x6_1423_547_20230827T134204Z_gen_g4_detsim_hitreco__20240221T023504Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.422515       3.66968       97.9554       2.13028       9.8608         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0677243     0.143592      0.855831      0.117299      0.109363        100    
reco:pandora2:StandardPandora                  0.295085       3.52533       97.8478       1.97164       9.86561        100    
[art]:TriggerResults:TriggerResultInserter     9.92e-06     2.01155e-05   0.000110749   1.51745e-05   1.38346e-05      100    
end_path:out1:RootOutput                       2.96e-06     4.24345e-06    2.515e-05     3.43e-06     2.92125e-06      100    
end_path:out1:RootOutput(write)               0.000149788   0.000545693   0.00378894    0.000315531   0.000521956      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2232.3 MB
  Peak resident set size usage (VmHWM): 1220.43 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 788
-rw-r--r--. 1 cuser cuser    212 Oct  9 03:27 all-input-dids.txt
-rw-r--r--. 1 cuser cuser      0 Oct  9 03:28 debugprod.log
-rw-r--r--. 1 cuser cuser  48244 Oct  9 03:54 jobscript.log
-rw-r--r--. 1 cuser cuser    182 Oct  9 03:54 justin-processed-pfns.txt
drwxr-xr-x. 4 cuser cuser     60 Oct  9 03:27 larpandoracontent
-rw-r--r--. 1 cuser cuser 358991 Oct  9 03:54 nue_dune10kt_1x2x6_1423_547_20230827T134204Z_gen_g4_detsim_hitreco__20240221T023504Z_reco2_reco2.root
-rw-r--r--. 1 cuser cuser    519 Oct  9 03:54 reco2_hist.root
-rw-r--r--. 1 cuser cuser 382845 Oct  9 03:54 trainingFile_nue_dune10kt_1x2x6_1423_547_20230827T134204Z_gen_g4_detsim_hitreco__20240221T023504Z_reco2.root
justIN time: 2025-11-05 12:16:55 UTC       justIN version: 01.05.01