justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 13540.0@dunegpschedd02.fnal.gov

Jobsub ID13540.0@dunegpschedd02.fnal.gov
Workflow ID168
Stage ID1
User namelwhite86@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-08-01 01:11:26
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_condce_mcore
Last heartbeat2025-08-01 01:20:46
From worker nodeHostnamemwt2-c095.campuscluster.illinois.edu
cpuinfoAMD EPYC 7443 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-01 01:12:47
Input filesfardet-hd:anu_dune10kt_1x2x6_1087_82_20230825T025411Z_gen_g4_detsim_hitreco__20240221T051513Z_reco2.root
JobscriptExit code0
Real time7m (459s)
CPU time6m (414s = 90%)
Max RSS bytes1337499648 (1275 MiB)
Outputting started2025-08-01 01:20:26
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/00168/1/001/trainingFile_anu_dune10kt_1x2x6_1087_82_20230825T025411Z_gen_g4_detsim_hitreco__20240221T051513Z_reco2.root
Finished2025-08-01 01:20:46
Saved logsjustin-logs:13540.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sing the 43rd record. run: 1087 subRun: 1 event: 8243 at 31-Jul-2025 20:16:44 CDT
Begin processing the 44th record. run: 1087 subRun: 1 event: 8244 at 31-Jul-2025 20:16:46 CDT
Begin processing the 45th record. run: 1087 subRun: 1 event: 8245 at 31-Jul-2025 20:16:48 CDT
Begin processing the 46th record. run: 1087 subRun: 1 event: 8246 at 31-Jul-2025 20:16:50 CDT
Begin processing the 47th record. run: 1087 subRun: 1 event: 8247 at 31-Jul-2025 20:16:52 CDT
Begin processing the 48th record. run: 1087 subRun: 1 event: 8248 at 31-Jul-2025 20:17:06 CDT
Begin processing the 49th record. run: 1087 subRun: 1 event: 8249 at 31-Jul-2025 20:17:10 CDT
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_NOT_INITIALIZED
Begin processing the 50th record. run: 1087 subRun: 1 event: 8250 at 31-Jul-2025 20:17:12 CDT
Begin processing the 51st record. run: 1087 subRun: 1 event: 8251 at 31-Jul-2025 20:17:14 CDT
Begin processing the 52nd record. run: 1087 subRun: 1 event: 8252 at 31-Jul-2025 20:17:21 CDT
Begin processing the 53rd record. run: 1087 subRun: 1 event: 8253 at 31-Jul-2025 20:17:31 CDT
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_NOT_INITIALIZED
Begin processing the 54th record. run: 1087 subRun: 1 event: 8254 at 31-Jul-2025 20:17:33 CDT
Begin processing the 55th record. run: 1087 subRun: 1 event: 8255 at 31-Jul-2025 20:17:35 CDT
Begin processing the 56th record. run: 1087 subRun: 1 event: 8256 at 31-Jul-2025 20:17:37 CDT
Begin processing the 57th record. run: 1087 subRun: 1 event: 8257 at 31-Jul-2025 20:17:41 CDT
Begin processing the 58th record. run: 1087 subRun: 1 event: 8258 at 31-Jul-2025 20:17:43 CDT
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_NOT_INITIALIZED
Begin processing the 59th record. run: 1087 subRun: 1 event: 8259 at 31-Jul-2025 20:17:45 CDT
Begin processing the 60th record. run: 1087 subRun: 1 event: 8260 at 31-Jul-2025 20:17:46 CDT
Begin processing the 61st record. run: 1087 subRun: 1 event: 8261 at 31-Jul-2025 20:17:48 CDT
Begin processing the 62nd record. run: 1087 subRun: 1 event: 8262 at 31-Jul-2025 20:17:50 CDT
Begin processing the 63rd record. run: 1087 subRun: 1 event: 8263 at 31-Jul-2025 20:17:52 CDT
Begin processing the 64th record. run: 1087 subRun: 1 event: 8264 at 31-Jul-2025 20:17:53 CDT
Begin processing the 65th record. run: 1087 subRun: 1 event: 8265 at 31-Jul-2025 20:17:55 CDT
Begin processing the 66th record. run: 1087 subRun: 1 event: 8266 at 31-Jul-2025 20:17:57 CDT
Begin processing the 67th record. run: 1087 subRun: 1 event: 8267 at 31-Jul-2025 20:17:58 CDT
Begin processing the 68th record. run: 1087 subRun: 1 event: 8268 at 31-Jul-2025 20:18:00 CDT
Begin processing the 69th record. run: 1087 subRun: 1 event: 8269 at 31-Jul-2025 20:18:07 CDT
Begin processing the 70th record. run: 1087 subRun: 1 event: 8270 at 31-Jul-2025 20:18:09 CDT
Begin processing the 71st record. run: 1087 subRun: 1 event: 8271 at 31-Jul-2025 20:18:52 CDT
Begin processing the 72nd record. run: 1087 subRun: 1 event: 8272 at 31-Jul-2025 20:18:54 CDT
Begin processing the 73rd record. run: 1087 subRun: 1 event: 8273 at 31-Jul-2025 20:18:56 CDT
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
Begin processing the 74th record. run: 1087 subRun: 1 event: 8274 at 31-Jul-2025 20:18:57 CDT
Begin processing the 75th record. run: 1087 subRun: 1 event: 8275 at 31-Jul-2025 20:19:00 CDT
Begin processing the 76th record. run: 1087 subRun: 1 event: 8276 at 31-Jul-2025 20:19:03 CDT
Begin processing the 77th record. run: 1087 subRun: 1 event: 8277 at 31-Jul-2025 20:19:06 CDT
Begin processing the 78th record. run: 1087 subRun: 1 event: 8278 at 31-Jul-2025 20:19:07 CDT
Begin processing the 79th record. run: 1087 subRun: 1 event: 8279 at 31-Jul-2025 20:19:10 CDT
Begin processing the 80th record. run: 1087 subRun: 1 event: 8280 at 31-Jul-2025 20:19:12 CDT
Begin processing the 81st record. run: 1087 subRun: 1 event: 8281 at 31-Jul-2025 20:19:13 CDT
Begin processing the 82nd record. run: 1087 subRun: 1 event: 8282 at 31-Jul-2025 20:19:29 CDT
Begin processing the 83rd record. run: 1087 subRun: 1 event: 8283 at 31-Jul-2025 20:19:37 CDT
Begin processing the 84th record. run: 1087 subRun: 1 event: 8284 at 31-Jul-2025 20:19:38 CDT
Begin processing the 85th record. run: 1087 subRun: 1 event: 8285 at 31-Jul-2025 20:19:40 CDT
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
Begin processing the 86th record. run: 1087 subRun: 1 event: 8286 at 31-Jul-2025 20:19:42 CDT
Begin processing the 87th record. run: 1087 subRun: 1 event: 8287 at 31-Jul-2025 20:19:44 CDT
Begin processing the 88th record. run: 1087 subRun: 1 event: 8288 at 31-Jul-2025 20:19:51 CDT
Begin processing the 89th record. run: 1087 subRun: 1 event: 8289 at 31-Jul-2025 20:19:52 CDT
Begin processing the 90th record. run: 1087 subRun: 1 event: 8290 at 31-Jul-2025 20:19:54 CDT
Begin processing the 91st record. run: 1087 subRun: 1 event: 8291 at 31-Jul-2025 20:19:56 CDT
Begin processing the 92nd record. run: 1087 subRun: 1 event: 8292 at 31-Jul-2025 20:19:57 CDT
Begin processing the 93rd record. run: 1087 subRun: 1 event: 8293 at 31-Jul-2025 20:20:00 CDT
Begin processing the 94th record. run: 1087 subRun: 1 event: 8294 at 31-Jul-2025 20:20:04 CDT
Begin processing the 95th record. run: 1087 subRun: 1 event: 8295 at 31-Jul-2025 20:20:06 CDT
Begin processing the 96th record. run: 1087 subRun: 1 event: 8296 at 31-Jul-2025 20:20:08 CDT
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
Begin processing the 97th record. run: 1087 subRun: 1 event: 8297 at 31-Jul-2025 20:20:09 CDT
Begin processing the 98th record. run: 1087 subRun: 1 event: 8298 at 31-Jul-2025 20:20:11 CDT
Begin processing the 99th record. run: 1087 subRun: 1 event: 8299 at 31-Jul-2025 20:20:19 CDT
Begin processing the 100th record. run: 1087 subRun: 1 event: 8300 at 31-Jul-2025 20:20:22 CDT
31-Jul-2025 20:20:25 CDT  Closed output file "anu_dune10kt_1x2x6_1087_82_20230825T025411Z_gen_g4_detsim_hitreco__20240221T051513Z_reco2_reco2.root"
31-Jul-2025 20:20:25 CDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e5/a0/anu_dune10kt_1x2x6_1087_82_20230825T025411Z_gen_g4_detsim_hitreco__20240221T051513Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.19386       3.70131       42.703        1.68955       6.60018        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.010349      0.0438357     0.744026      0.0278965     0.079433        100    
reco:pandora2:StandardPandora                   1.17278       3.6565        42.6822       1.6454        6.60165        100    
[art]:TriggerResults:TriggerResultInserter    1.4958e-05    3.06103e-05   9.4208e-05    2.77225e-05   1.50927e-05      100    
end_path:out1:RootOutput                       2.976e-06    5.86722e-06   2.8624e-05     4.158e-06    4.44758e-06      100    
end_path:out1:RootOutput(write)               0.000197602   0.000674463   0.00361641    0.000380927   0.000680481      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2341.53 MB
  Peak resident set size usage (VmHWM): 1337.5 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 2088
-rw-r--r-- 1 dune osgvo     210 Jul 31 20:12 all-input-dids.txt
-rw-r--r-- 1 dune osgvo  350674 Jul 31 20:20 anu_dune10kt_1x2x6_1087_82_20230825T025411Z_gen_g4_detsim_hitreco__20240221T051513Z_reco2_reco2.root
-rw-r--r-- 1 dune osgvo       0 Jul 31 20:13 debugprod.log
-rw-r--r-- 1 dune osgvo   44149 Jul 31 20:20 jobscript.log
-rw-r--r-- 1 dune osgvo     181 Jul 31 20:20 justin-processed-pfns.txt
drwxr-xr-x 4 dune osgvo    4096 Jul 31 20:12 larpandoracontent
-rw-r--r-- 1 dune osgvo     519 Jul 31 20:20 reco2_hist.root
-rw-r--r-- 1 dune osgvo 1714023 Jul 31 20:20 trainingFile_anu_dune10kt_1x2x6_1087_82_20230825T025411Z_gen_g4_detsim_hitreco__20240221T051513Z_reco2.root
justIN time: 2025-08-04 17:38:03 UTC       justIN version: 01.04.00