justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268861.1@dunegpschedd02.fnal.gov

Jobsub ID268861.1@dunegpschedd02.fnal.gov
Workflow ID11624
Stage ID1
User nameimawby@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-01-11 13:04:48
SiteUS_BNL
EntryDUNE_US_BNL_spoolce01
Last heartbeat2026-01-11 13:16:49
From worker nodeHostnamespool0627.sdcc.bnl.gov
cpuinfoAMD EPYC 7351 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-11 13:06:01
Input filesfardet-hd:nu_dune10kt_1x2x6_1412_645_20230826T153330Z_gen_g4_detsim_hitreco__20240223T005443Z_reco2.root
JobscriptExit code0
Real time10m (626s)
CPU time6m (369s = 58%)
Max RSS bytes1039101952 (990 MiB)
Outputting started2026-01-11 13:16:28
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/metrics_nu_0/fnal/11624/1/001/Validation_nu_dune10kt_1x2x6_1412_645_20230826T153330Z_gen_g4_detsim_hitreco__20240223T005443Z_reco2.root
Finished2026-01-11 13:16:49
Saved logsjustin-logs:268861.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

ft_v10_13_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e1/27/nu_dune10kt_1x2x6_1412_645_20230826T153330Z_gen_g4_detsim_hitreco__20240223T005443Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 33rd record. run: 1412 subRun: 1 event: 64533 at 11-Jan-2026 08:12:08 EST
Begin processing the 34th record. run: 1412 subRun: 1 event: 64534 at 11-Jan-2026 08:12:11 EST
Begin processing the 35th record. run: 1412 subRun: 1 event: 64535 at 11-Jan-2026 08:12:18 EST
Begin processing the 36th record. run: 1412 subRun: 1 event: 64536 at 11-Jan-2026 08:12:22 EST
Begin processing the 37th record. run: 1412 subRun: 1 event: 64537 at 11-Jan-2026 08:12:24 EST
Begin processing the 38th record. run: 1412 subRun: 1 event: 64538 at 11-Jan-2026 08:12:28 EST
Begin processing the 39th record. run: 1412 subRun: 1 event: 64539 at 11-Jan-2026 08:12:32 EST
Begin processing the 40th record. run: 1412 subRun: 1 event: 64540 at 11-Jan-2026 08:12:35 EST
Begin processing the 41st record. run: 1412 subRun: 1 event: 64541 at 11-Jan-2026 08:12:38 EST
Begin processing the 42nd record. run: 1412 subRun: 1 event: 64542 at 11-Jan-2026 08:12:42 EST
Begin processing the 43rd record. run: 1412 subRun: 1 event: 64543 at 11-Jan-2026 08:12:45 EST
Begin processing the 44th record. run: 1412 subRun: 1 event: 64544 at 11-Jan-2026 08:12:49 EST
Begin processing the 45th record. run: 1412 subRun: 1 event: 64545 at 11-Jan-2026 08:12:52 EST
Begin processing the 46th record. run: 1412 subRun: 1 event: 64546 at 11-Jan-2026 08:12:56 EST
Begin processing the 47th record. run: 1412 subRun: 1 event: 64547 at 11-Jan-2026 08:12:59 EST
Begin processing the 48th record. run: 1412 subRun: 1 event: 64548 at 11-Jan-2026 08:13:03 EST
Begin processing the 49th record. run: 1412 subRun: 1 event: 64549 at 11-Jan-2026 08:13:08 EST
Begin processing the 50th record. run: 1412 subRun: 1 event: 64550 at 11-Jan-2026 08:13:14 EST
Begin processing the 51st record. run: 1412 subRun: 1 event: 64551 at 11-Jan-2026 08:13:18 EST
Begin processing the 52nd record. run: 1412 subRun: 1 event: 64552 at 11-Jan-2026 08:13:22 EST
Begin processing the 53rd record. run: 1412 subRun: 1 event: 64553 at 11-Jan-2026 08:13:25 EST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 54th record. run: 1412 subRun: 1 event: 64554 at 11-Jan-2026 08:13:25 EST
Begin processing the 55th record. run: 1412 subRun: 1 event: 64555 at 11-Jan-2026 08:13:28 EST
Begin processing the 56th record. run: 1412 subRun: 1 event: 64556 at 11-Jan-2026 08:13:31 EST
Begin processing the 57th record. run: 1412 subRun: 1 event: 64557 at 11-Jan-2026 08:13:34 EST
Begin processing the 58th record. run: 1412 subRun: 1 event: 64558 at 11-Jan-2026 08:13:37 EST
Begin processing the 59th record. run: 1412 subRun: 1 event: 64559 at 11-Jan-2026 08:13:40 EST
Begin processing the 60th record. run: 1412 subRun: 1 event: 64560 at 11-Jan-2026 08:13:44 EST
Begin processing the 61st record. run: 1412 subRun: 1 event: 64561 at 11-Jan-2026 08:13:47 EST
Begin processing the 62nd record. run: 1412 subRun: 1 event: 64562 at 11-Jan-2026 08:13:51 EST
Begin processing the 63rd record. run: 1412 subRun: 1 event: 64563 at 11-Jan-2026 08:13:55 EST
Begin processing the 64th record. run: 1412 subRun: 1 event: 64564 at 11-Jan-2026 08:13:58 EST
Begin processing the 65th record. run: 1412 subRun: 1 event: 64565 at 11-Jan-2026 08:14:01 EST
Begin processing the 66th record. run: 1412 subRun: 1 event: 64566 at 11-Jan-2026 08:14:05 EST
Begin processing the 67th record. run: 1412 subRun: 1 event: 64567 at 11-Jan-2026 08:14:09 EST
Begin processing the 68th record. run: 1412 subRun: 1 event: 64568 at 11-Jan-2026 08:14:12 EST
Begin processing the 69th record. run: 1412 subRun: 1 event: 64569 at 11-Jan-2026 08:14:15 EST
Begin processing the 70th record. run: 1412 subRun: 1 event: 64570 at 11-Jan-2026 08:14:19 EST
Begin processing the 71st record. run: 1412 subRun: 1 event: 64571 at 11-Jan-2026 08:14:22 EST
Begin processing the 72nd record. run: 1412 subRun: 1 event: 64572 at 11-Jan-2026 08:14:26 EST
Begin processing the 73rd record. run: 1412 subRun: 1 event: 64573 at 11-Jan-2026 08:14:29 EST
Begin processing the 74th record. run: 1412 subRun: 1 event: 64574 at 11-Jan-2026 08:14:34 EST
Begin processing the 75th record. run: 1412 subRun: 1 event: 64575 at 11-Jan-2026 08:14:37 EST
Begin processing the 76th record. run: 1412 subRun: 1 event: 64576 at 11-Jan-2026 08:14:40 EST
Begin processing the 77th record. run: 1412 subRun: 1 event: 64577 at 11-Jan-2026 08:14:44 EST
Begin processing the 78th record. run: 1412 subRun: 1 event: 64578 at 11-Jan-2026 08:14:48 EST
Begin processing the 79th record. run: 1412 subRun: 1 event: 64579 at 11-Jan-2026 08:14:51 EST
Begin processing the 80th record. run: 1412 subRun: 1 event: 64580 at 11-Jan-2026 08:14:54 EST
Begin processing the 81st record. run: 1412 subRun: 1 event: 64581 at 11-Jan-2026 08:14:57 EST
Begin processing the 82nd record. run: 1412 subRun: 1 event: 64582 at 11-Jan-2026 08:15:00 EST
Begin processing the 83rd record. run: 1412 subRun: 1 event: 64583 at 11-Jan-2026 08:15:03 EST
Begin processing the 84th record. run: 1412 subRun: 1 event: 64584 at 11-Jan-2026 08:15:06 EST
Begin processing the 85th record. run: 1412 subRun: 1 event: 64585 at 11-Jan-2026 08:15:10 EST
Begin processing the 86th record. run: 1412 subRun: 1 event: 64586 at 11-Jan-2026 08:15:13 EST
Begin processing the 87th record. run: 1412 subRun: 1 event: 64587 at 11-Jan-2026 08:15:24 EST
Begin processing the 88th record. run: 1412 subRun: 1 event: 64588 at 11-Jan-2026 08:15:40 EST
Begin processing the 89th record. run: 1412 subRun: 1 event: 64589 at 11-Jan-2026 08:15:43 EST
Begin processing the 90th record. run: 1412 subRun: 1 event: 64590 at 11-Jan-2026 08:15:47 EST
Begin processing the 91st record. run: 1412 subRun: 1 event: 64591 at 11-Jan-2026 08:15:51 EST
Begin processing the 92nd record. run: 1412 subRun: 1 event: 64592 at 11-Jan-2026 08:15:55 EST
Begin processing the 93rd record. run: 1412 subRun: 1 event: 64593 at 11-Jan-2026 08:15:59 EST
Begin processing the 94th record. run: 1412 subRun: 1 event: 64594 at 11-Jan-2026 08:16:02 EST
Begin processing the 95th record. run: 1412 subRun: 1 event: 64595 at 11-Jan-2026 08:16:06 EST
Begin processing the 96th record. run: 1412 subRun: 1 event: 64596 at 11-Jan-2026 08:16:09 EST
Begin processing the 97th record. run: 1412 subRun: 1 event: 64597 at 11-Jan-2026 08:16:12 EST
Begin processing the 98th record. run: 1412 subRun: 1 event: 64598 at 11-Jan-2026 08:16:15 EST
Begin processing the 99th record. run: 1412 subRun: 1 event: 64599 at 11-Jan-2026 08:16:19 EST
Begin processing the 100th record. run: 1412 subRun: 1 event: 64600 at 11-Jan-2026 08:16:22 EST
11-Jan-2026 08:16:26 EST  Closed output file "nu_dune10kt_1x2x6_1412_645_20230826T153330Z_gen_g4_detsim_hitreco__20240223T005443Z_reco2_reco2.root"
11-Jan-2026 08:16:26 EST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e1/27/nu_dune10kt_1x2x6_1412_645_20230826T153330Z_gen_g4_detsim_hitreco__20240223T005443Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.215242       3.36113       19.6978       2.9154        2.25693        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0389145     0.0731601      0.12561      0.0718392     0.0143061       100    
reco:pandora:StandardPandora                   0.154429       3.27144       19.6445       2.82324       2.25405        100    
[art]:TriggerResults:TriggerResultInserter     1.449e-05    3.64934e-05   0.000109481   3.22405e-05   1.76314e-05      100    
end_path:out1:RootOutput                       3.86e-06     6.30524e-06    3.586e-05     4.92e-06     3.82832e-06      100    
end_path:out1:RootOutput(write)                0.0013382     0.0162106     0.060345      0.0115611     0.0133546       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2081.59 MB
  Peak resident set size usage (VmHWM): 1039.1 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-04 07:20:32 UTC       justIN version: 01.06.00