justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252944.22@dunegpschedd02.fnal.gov

Jobsub ID252944.22@dunegpschedd02.fnal.gov
Workflow ID10785
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2025-11-26 10:49:27
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-26 14:06:08
From worker nodeHostnamedunegli-7735362-0-fnpc18003.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 10:51:26
Input filesfardet-hd:nu_dune10kt_1x2x6_1108_834_20230826T214320Z_gen_g4_detsim_hitreco__20240229T194408Z_reco2.root
JobscriptExit code0
Real time13m (832s)
CPU time13m (797s = 95%)
Max RSS bytes1466474496 (1398 MiB)
Outputting started2025-11-26 11:05:18
Output files
Finished2025-11-26 14:06:08
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

oft_v10_08_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/57/a9/nu_dune10kt_1x2x6_1108_834_20230826T214320Z_gen_g4_detsim_hitreco__20240229T194408Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 33rd record. run: 1108 subRun: 1 event: 83433 at 26-Nov-2025 10:59:55 UTC
Begin processing the 34th record. run: 1108 subRun: 1 event: 83434 at 26-Nov-2025 10:59:58 UTC
Begin processing the 35th record. run: 1108 subRun: 1 event: 83435 at 26-Nov-2025 11:00:10 UTC
Begin processing the 36th record. run: 1108 subRun: 1 event: 83436 at 26-Nov-2025 11:00:13 UTC
Begin processing the 37th record. run: 1108 subRun: 1 event: 83437 at 26-Nov-2025 11:00:17 UTC
Begin processing the 38th record. run: 1108 subRun: 1 event: 83438 at 26-Nov-2025 11:00:20 UTC
Begin processing the 39th record. run: 1108 subRun: 1 event: 83439 at 26-Nov-2025 11:00:26 UTC
Begin processing the 40th record. run: 1108 subRun: 1 event: 83440 at 26-Nov-2025 11:00:29 UTC
Begin processing the 41st record. run: 1108 subRun: 1 event: 83441 at 26-Nov-2025 11:00:32 UTC
Begin processing the 42nd record. run: 1108 subRun: 1 event: 83442 at 26-Nov-2025 11:00:34 UTC
Begin processing the 43rd record. run: 1108 subRun: 1 event: 83443 at 26-Nov-2025 11:00:37 UTC
Begin processing the 44th record. run: 1108 subRun: 1 event: 83444 at 26-Nov-2025 11:00:39 UTC
Begin processing the 45th record. run: 1108 subRun: 1 event: 83445 at 26-Nov-2025 11:00:42 UTC
Begin processing the 46th record. run: 1108 subRun: 1 event: 83446 at 26-Nov-2025 11:00:58 UTC
Begin processing the 47th record. run: 1108 subRun: 1 event: 83447 at 26-Nov-2025 11:01:01 UTC
Begin processing the 48th record. run: 1108 subRun: 1 event: 83448 at 26-Nov-2025 11:01:04 UTC
Begin processing the 49th record. run: 1108 subRun: 1 event: 83449 at 26-Nov-2025 11:01:08 UTC
Begin processing the 50th record. run: 1108 subRun: 1 event: 83450 at 26-Nov-2025 11:01:13 UTC
Begin processing the 51st record. run: 1108 subRun: 1 event: 83451 at 26-Nov-2025 11:01:16 UTC
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 52nd record. run: 1108 subRun: 1 event: 83452 at 26-Nov-2025 11:01:16 UTC
Begin processing the 53rd record. run: 1108 subRun: 1 event: 83453 at 26-Nov-2025 11:01:20 UTC
Begin processing the 54th record. run: 1108 subRun: 1 event: 83454 at 26-Nov-2025 11:01:27 UTC
Begin processing the 55th record. run: 1108 subRun: 1 event: 83455 at 26-Nov-2025 11:01:30 UTC
Begin processing the 56th record. run: 1108 subRun: 1 event: 83456 at 26-Nov-2025 11:01:34 UTC
Begin processing the 57th record. run: 1108 subRun: 1 event: 83457 at 26-Nov-2025 11:01:37 UTC
Begin processing the 58th record. run: 1108 subRun: 1 event: 83458 at 26-Nov-2025 11:01:47 UTC
Begin processing the 59th record. run: 1108 subRun: 1 event: 83459 at 26-Nov-2025 11:01:50 UTC
Begin processing the 60th record. run: 1108 subRun: 1 event: 83460 at 26-Nov-2025 11:01:54 UTC
Begin processing the 61st record. run: 1108 subRun: 1 event: 83461 at 26-Nov-2025 11:02:00 UTC
Begin processing the 62nd record. run: 1108 subRun: 1 event: 83462 at 26-Nov-2025 11:02:04 UTC
Begin processing the 63rd record. run: 1108 subRun: 1 event: 83463 at 26-Nov-2025 11:02:07 UTC
Begin processing the 64th record. run: 1108 subRun: 1 event: 83464 at 26-Nov-2025 11:02:10 UTC
Begin processing the 65th record. run: 1108 subRun: 1 event: 83465 at 26-Nov-2025 11:02:13 UTC
Begin processing the 66th record. run: 1108 subRun: 1 event: 83466 at 26-Nov-2025 11:02:16 UTC
Begin processing the 67th record. run: 1108 subRun: 1 event: 83467 at 26-Nov-2025 11:02:25 UTC
Begin processing the 68th record. run: 1108 subRun: 1 event: 83468 at 26-Nov-2025 11:02:28 UTC
Begin processing the 69th record. run: 1108 subRun: 1 event: 83469 at 26-Nov-2025 11:02:32 UTC
Begin processing the 70th record. run: 1108 subRun: 1 event: 83470 at 26-Nov-2025 11:02:37 UTC
Begin processing the 71st record. run: 1108 subRun: 1 event: 83471 at 26-Nov-2025 11:02:41 UTC
Begin processing the 72nd record. run: 1108 subRun: 1 event: 83472 at 26-Nov-2025 11:02:48 UTC
Begin processing the 73rd record. run: 1108 subRun: 1 event: 83473 at 26-Nov-2025 11:02:53 UTC
Begin processing the 74th record. run: 1108 subRun: 1 event: 83474 at 26-Nov-2025 11:02:57 UTC
Begin processing the 75th record. run: 1108 subRun: 1 event: 83475 at 26-Nov-2025 11:03:04 UTC
Begin processing the 76th record. run: 1108 subRun: 1 event: 83476 at 26-Nov-2025 11:03:21 UTC
Begin processing the 77th record. run: 1108 subRun: 1 event: 83477 at 26-Nov-2025 11:03:25 UTC
Begin processing the 78th record. run: 1108 subRun: 1 event: 83478 at 26-Nov-2025 11:03:29 UTC
Begin processing the 79th record. run: 1108 subRun: 1 event: 83479 at 26-Nov-2025 11:03:33 UTC
Begin processing the 80th record. run: 1108 subRun: 1 event: 83480 at 26-Nov-2025 11:03:37 UTC
Begin processing the 81st record. run: 1108 subRun: 1 event: 83481 at 26-Nov-2025 11:03:40 UTC
Begin processing the 82nd record. run: 1108 subRun: 1 event: 83482 at 26-Nov-2025 11:03:46 UTC
Begin processing the 83rd record. run: 1108 subRun: 1 event: 83483 at 26-Nov-2025 11:03:51 UTC
Begin processing the 84th record. run: 1108 subRun: 1 event: 83484 at 26-Nov-2025 11:03:55 UTC
Begin processing the 85th record. run: 1108 subRun: 1 event: 83485 at 26-Nov-2025 11:03:58 UTC
Begin processing the 86th record. run: 1108 subRun: 1 event: 83486 at 26-Nov-2025 11:04:19 UTC
Begin processing the 87th record. run: 1108 subRun: 1 event: 83487 at 26-Nov-2025 11:04:23 UTC
Begin processing the 88th record. run: 1108 subRun: 1 event: 83488 at 26-Nov-2025 11:04:29 UTC
Begin processing the 89th record. run: 1108 subRun: 1 event: 83489 at 26-Nov-2025 11:04:33 UTC
Begin processing the 90th record. run: 1108 subRun: 1 event: 83490 at 26-Nov-2025 11:04:37 UTC
Begin processing the 91st record. run: 1108 subRun: 1 event: 83491 at 26-Nov-2025 11:04:42 UTC
Begin processing the 92nd record. run: 1108 subRun: 1 event: 83492 at 26-Nov-2025 11:04:46 UTC
Begin processing the 93rd record. run: 1108 subRun: 1 event: 83493 at 26-Nov-2025 11:04:49 UTC
Begin processing the 94th record. run: 1108 subRun: 1 event: 83494 at 26-Nov-2025 11:04:52 UTC
Begin processing the 95th record. run: 1108 subRun: 1 event: 83495 at 26-Nov-2025 11:04:56 UTC
Begin processing the 96th record. run: 1108 subRun: 1 event: 83496 at 26-Nov-2025 11:04:59 UTC
Begin processing the 97th record. run: 1108 subRun: 1 event: 83497 at 26-Nov-2025 11:05:02 UTC
Begin processing the 98th record. run: 1108 subRun: 1 event: 83498 at 26-Nov-2025 11:05:06 UTC
Begin processing the 99th record. run: 1108 subRun: 1 event: 83499 at 26-Nov-2025 11:05:09 UTC
Begin processing the 100th record. run: 1108 subRun: 1 event: 83500 at 26-Nov-2025 11:05:13 UTC
26-Nov-2025 11:05:17 UTC  Closed output file "nu_dune10kt_1x2x6_1108_834_20230826T214320Z_gen_g4_detsim_hitreco__20240229T194408Z_reco2_reco2.root"
26-Nov-2025 11:05:17 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/57/a9/nu_dune10kt_1x2x6_1108_834_20230826T214320Z_gen_g4_detsim_hitreco__20240229T194408Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0174133      7.45831       298.957       3.43145       29.5058        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00149311     0.0116995     0.0370897     0.0126831    0.00789381       100    
reco:pandora:StandardPandora                   0.0112075      7.43188       298.855       3.40645       29.4968        100    
[art]:TriggerResults:TriggerResultInserter    1.0029e-05    4.50164e-05   0.000869105    1.839e-05    0.000114499      100    
end_path:out1:RootOutput                       2.184e-06    3.30138e-06   2.3154e-05     2.805e-06    2.55527e-06      100    
end_path:out1:RootOutput(write)               0.000815894    0.0144946     0.087767      0.0100923     0.0154225       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2449.25 MB
  Peak resident set size usage (VmHWM): 1466.47 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2025-12-20 00:49:35 UTC       justIN version: 01.05.03