justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 259937.1@dunegpschedd01.fnal.gov

Jobsub ID259937.1@dunegpschedd01.fnal.gov
Workflow ID10833
Stage ID1
User namelwhite86@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-11-27 19:14:52
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce02
Last heartbeat2025-11-27 19:27:19
From worker nodeHostnamedune002-509960.0-lcg2675.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-27 19:18:53
Input filesfardet-hd:nu_dune10kt_1x2x6_1408_633_20230826T093808Z_gen_g4_detsim_hitreco__20240110T200350Z_reco2.root
JobscriptExit code0
Real time7m (478s)
CPU time7m (456s = 95%)
Max RSS bytes1314676736 (1253 MiB)
Outputting started2025-11-27 19:26:52
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/10833/1/001/trainingFile_nu_dune10kt_1x2x6_1408_633_20230826T093808Z_gen_g4_detsim_hitreco__20240110T200350Z_reco2.root
Finished2025-11-27 19:27:19
Saved logsjustin-logs:259937.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

run: 1408 subRun: 1 event: 63332 at 27-Nov-2025 19:21:07 UTC
Begin processing the 33rd record. run: 1408 subRun: 1 event: 63333 at 27-Nov-2025 19:21:09 UTC
Begin processing the 34th record. run: 1408 subRun: 1 event: 63334 at 27-Nov-2025 19:21:11 UTC
Begin processing the 35th record. run: 1408 subRun: 1 event: 63335 at 27-Nov-2025 19:21:13 UTC
Begin processing the 36th record. run: 1408 subRun: 1 event: 63336 at 27-Nov-2025 19:21:15 UTC
Begin processing the 37th record. run: 1408 subRun: 1 event: 63337 at 27-Nov-2025 19:21:17 UTC
Begin processing the 38th record. run: 1408 subRun: 1 event: 63338 at 27-Nov-2025 19:21:19 UTC
Begin processing the 39th record. run: 1408 subRun: 1 event: 63339 at 27-Nov-2025 19:21:26 UTC
Begin processing the 40th record. run: 1408 subRun: 1 event: 63340 at 27-Nov-2025 19:21:29 UTC
PcaShowerParticleBuildingAlgorithm::OpeningAngle - principal eigenvalue less than or equal to 0.
PcaShowerParticleBuildingAlgorithm::OpeningAngle - principal eigenvalue less than or equal to 0.
Begin processing the 41st record. run: 1408 subRun: 1 event: 63341 at 27-Nov-2025 19:21:31 UTC
Begin processing the 42nd record. run: 1408 subRun: 1 event: 63342 at 27-Nov-2025 19:21:33 UTC
Begin processing the 43rd record. run: 1408 subRun: 1 event: 63343 at 27-Nov-2025 19:21:35 UTC
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 44th record. run: 1408 subRun: 1 event: 63344 at 27-Nov-2025 19:21:36 UTC
Begin processing the 45th record. run: 1408 subRun: 1 event: 63345 at 27-Nov-2025 19:21:41 UTC
Begin processing the 46th record. run: 1408 subRun: 1 event: 63346 at 27-Nov-2025 19:21:43 UTC
Begin processing the 47th record. run: 1408 subRun: 1 event: 63347 at 27-Nov-2025 19:21:45 UTC
Begin processing the 48th record. run: 1408 subRun: 1 event: 63348 at 27-Nov-2025 19:21:47 UTC
Begin processing the 49th record. run: 1408 subRun: 1 event: 63349 at 27-Nov-2025 19:21:50 UTC
Begin processing the 50th record. run: 1408 subRun: 1 event: 63350 at 27-Nov-2025 19:21:54 UTC
Begin processing the 51st record. run: 1408 subRun: 1 event: 63351 at 27-Nov-2025 19:21:57 UTC
Begin processing the 52nd record. run: 1408 subRun: 1 event: 63352 at 27-Nov-2025 19:21:58 UTC
Begin processing the 53rd record. run: 1408 subRun: 1 event: 63353 at 27-Nov-2025 19:22:00 UTC
Begin processing the 54th record. run: 1408 subRun: 1 event: 63354 at 27-Nov-2025 19:22:02 UTC
Begin processing the 55th record. run: 1408 subRun: 1 event: 63355 at 27-Nov-2025 19:22:04 UTC
Begin processing the 56th record. run: 1408 subRun: 1 event: 63356 at 27-Nov-2025 19:22:05 UTC
Begin processing the 57th record. run: 1408 subRun: 1 event: 63357 at 27-Nov-2025 19:22:07 UTC
Begin processing the 58th record. run: 1408 subRun: 1 event: 63358 at 27-Nov-2025 19:22:37 UTC
Begin processing the 59th record. run: 1408 subRun: 1 event: 63359 at 27-Nov-2025 19:22:39 UTC
Begin processing the 60th record. run: 1408 subRun: 1 event: 63360 at 27-Nov-2025 19:22:41 UTC
Begin processing the 61st record. run: 1408 subRun: 1 event: 63361 at 27-Nov-2025 19:22:43 UTC
Begin processing the 62nd record. run: 1408 subRun: 1 event: 63362 at 27-Nov-2025 19:22:45 UTC
Begin processing the 63rd record. run: 1408 subRun: 1 event: 63363 at 27-Nov-2025 19:22:47 UTC
Begin processing the 64th record. run: 1408 subRun: 1 event: 63364 at 27-Nov-2025 19:22:49 UTC
Begin processing the 65th record. run: 1408 subRun: 1 event: 63365 at 27-Nov-2025 19:22:51 UTC
Begin processing the 66th record. run: 1408 subRun: 1 event: 63366 at 27-Nov-2025 19:22:52 UTC
Begin processing the 67th record. run: 1408 subRun: 1 event: 63367 at 27-Nov-2025 19:22:55 UTC
Begin processing the 68th record. run: 1408 subRun: 1 event: 63368 at 27-Nov-2025 19:22:57 UTC
Begin processing the 69th record. run: 1408 subRun: 1 event: 63369 at 27-Nov-2025 19:22:59 UTC
Begin processing the 70th record. run: 1408 subRun: 1 event: 63370 at 27-Nov-2025 19:23:01 UTC
Begin processing the 71st record. run: 1408 subRun: 1 event: 63371 at 27-Nov-2025 19:23:03 UTC
Begin processing the 72nd record. run: 1408 subRun: 1 event: 63372 at 27-Nov-2025 19:23:05 UTC
Begin processing the 73rd record. run: 1408 subRun: 1 event: 63373 at 27-Nov-2025 19:23:07 UTC
Begin processing the 74th record. run: 1408 subRun: 1 event: 63374 at 27-Nov-2025 19:23:09 UTC
Begin processing the 75th record. run: 1408 subRun: 1 event: 63375 at 27-Nov-2025 19:23:18 UTC
Begin processing the 76th record. run: 1408 subRun: 1 event: 63376 at 27-Nov-2025 19:23:24 UTC
Begin processing the 77th record. run: 1408 subRun: 1 event: 63377 at 27-Nov-2025 19:23:32 UTC
Begin processing the 78th record. run: 1408 subRun: 1 event: 63378 at 27-Nov-2025 19:23:37 UTC
Begin processing the 79th record. run: 1408 subRun: 1 event: 63379 at 27-Nov-2025 19:23:38 UTC
Begin processing the 80th record. run: 1408 subRun: 1 event: 63380 at 27-Nov-2025 19:23:43 UTC
Begin processing the 81st record. run: 1408 subRun: 1 event: 63381 at 27-Nov-2025 19:23:45 UTC
Begin processing the 82nd record. run: 1408 subRun: 1 event: 63382 at 27-Nov-2025 19:23:47 UTC
Begin processing the 83rd record. run: 1408 subRun: 1 event: 63383 at 27-Nov-2025 19:23:49 UTC
Begin processing the 84th record. run: 1408 subRun: 1 event: 63384 at 27-Nov-2025 19:24:19 UTC
Begin processing the 85th record. run: 1408 subRun: 1 event: 63385 at 27-Nov-2025 19:24:23 UTC
Begin processing the 86th record. run: 1408 subRun: 1 event: 63386 at 27-Nov-2025 19:24:25 UTC
Begin processing the 87th record. run: 1408 subRun: 1 event: 63387 at 27-Nov-2025 19:24:39 UTC
Begin processing the 88th record. run: 1408 subRun: 1 event: 63388 at 27-Nov-2025 19:24:41 UTC
Begin processing the 89th record. run: 1408 subRun: 1 event: 63389 at 27-Nov-2025 19:24:43 UTC
Begin processing the 90th record. run: 1408 subRun: 1 event: 63390 at 27-Nov-2025 19:24:45 UTC
Begin processing the 91st record. run: 1408 subRun: 1 event: 63391 at 27-Nov-2025 19:24:47 UTC
Begin processing the 92nd record. run: 1408 subRun: 1 event: 63392 at 27-Nov-2025 19:24:49 UTC
Begin processing the 93rd record. run: 1408 subRun: 1 event: 63393 at 27-Nov-2025 19:25:54 UTC
Begin processing the 94th record. run: 1408 subRun: 1 event: 63394 at 27-Nov-2025 19:25:56 UTC
Begin processing the 95th record. run: 1408 subRun: 1 event: 63395 at 27-Nov-2025 19:26:26 UTC
Begin processing the 96th record. run: 1408 subRun: 1 event: 63396 at 27-Nov-2025 19:26:29 UTC
Begin processing the 97th record. run: 1408 subRun: 1 event: 63397 at 27-Nov-2025 19:26:32 UTC
Begin processing the 98th record. run: 1408 subRun: 1 event: 63398 at 27-Nov-2025 19:26:37 UTC
Begin processing the 99th record. run: 1408 subRun: 1 event: 63399 at 27-Nov-2025 19:26:39 UTC
Begin processing the 100th record. run: 1408 subRun: 1 event: 63400 at 27-Nov-2025 19:26:41 UTC
27-Nov-2025 19:26:50 UTC  Closed output file "nu_dune10kt_1x2x6_1408_633_20230826T093808Z_gen_g4_detsim_hitreco__20240110T200350Z_reco2_reco2.root"
27-Nov-2025 19:26:50 UTC  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c7/93/nu_dune10kt_1x2x6_1408_633_20230826T093808Z_gen_g4_detsim_hitreco__20240110T200350Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0154446      3.91808       63.7473       1.97217       7.73398        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00207087     0.0248049     0.066359      0.021058      0.0104775       100    
reco:pandora2:StandardPandora                  0.0129281      3.89193       63.7169       1.93816       7.73408        100    
[art]:TriggerResults:TriggerResultInserter    1.4578e-05    3.9456e-05    0.000140905   3.3588e-05    2.24305e-05      100    
end_path:out1:RootOutput                       3.878e-06    6.21731e-06   3.0938e-05     5.35e-06     3.14292e-06      100    
end_path:out1:RootOutput(write)               0.00021808    0.000979718    0.0125651    0.000500788   0.00161767       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2384.86 MB
  Peak resident set size usage (VmHWM): 1314.68 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 2640
-rw-r--r-- 1 dune002 dune     210 Nov 27 19:18 all-input-dids.txt
-rw-r--r-- 1 dune002 dune       0 Nov 27 19:19 debugprod.log
-rw-r--r-- 1 dune002 dune   43847 Nov 27 19:26 jobscript.log
-rw-r--r-- 1 dune002 dune     168 Nov 27 19:26 justin-processed-pfns.txt
drwxr-xr-x 4 dune002 dune      60 Nov 27 19:18 larpandoracontent
-rw-r--r-- 1 dune002 dune  355013 Nov 27 19:26 nu_dune10kt_1x2x6_1408_633_20230826T093808Z_gen_g4_detsim_hitreco__20240110T200350Z_reco2_reco2.root
-rw-r--r-- 1 dune002 dune     519 Nov 27 19:26 reco2_hist.root
-rw-r--r-- 1 dune002 dune 2286987 Nov 27 19:26 trainingFile_nu_dune10kt_1x2x6_1408_633_20230826T093808Z_gen_g4_detsim_hitreco__20240110T200350Z_reco2.root
justIN time: 2026-02-04 07:20:29 UTC       justIN version: 01.06.00