justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301483.29@dunegpschedd01.fnal.gov

Jobsub ID301483.29@dunegpschedd01.fnal.gov
Workflow ID12439
Stage ID1
User namelwhite86@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2026-01-27 12:41:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-01-27 12:55:21
From worker nodeHostnamewn1205240.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-27 12:43:09
Input filesfardet-hd:nutau_dune10kt_1x2x6_1415_99_20230826T203329Z_gen_g4_detsim_hitreco__20240219T213132Z_reco2.root
JobscriptExit code0
Real time11m (713s)
CPU time10m (619s = 86%)
Max RSS bytes1443995648 (1377 MiB)
Outputting started2026-01-27 12:55:03
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/12439/1/001/trainingFile_nutau_dune10kt_1x2x6_1415_99_20230826T203329Z_gen_g4_detsim_hitreco__20240219T213132Z_reco2.root
Finished2026-01-27 12:55:21
Saved logsjustin-logs:301483.29-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

white86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 218
Begin processing the 54th record. run: 1415 subRun: 1 event: 9954 at 27-Jan-2026 12:50:07 GMT
Begin processing the 55th record. run: 1415 subRun: 1 event: 9955 at 27-Jan-2026 12:50:09 GMT
Begin processing the 56th record. run: 1415 subRun: 1 event: 9956 at 27-Jan-2026 12:50:26 GMT
Begin processing the 57th record. run: 1415 subRun: 1 event: 9957 at 27-Jan-2026 12:50:27 GMT
Begin processing the 58th record. run: 1415 subRun: 1 event: 9958 at 27-Jan-2026 12:50:29 GMT
Begin processing the 59th record. run: 1415 subRun: 1 event: 9959 at 27-Jan-2026 12:50:31 GMT
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 60th record. run: 1415 subRun: 1 event: 9960 at 27-Jan-2026 12:50:33 GMT
Begin processing the 61st record. run: 1415 subRun: 1 event: 9961 at 27-Jan-2026 12:50:34 GMT
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetAllTrackFeatures
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 218
Begin processing the 62nd record. run: 1415 subRun: 1 event: 9962 at 27-Jan-2026 12:50:36 GMT
Begin processing the 63rd record. run: 1415 subRun: 1 event: 9963 at 27-Jan-2026 12:50:37 GMT
Begin processing the 64th record. run: 1415 subRun: 1 event: 9964 at 27-Jan-2026 12:50:39 GMT
Begin processing the 65th record. run: 1415 subRun: 1 event: 9965 at 27-Jan-2026 12:50:40 GMT
Begin processing the 66th record. run: 1415 subRun: 1 event: 9966 at 27-Jan-2026 12:50:43 GMT
Begin processing the 67th record. run: 1415 subRun: 1 event: 9967 at 27-Jan-2026 12:50:44 GMT
Begin processing the 68th record. run: 1415 subRun: 1 event: 9968 at 27-Jan-2026 12:50:46 GMT
Begin processing the 69th record. run: 1415 subRun: 1 event: 9969 at 27-Jan-2026 12:50:48 GMT
Begin processing the 70th record. run: 1415 subRun: 1 event: 9970 at 27-Jan-2026 12:50:55 GMT
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 71st record. run: 1415 subRun: 1 event: 9971 at 27-Jan-2026 12:50:57 GMT
Begin processing the 72nd record. run: 1415 subRun: 1 event: 9972 at 27-Jan-2026 12:51:08 GMT
Begin processing the 73rd record. run: 1415 subRun: 1 event: 9973 at 27-Jan-2026 12:51:11 GMT
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 74th record. run: 1415 subRun: 1 event: 9974 at 27-Jan-2026 12:51:13 GMT
Begin processing the 75th record. run: 1415 subRun: 1 event: 9975 at 27-Jan-2026 12:51:14 GMT
Begin processing the 76th record. run: 1415 subRun: 1 event: 9976 at 27-Jan-2026 12:51:16 GMT
Begin processing the 77th record. run: 1415 subRun: 1 event: 9977 at 27-Jan-2026 12:51:17 GMT
Begin processing the 78th record. run: 1415 subRun: 1 event: 9978 at 27-Jan-2026 12:51:20 GMT
Begin processing the 79th record. run: 1415 subRun: 1 event: 9979 at 27-Jan-2026 12:51:21 GMT
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetAllTrackFeatures
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 218
Begin processing the 80th record. run: 1415 subRun: 1 event: 9980 at 27-Jan-2026 12:51:23 GMT
Begin processing the 81st record. run: 1415 subRun: 1 event: 9981 at 27-Jan-2026 12:51:27 GMT
Begin processing the 82nd record. run: 1415 subRun: 1 event: 9982 at 27-Jan-2026 12:51:28 GMT
Begin processing the 83rd record. run: 1415 subRun: 1 event: 9983 at 27-Jan-2026 12:51:30 GMT
Begin processing the 84th record. run: 1415 subRun: 1 event: 9984 at 27-Jan-2026 12:51:37 GMT
Begin processing the 85th record. run: 1415 subRun: 1 event: 9985 at 27-Jan-2026 12:51:42 GMT
Begin processing the 86th record. run: 1415 subRun: 1 event: 9986 at 27-Jan-2026 12:51:44 GMT
Begin processing the 87th record. run: 1415 subRun: 1 event: 9987 at 27-Jan-2026 12:51:45 GMT
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetAllTrackFeatures
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 218
Begin processing the 88th record. run: 1415 subRun: 1 event: 9988 at 27-Jan-2026 12:51:47 GMT
Begin processing the 89th record. run: 1415 subRun: 1 event: 9989 at 27-Jan-2026 12:51:48 GMT
Begin processing the 90th record. run: 1415 subRun: 1 event: 9990 at 27-Jan-2026 12:51:50 GMT
Begin processing the 91st record. run: 1415 subRun: 1 event: 9991 at 27-Jan-2026 12:51:55 GMT
Begin processing the 92nd record. run: 1415 subRun: 1 event: 9992 at 27-Jan-2026 12:51:57 GMT
Begin processing the 93rd record. run: 1415 subRun: 1 event: 9993 at 27-Jan-2026 12:51:58 GMT
Begin processing the 94th record. run: 1415 subRun: 1 event: 9994 at 27-Jan-2026 12:53:42 GMT
Begin processing the 95th record. run: 1415 subRun: 1 event: 9995 at 27-Jan-2026 12:53:43 GMT
Begin processing the 96th record. run: 1415 subRun: 1 event: 9996 at 27-Jan-2026 12:54:46 GMT
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-lar-products/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/ALMA9/build/pandora/v04_16_02/src/pandora-v04-16-02/PandoraSDK-v04-00-02/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 97th record. run: 1415 subRun: 1 event: 9997 at 27-Jan-2026 12:54:48 GMT
Begin processing the 98th record. run: 1415 subRun: 1 event: 9998 at 27-Jan-2026 12:54:49 GMT
Begin processing the 99th record. run: 1415 subRun: 1 event: 9999 at 27-Jan-2026 12:54:51 GMT
Begin processing the 100th record. run: 1415 subRun: 1 event: 10000 at 27-Jan-2026 12:54:52 GMT
27-Jan-2026 12:55:02 GMT  Closed output file "nutau_dune10kt_1x2x6_1415_99_20230826T203329Z_gen_g4_detsim_hitreco__20240219T213132Z_reco2_reco2.root"
27-Jan-2026 12:55:02 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/22/3c/nutau_dune10kt_1x2x6_1415_99_20230826T203329Z_gen_g4_detsim_hitreco__20240219T213132Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.01433       5.97504       123.13        1.27483       17.681         100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.026414      0.0515199     0.119676      0.0493459     0.0137694       100    
reco:pandora2:StandardPandora                  0.976273       5.92303       123.031       1.23201       17.6783        100    
[art]:TriggerResults:TriggerResultInserter     9.508e-06    1.70399e-05   0.000119836   1.3917e-05    1.3642e-05       100    
end_path:out1:RootOutput                       2.144e-06    3.03103e-06   2.0929e-05     2.71e-06     2.23108e-06      100    
end_path:out1:RootOutput(write)               0.00014313    0.000311585   0.00110872    0.000236687   0.000189871      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2429.87 MB
  Peak resident set size usage (VmHWM): 1444 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 1720
-rw-r--r-- 1 dune01 dune01     214 Jan 27 12:43 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01       0 Jan 27 12:43 debugprod.log
-rw-r--r-- 1 dune01 dune01   25299 Jan 27 12:55 jobscript.log
-rw-r--r-- 1 dune01 dune01     186 Jan 27 12:55 justin-processed-pfns.txt
drwxr-xr-x 4 dune01 dune01    4096 Jan 27 12:43 larpandoracontent
-rw-r--r-- 1 dune01 dune01  359090 Jan 27 12:55 nutau_dune10kt_1x2x6_1415_99_20230826T203329Z_gen_g4_detsim_hitreco__20240219T213132Z_reco2_reco2.root
-rw-r--r-- 1 dune01 dune01     519 Jan 27 12:55 reco2_hist.root
-rw-r--r-- 1 dune01 dune01 1347398 Jan 27 12:55 trainingFile_nutau_dune10kt_1x2x6_1415_99_20230826T203329Z_gen_g4_detsim_hitreco__20240219T213132Z_reco2.root
justIN time: 2026-02-04 12:02:10 UTC       justIN version: 01.06.00