Jobsub ID 13471.105@dunegpschedd02.fnal.gov

Jobsub ID	13471.105@dunegpschedd02.fnal.gov
Workflow ID	177
Stage ID	1
User name	lwhite86@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	3600 (1 hours)
Submitted time	2025-07-31 21:13:20
Site	UK_Oxford
Entry	DUNE_UK_SGrid_Oxford_arc01
Last heartbeat	2025-07-31 21:25:11
From worker node	Hostname	t2wn068.physics.ox.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 21:15:09
Input files	fardet-hd:anutau_dune10kt_1x2x6_1085_94_20230825T013236Z_gen_g4_detsim_hitreco__20240220T195916Z_reco2.root
Outputting started	2025-07-31 21:24:31
Output files
Finished	2025-07-31 22:25:57
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ssing the 58th record. run: 1085 subRun: 1 event: 9458 at 31-Jul-2025 22:20:50 BST
Begin processing the 59th record. run: 1085 subRun: 1 event: 9459 at 31-Jul-2025 22:20:52 BST
Begin processing the 60th record. run: 1085 subRun: 1 event: 9460 at 31-Jul-2025 22:20:56 BST
Begin processing the 61st record. run: 1085 subRun: 1 event: 9461 at 31-Jul-2025 22:20:58 BST
Begin processing the 62nd record. run: 1085 subRun: 1 event: 9462 at 31-Jul-2025 22:21:01 BST
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
Begin processing the 63rd record. run: 1085 subRun: 1 event: 9463 at 31-Jul-2025 22:21:03 BST
Begin processing the 64th record. run: 1085 subRun: 1 event: 9464 at 31-Jul-2025 22:21:06 BST
Begin processing the 65th record. run: 1085 subRun: 1 event: 9465 at 31-Jul-2025 22:21:10 BST
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
Begin processing the 66th record. run: 1085 subRun: 1 event: 9466 at 31-Jul-2025 22:21:12 BST
Begin processing the 67th record. run: 1085 subRun: 1 event: 9467 at 31-Jul-2025 22:21:22 BST
Begin processing the 68th record. run: 1085 subRun: 1 event: 9468 at 31-Jul-2025 22:21:25 BST
Begin processing the 69th record. run: 1085 subRun: 1 event: 9469 at 31-Jul-2025 22:21:41 BST
Begin processing the 70th record. run: 1085 subRun: 1 event: 9470 at 31-Jul-2025 22:21:44 BST
Begin processing the 71st record. run: 1085 subRun: 1 event: 9471 at 31-Jul-2025 22:21:47 BST
Begin processing the 72nd record. run: 1085 subRun: 1 event: 9472 at 31-Jul-2025 22:21:50 BST
Begin processing the 73rd record. run: 1085 subRun: 1 event: 9473 at 31-Jul-2025 22:21:52 BST
Begin processing the 74th record. run: 1085 subRun: 1 event: 9474 at 31-Jul-2025 22:22:25 BST
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
Begin processing the 75th record. run: 1085 subRun: 1 event: 9475 at 31-Jul-2025 22:22:28 BST
Begin processing the 76th record. run: 1085 subRun: 1 event: 9476 at 31-Jul-2025 22:22:31 BST
Begin processing the 77th record. run: 1085 subRun: 1 event: 9477 at 31-Jul-2025 22:22:34 BST
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
Begin processing the 78th record. run: 1085 subRun: 1 event: 9478 at 31-Jul-2025 22:22:36 BST
Begin processing the 79th record. run: 1085 subRun: 1 event: 9479 at 31-Jul-2025 22:22:40 BST
Begin processing the 80th record. run: 1085 subRun: 1 event: 9480 at 31-Jul-2025 22:22:47 BST
Begin processing the 81st record. run: 1085 subRun: 1 event: 9481 at 31-Jul-2025 22:22:49 BST
Begin processing the 82nd record. run: 1085 subRun: 1 event: 9482 at 31-Jul-2025 22:22:52 BST
Begin processing the 83rd record. run: 1085 subRun: 1 event: 9483 at 31-Jul-2025 22:23:08 BST
Begin processing the 84th record. run: 1085 subRun: 1 event: 9484 at 31-Jul-2025 22:23:11 BST
Begin processing the 85th record. run: 1085 subRun: 1 event: 9485 at 31-Jul-2025 22:23:14 BST
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
Begin processing the 86th record. run: 1085 subRun: 1 event: 9486 at 31-Jul-2025 22:23:17 BST
Begin processing the 87th record. run: 1085 subRun: 1 event: 9487 at 31-Jul-2025 22:23:20 BST
Begin processing the 88th record. run: 1085 subRun: 1 event: 9488 at 31-Jul-2025 22:23:22 BST
Begin processing the 89th record. run: 1085 subRun: 1 event: 9489 at 31-Jul-2025 22:23:25 BST
Begin processing the 90th record. run: 1085 subRun: 1 event: 9490 at 31-Jul-2025 22:23:28 BST
Begin processing the 91st record. run: 1085 subRun: 1 event: 9491 at 31-Jul-2025 22:23:33 BST
CNNTrackShowerCountingAlgorithm::GetCrop1D: reconstructed vertex outside cropped region! 362.63, 106.615, 362.615
CNNTrackShowerCountingAlgorithm::GetCrop1D: reconstructed vertex outside cropped region! 362.63, 106.213, 362.213
Begin processing the 92nd record. run: 1085 subRun: 1 event: 9492 at 31-Jul-2025 22:23:36 BST
Begin processing the 93rd record. run: 1085 subRun: 1 event: 9493 at 31-Jul-2025 22:23:45 BST
Begin processing the 94th record. run: 1085 subRun: 1 event: 9494 at 31-Jul-2025 22:23:48 BST
Begin processing the 95th record. run: 1085 subRun: 1 event: 9495 at 31-Jul-2025 22:24:04 BST
Begin processing the 96th record. run: 1085 subRun: 1 event: 9496 at 31-Jul-2025 22:24:07 BST
Begin processing the 97th record. run: 1085 subRun: 1 event: 9497 at 31-Jul-2025 22:24:10 BST
Begin processing the 98th record. run: 1085 subRun: 1 event: 9498 at 31-Jul-2025 22:24:13 BST
Begin processing the 99th record. run: 1085 subRun: 1 event: 9499 at 31-Jul-2025 22:24:16 BST
Begin processing the 100th record. run: 1085 subRun: 1 event: 9500 at 31-Jul-2025 22:24:26 BST
Skipping event as it does not have enough hits or associated primary particles to make a training sample
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArCNNTrackShowerCounting, STATUS_CODE_FAILURE
31-Jul-2025 22:24:30 BST  Closed output file "anutau_dune10kt_1x2x6_1085_94_20230825T013236Z_gen_g4_detsim_hitreco__20240220T195916Z_reco2_reco2.root"
31-Jul-2025 22:24:30 BST  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/53/aa/anutau_dune10kt_1x2x6_1085_94_20230825T013236Z_gen_g4_detsim_hitreco__20240220T195916Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.430674       3.70222       30.5972       1.98343       5.11311        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0660014     0.184775      0.477189      0.174556      0.0717899       100    
reco:pandora2:StandardPandora                  0.220222       3.51634       30.3659       1.80306       5.11064        100    
[art]:TriggerResults:TriggerResultInserter    1.2514e-05    3.28972e-05   0.000488733   2.2893e-05    4.82232e-05      100    
end_path:out1:RootOutput                       3.476e-06    5.35567e-06   3.6939e-05    4.6085e-06    3.82572e-06      100    
end_path:out1:RootOutput(write)               0.000207517   0.000795307   0.00451409    0.000480678   0.000808545      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2247.85 MB
  Peak resident set size usage (VmHWM): 1172.47 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 1596
-rw-r--r-- 1 dune020 dune     216 Jul 31 22:15 all-input-dids.txt
-rw-r--r-- 1 dune020 dune  350698 Jul 31 22:24 anutau_dune10kt_1x2x6_1085_94_20230825T013236Z_gen_g4_detsim_hitreco__20240220T195916Z_reco2_reco2.root
-rw-r--r-- 1 dune020 dune       0 Jul 31 22:15 debugprod.log
-rw-r--r-- 1 dune020 dune   50465 Jul 31 22:24 jobscript.log
-rw-r--r-- 1 dune020 dune     169 Jul 31 22:24 justin-processed-pfns.txt
drwxr-xr-x 4 dune020 dune      60 Jul 31 22:15 larpandoracontent
-rw-r--r-- 1 dune020 dune     519 Jul 31 22:24 reco2_hist.root
-rw-r--r-- 1 dune020 dune 1213409 Jul 31 22:24 trainingFile_anutau_dune10kt_1x2x6_1085_94_20230825T013236Z_gen_g4_detsim_hitreco__20240220T195916Z_reco2.root