justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 288571.5@dunegpschedd02.fnal.gov

Jobsub ID288571.5@dunegpschedd02.fnal.gov
Workflow ID12095
Stage ID1
User nameimawby@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-01-20 14:50:46
SiteUS_BNL
EntryDUNE_US_BNL_spoolce02
Last heartbeat2026-01-20 15:19:51
From worker nodeHostnamespool0238.sdcc.bnl.gov
cpuinfoAMD EPYC 7351 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-20 14:51:38
Input filesfardet-hd:nue_dune10kt_1x2x6_1414_279_20230826T193313Z_gen_g4_detsim_hitreco__20240221T061501Z_reco2.root
JobscriptExit code0
Real time27m (1664s)
CPU time21m (1271s = 76%)
Max RSS bytes1358348288 (1295 MiB)
Outputting started2026-01-20 15:19:23
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/metrics_nue_1/fnal/12095/1/001/Validation_nue_dune10kt_1x2x6_1414_279_20230826T193313Z_gen_g4_detsim_hitreco__20240221T061501Z_reco2.root
Finished2026-01-20 15:19:51
Saved logsjustin-logs:288571.5-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

_hitreco__20240221T061501Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 35th record. run: 1414 subRun: 1 event: 27935 at 20-Jan-2026 10:03:45 EST
Begin processing the 36th record. run: 1414 subRun: 1 event: 27936 at 20-Jan-2026 10:03:51 EST
Begin processing the 37th record. run: 1414 subRun: 1 event: 27937 at 20-Jan-2026 10:03:56 EST
Begin processing the 38th record. run: 1414 subRun: 1 event: 27938 at 20-Jan-2026 10:04:02 EST
Begin processing the 39th record. run: 1414 subRun: 1 event: 27939 at 20-Jan-2026 10:04:07 EST
Begin processing the 40th record. run: 1414 subRun: 1 event: 27940 at 20-Jan-2026 10:04:12 EST
Begin processing the 41st record. run: 1414 subRun: 1 event: 27941 at 20-Jan-2026 10:04:18 EST
Begin processing the 42nd record. run: 1414 subRun: 1 event: 27942 at 20-Jan-2026 10:04:25 EST
Begin processing the 43rd record. run: 1414 subRun: 1 event: 27943 at 20-Jan-2026 10:05:23 EST
Begin processing the 44th record. run: 1414 subRun: 1 event: 27944 at 20-Jan-2026 10:05:29 EST
Begin processing the 45th record. run: 1414 subRun: 1 event: 27945 at 20-Jan-2026 10:05:35 EST
Begin processing the 46th record. run: 1414 subRun: 1 event: 27946 at 20-Jan-2026 10:05:42 EST
Begin processing the 47th record. run: 1414 subRun: 1 event: 27947 at 20-Jan-2026 10:05:59 EST
Begin processing the 48th record. run: 1414 subRun: 1 event: 27948 at 20-Jan-2026 10:06:06 EST
Begin processing the 49th record. run: 1414 subRun: 1 event: 27949 at 20-Jan-2026 10:06:23 EST
Begin processing the 50th record. run: 1414 subRun: 1 event: 27950 at 20-Jan-2026 10:06:29 EST
Begin processing the 51st record. run: 1414 subRun: 1 event: 27951 at 20-Jan-2026 10:06:35 EST
Begin processing the 52nd record. run: 1414 subRun: 1 event: 27952 at 20-Jan-2026 10:06:43 EST
Begin processing the 53rd record. run: 1414 subRun: 1 event: 27953 at 20-Jan-2026 10:06:50 EST
Begin processing the 54th record. run: 1414 subRun: 1 event: 27954 at 20-Jan-2026 10:06:58 EST
Begin processing the 55th record. run: 1414 subRun: 1 event: 27955 at 20-Jan-2026 10:07:04 EST
Begin processing the 56th record. run: 1414 subRun: 1 event: 27956 at 20-Jan-2026 10:07:12 EST
Begin processing the 57th record. run: 1414 subRun: 1 event: 27957 at 20-Jan-2026 10:11:17 EST
Begin processing the 58th record. run: 1414 subRun: 1 event: 27958 at 20-Jan-2026 10:11:22 EST
Begin processing the 59th record. run: 1414 subRun: 1 event: 27959 at 20-Jan-2026 10:11:29 EST
Begin processing the 60th record. run: 1414 subRun: 1 event: 27960 at 20-Jan-2026 10:13:41 EST
Begin processing the 61st record. run: 1414 subRun: 1 event: 27961 at 20-Jan-2026 10:13:51 EST
Begin processing the 62nd record. run: 1414 subRun: 1 event: 27962 at 20-Jan-2026 10:13:57 EST
Begin processing the 63rd record. run: 1414 subRun: 1 event: 27963 at 20-Jan-2026 10:14:03 EST
Begin processing the 64th record. run: 1414 subRun: 1 event: 27964 at 20-Jan-2026 10:14:10 EST
Begin processing the 65th record. run: 1414 subRun: 1 event: 27965 at 20-Jan-2026 10:14:16 EST
Begin processing the 66th record. run: 1414 subRun: 1 event: 27966 at 20-Jan-2026 10:14:22 EST
Begin processing the 67th record. run: 1414 subRun: 1 event: 27967 at 20-Jan-2026 10:14:33 EST
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
Begin processing the 68th record. run: 1414 subRun: 1 event: 27968 at 20-Jan-2026 10:14:38 EST
Begin processing the 69th record. run: 1414 subRun: 1 event: 27969 at 20-Jan-2026 10:14:44 EST
Begin processing the 70th record. run: 1414 subRun: 1 event: 27970 at 20-Jan-2026 10:14:49 EST
Begin processing the 71st record. run: 1414 subRun: 1 event: 27971 at 20-Jan-2026 10:15:17 EST
Begin processing the 72nd record. run: 1414 subRun: 1 event: 27972 at 20-Jan-2026 10:15:27 EST
Begin processing the 73rd record. run: 1414 subRun: 1 event: 27973 at 20-Jan-2026 10:15:33 EST
Begin processing the 74th record. run: 1414 subRun: 1 event: 27974 at 20-Jan-2026 10:15:40 EST
Begin processing the 75th record. run: 1414 subRun: 1 event: 27975 at 20-Jan-2026 10:15:46 EST
Begin processing the 76th record. run: 1414 subRun: 1 event: 27976 at 20-Jan-2026 10:15:51 EST
Begin processing the 77th record. run: 1414 subRun: 1 event: 27977 at 20-Jan-2026 10:15:56 EST
Begin processing the 78th record. run: 1414 subRun: 1 event: 27978 at 20-Jan-2026 10:16:01 EST
Begin processing the 79th record. run: 1414 subRun: 1 event: 27979 at 20-Jan-2026 10:16:06 EST
Begin processing the 80th record. run: 1414 subRun: 1 event: 27980 at 20-Jan-2026 10:16:12 EST
Begin processing the 81st record. run: 1414 subRun: 1 event: 27981 at 20-Jan-2026 10:16:17 EST
Begin processing the 82nd record. run: 1414 subRun: 1 event: 27982 at 20-Jan-2026 10:16:22 EST
Begin processing the 83rd record. run: 1414 subRun: 1 event: 27983 at 20-Jan-2026 10:16:28 EST
Begin processing the 84th record. run: 1414 subRun: 1 event: 27984 at 20-Jan-2026 10:16:33 EST
Begin processing the 85th record. run: 1414 subRun: 1 event: 27985 at 20-Jan-2026 10:16:39 EST
Begin processing the 86th record. run: 1414 subRun: 1 event: 27986 at 20-Jan-2026 10:17:01 EST
Begin processing the 87th record. run: 1414 subRun: 1 event: 27987 at 20-Jan-2026 10:17:06 EST
Begin processing the 88th record. run: 1414 subRun: 1 event: 27988 at 20-Jan-2026 10:17:12 EST
Begin processing the 89th record. run: 1414 subRun: 1 event: 27989 at 20-Jan-2026 10:17:18 EST
Begin processing the 90th record. run: 1414 subRun: 1 event: 27990 at 20-Jan-2026 10:17:24 EST
Begin processing the 91st record. run: 1414 subRun: 1 event: 27991 at 20-Jan-2026 10:17:31 EST
Begin processing the 92nd record. run: 1414 subRun: 1 event: 27992 at 20-Jan-2026 10:17:49 EST
Begin processing the 93rd record. run: 1414 subRun: 1 event: 27993 at 20-Jan-2026 10:17:55 EST
Begin processing the 94th record. run: 1414 subRun: 1 event: 27994 at 20-Jan-2026 10:18:00 EST
Begin processing the 95th record. run: 1414 subRun: 1 event: 27995 at 20-Jan-2026 10:18:05 EST
Begin processing the 96th record. run: 1414 subRun: 1 event: 27996 at 20-Jan-2026 10:18:11 EST
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 5.41061e-05 5.40987e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 
----view 1: 0.00155005 0.00344771 0.0082677 0.0120721 0.00992489 0.0157778 0.00808716 0.00382137 0.00311142 0.00353712 0.00282657 0.00128531 0.00339478 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 5.40987e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 5.41024e-05 0.0001701 
----view 1: 0.00344771 0.0082677 0.0120721 0.00992489 0.0157778 0.00808716 0.00382137 0.00311142 0.00353712 0.00282657 0.00128531 0.00339478 0.00755709 
Begin processing the 97th record. run: 1414 subRun: 1 event: 27997 at 20-Jan-2026 10:18:52 EST
Begin processing the 98th record. run: 1414 subRun: 1 event: 27998 at 20-Jan-2026 10:18:59 EST
Begin processing the 99th record. run: 1414 subRun: 1 event: 27999 at 20-Jan-2026 10:19:09 EST
Begin processing the 100th record. run: 1414 subRun: 1 event: 28000 at 20-Jan-2026 10:19:15 EST
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
20-Jan-2026 10:19:22 EST  Closed output file "nue_dune10kt_1x2x6_1414_279_20230826T193313Z_gen_g4_detsim_hitreco__20240221T061501Z_reco2_reco2.root"
20-Jan-2026 10:19:22 EST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/4a/5c/nue_dune10kt_1x2x6_1414_279_20230826T193313Z_gen_g4_detsim_hitreco__20240221T061501Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      4.11514       12.4494       240.16        5.51914       27.6304        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0312979     0.0692731     0.113412      0.066847      0.0119786       100    
reco:pandora:StandardPandora                    4.05314       12.3542       239.797       5.43621       27.5983        100    
[art]:TriggerResults:TriggerResultInserter     1.772e-05    4.93914e-05   0.000274103     3.7e-05     4.18444e-05      100    
end_path:out1:RootOutput                       4.44e-06     8.5418e-06     3.526e-05      8.5e-06     4.25064e-06      100    
end_path:out1:RootOutput(write)               0.00188425     0.0255554     0.275313      0.0142647     0.0371591       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2358.06 MB
  Peak resident set size usage (VmHWM): 1358.35 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-04 07:20:06 UTC       justIN version: 01.06.00