justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 276334.162@dunegpschedd02.fnal.gov

Jobsub ID276334.162@dunegpschedd02.fnal.gov
Workflow ID12096
Stage ID1
User nameimawby@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-01-19 14:15:16
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2026-01-19 14:59:50
From worker nodeHostnamedunegli-8213621-0-fnpc17010.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-19 14:34:05
Input filesfardet-hd:nue_dune10kt_1x2x6_1409_443_20230826T103321Z_gen_g4_detsim_hitreco__20240221T025611Z_reco2.root
JobscriptExit code0
Real time25m (1529s)
CPU time24m (1497s = 97%)
Max RSS bytes1329688576 (1268 MiB)
Outputting started2026-01-19 14:59:35
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/metrics_nue_2/fnal/12096/1/001/Validation_nue_dune10kt_1x2x6_1409_443_20230826T103321Z_gen_g4_detsim_hitreco__20240221T025611Z_reco2.root
Finished2026-01-19 14:59:50
Saved logsjustin-logs:276334.162-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/15/90/nue_dune10kt_1x2x6_1409_443_20230826T103321Z_gen_g4_detsim_hitreco__20240221T025611Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Begin processing the 32nd record. run: 1409 subRun: 1 event: 44332 at 19-Jan-2026 14:43:04 UTC
Begin processing the 33rd record. run: 1409 subRun: 1 event: 44333 at 19-Jan-2026 14:43:08 UTC
Begin processing the 34th record. run: 1409 subRun: 1 event: 44334 at 19-Jan-2026 14:43:12 UTC
Begin processing the 35th record. run: 1409 subRun: 1 event: 44335 at 19-Jan-2026 14:43:17 UTC
Begin processing the 36th record. run: 1409 subRun: 1 event: 44336 at 19-Jan-2026 14:43:26 UTC
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
Begin processing the 37th record. run: 1409 subRun: 1 event: 44337 at 19-Jan-2026 14:43:30 UTC
Begin processing the 38th record. run: 1409 subRun: 1 event: 44338 at 19-Jan-2026 14:43:36 UTC
Begin processing the 39th record. run: 1409 subRun: 1 event: 44339 at 19-Jan-2026 14:43:44 UTC
Begin processing the 40th record. run: 1409 subRun: 1 event: 44340 at 19-Jan-2026 14:43:48 UTC
Begin processing the 41st record. run: 1409 subRun: 1 event: 44341 at 19-Jan-2026 14:43:54 UTC
Begin processing the 42nd record. run: 1409 subRun: 1 event: 44342 at 19-Jan-2026 14:43:59 UTC
Begin processing the 43rd record. run: 1409 subRun: 1 event: 44343 at 19-Jan-2026 14:44:21 UTC
Begin processing the 44th record. run: 1409 subRun: 1 event: 44344 at 19-Jan-2026 14:44:28 UTC
Begin processing the 45th record. run: 1409 subRun: 1 event: 44345 at 19-Jan-2026 14:44:34 UTC
Begin processing the 46th record. run: 1409 subRun: 1 event: 44346 at 19-Jan-2026 14:44:39 UTC
Begin processing the 47th record. run: 1409 subRun: 1 event: 44347 at 19-Jan-2026 14:44:43 UTC
Begin processing the 48th record. run: 1409 subRun: 1 event: 44348 at 19-Jan-2026 14:44:48 UTC
Begin processing the 49th record. run: 1409 subRun: 1 event: 44349 at 19-Jan-2026 14:44:53 UTC
Begin processing the 50th record. run: 1409 subRun: 1 event: 44350 at 19-Jan-2026 14:44:58 UTC
Begin processing the 51st record. run: 1409 subRun: 1 event: 44351 at 19-Jan-2026 14:45:02 UTC
Begin processing the 52nd record. run: 1409 subRun: 1 event: 44352 at 19-Jan-2026 14:45:08 UTC
Begin processing the 53rd record. run: 1409 subRun: 1 event: 44353 at 19-Jan-2026 14:45:14 UTC
Begin processing the 54th record. run: 1409 subRun: 1 event: 44354 at 19-Jan-2026 14:46:26 UTC
Begin processing the 55th record. run: 1409 subRun: 1 event: 44355 at 19-Jan-2026 14:46:30 UTC
Begin processing the 56th record. run: 1409 subRun: 1 event: 44356 at 19-Jan-2026 14:48:19 UTC
Begin processing the 57th record. run: 1409 subRun: 1 event: 44357 at 19-Jan-2026 14:48:24 UTC
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
Begin processing the 58th record. run: 1409 subRun: 1 event: 44358 at 19-Jan-2026 14:48:28 UTC
Begin processing the 59th record. run: 1409 subRun: 1 event: 44359 at 19-Jan-2026 14:48:33 UTC
Begin processing the 60th record. run: 1409 subRun: 1 event: 44360 at 19-Jan-2026 14:48:42 UTC
Begin processing the 61st record. run: 1409 subRun: 1 event: 44361 at 19-Jan-2026 14:48:47 UTC
Begin processing the 62nd record. run: 1409 subRun: 1 event: 44362 at 19-Jan-2026 14:48:52 UTC
Begin processing the 63rd record. run: 1409 subRun: 1 event: 44363 at 19-Jan-2026 14:48:58 UTC
Begin processing the 64th record. run: 1409 subRun: 1 event: 44364 at 19-Jan-2026 14:49:02 UTC
PandoraContentApi::GetList(*this, listName, pList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetList
    in file:     /exp/dune/app/users/imawby/dunesw_validation/srcs/larpandoracontent/larpandoradlcontent/LArTwoDReco/DLThreeDClusterSplittingAlgorithm.cc line#: 194
Begin processing the 65th record. run: 1409 subRun: 1 event: 44365 at 19-Jan-2026 14:49:07 UTC
Begin processing the 66th record. run: 1409 subRun: 1 event: 44366 at 19-Jan-2026 14:49:12 UTC
Begin processing the 67th record. run: 1409 subRun: 1 event: 44367 at 19-Jan-2026 14:49:17 UTC
Begin processing the 68th record. run: 1409 subRun: 1 event: 44368 at 19-Jan-2026 14:49:21 UTC
Begin processing the 69th record. run: 1409 subRun: 1 event: 44369 at 19-Jan-2026 14:49:26 UTC
Begin processing the 70th record. run: 1409 subRun: 1 event: 44370 at 19-Jan-2026 14:49:41 UTC
Begin processing the 71st record. run: 1409 subRun: 1 event: 44371 at 19-Jan-2026 14:49:46 UTC
Begin processing the 72nd record. run: 1409 subRun: 1 event: 44372 at 19-Jan-2026 14:49:51 UTC
Begin processing the 73rd record. run: 1409 subRun: 1 event: 44373 at 19-Jan-2026 14:49:56 UTC
Begin processing the 74th record. run: 1409 subRun: 1 event: 44374 at 19-Jan-2026 14:50:03 UTC
Begin processing the 75th record. run: 1409 subRun: 1 event: 44375 at 19-Jan-2026 14:50:08 UTC
Begin processing the 76th record. run: 1409 subRun: 1 event: 44376 at 19-Jan-2026 14:50:12 UTC
Begin processing the 77th record. run: 1409 subRun: 1 event: 44377 at 19-Jan-2026 14:50:15 UTC
Begin processing the 78th record. run: 1409 subRun: 1 event: 44378 at 19-Jan-2026 14:50:25 UTC
Begin processing the 79th record. run: 1409 subRun: 1 event: 44379 at 19-Jan-2026 14:50:30 UTC
Begin processing the 80th record. run: 1409 subRun: 1 event: 44380 at 19-Jan-2026 14:50:35 UTC
Begin processing the 81st record. run: 1409 subRun: 1 event: 44381 at 19-Jan-2026 14:50:40 UTC
Begin processing the 82nd record. run: 1409 subRun: 1 event: 44382 at 19-Jan-2026 14:55:03 UTC
Begin processing the 83rd record. run: 1409 subRun: 1 event: 44383 at 19-Jan-2026 14:55:09 UTC
Begin processing the 84th record. run: 1409 subRun: 1 event: 44384 at 19-Jan-2026 14:55:12 UTC
Begin processing the 85th record. run: 1409 subRun: 1 event: 44385 at 19-Jan-2026 14:55:24 UTC
Begin processing the 86th record. run: 1409 subRun: 1 event: 44386 at 19-Jan-2026 14:55:30 UTC
Begin processing the 87th record. run: 1409 subRun: 1 event: 44387 at 19-Jan-2026 14:55:43 UTC
Begin processing the 88th record. run: 1409 subRun: 1 event: 44388 at 19-Jan-2026 14:55:47 UTC
Begin processing the 89th record. run: 1409 subRun: 1 event: 44389 at 19-Jan-2026 14:55:52 UTC
Begin processing the 90th record. run: 1409 subRun: 1 event: 44390 at 19-Jan-2026 14:55:57 UTC
Begin processing the 91st record. run: 1409 subRun: 1 event: 44391 at 19-Jan-2026 14:56:03 UTC
Begin processing the 92nd record. run: 1409 subRun: 1 event: 44392 at 19-Jan-2026 14:56:07 UTC
Begin processing the 93rd record. run: 1409 subRun: 1 event: 44393 at 19-Jan-2026 14:56:11 UTC
Begin processing the 94th record. run: 1409 subRun: 1 event: 44394 at 19-Jan-2026 14:56:15 UTC
Begin processing the 95th record. run: 1409 subRun: 1 event: 44395 at 19-Jan-2026 14:58:41 UTC
Begin processing the 96th record. run: 1409 subRun: 1 event: 44396 at 19-Jan-2026 14:58:47 UTC
Begin processing the 97th record. run: 1409 subRun: 1 event: 44397 at 19-Jan-2026 14:58:52 UTC
Begin processing the 98th record. run: 1409 subRun: 1 event: 44398 at 19-Jan-2026 14:59:18 UTC
Begin processing the 99th record. run: 1409 subRun: 1 event: 44399 at 19-Jan-2026 14:59:23 UTC
Begin processing the 100th record. run: 1409 subRun: 1 event: 44400 at 19-Jan-2026 14:59:28 UTC
19-Jan-2026 14:59:33 UTC  Closed output file "nue_dune10kt_1x2x6_1409_443_20230826T103321Z_gen_g4_detsim_hitreco__20240221T025611Z_reco2_reco2.root"
19-Jan-2026 14:59:33 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/15/90/nue_dune10kt_1x2x6_1409_443_20230826T103321Z_gen_g4_detsim_hitreco__20240221T025611Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0838342      13.9462       261.91        4.86243       35.7492        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00192525     0.0224861     0.0534304     0.020904      0.0102066       100    
reco:pandora:StandardPandora                   0.0695364      13.901        261.752       4.80937       35.7281        100    
[art]:TriggerResults:TriggerResultInserter    2.3999e-05    8.07688e-05   0.000478292   5.59825e-05   7.11571e-05      100    
end_path:out1:RootOutput                       3.784e-06    6.14213e-06   4.3587e-05     5.236e-06    4.26178e-06      100    
end_path:out1:RootOutput(write)               0.00202104     0.0223294     0.143803      0.0132264     0.0269432       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2318.62 MB
  Peak resident set size usage (VmHWM): 1329.69 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2026-02-05 17:07:58 UTC       justIN version: 01.06.00