justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 246694.0@dunegpschedd01.fnal.gov

Jobsub ID246694.0@dunegpschedd01.fnal.gov
Workflow ID9904
Stage ID1
User nameschellma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-10 15:51:09
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-11-10 15:53:32
From worker nodeHostnamedunegli-7716034-0-fnpc22027.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-10 15:52:40
Input filesmonte-carlo-009904-000001
JobscriptExit code0
Real time0m (27s)
CPU time0m (22s = 81%)
Max RSS bytes608387072 (580 MiB)
Outputting started2025-11-10 15:53:07
Output files
Finished2025-11-10 15:53:32
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

essing the 958th record. run: 8106 subRun: 0 event: 224758 at 10-Nov-2025 21:53:07 UTC
Begin processing the 959th record. run: 8106 subRun: 0 event: 224759 at 10-Nov-2025 21:53:07 UTC
Begin processing the 960th record. run: 8106 subRun: 0 event: 224760 at 10-Nov-2025 21:53:07 UTC
Begin processing the 961st record. run: 8106 subRun: 0 event: 224761 at 10-Nov-2025 21:53:07 UTC
Begin processing the 962nd record. run: 8106 subRun: 0 event: 224762 at 10-Nov-2025 21:53:07 UTC
Begin processing the 963rd record. run: 8106 subRun: 0 event: 224763 at 10-Nov-2025 21:53:07 UTC
Begin processing the 964th record. run: 8106 subRun: 0 event: 224764 at 10-Nov-2025 21:53:07 UTC
Begin processing the 965th record. run: 8106 subRun: 0 event: 224765 at 10-Nov-2025 21:53:07 UTC
Begin processing the 966th record. run: 8106 subRun: 0 event: 224766 at 10-Nov-2025 21:53:07 UTC
Begin processing the 967th record. run: 8106 subRun: 0 event: 224767 at 10-Nov-2025 21:53:07 UTC
Begin processing the 968th record. run: 8106 subRun: 0 event: 224768 at 10-Nov-2025 21:53:07 UTC
Begin processing the 969th record. run: 8106 subRun: 0 event: 224769 at 10-Nov-2025 21:53:07 UTC
Begin processing the 970th record. run: 8106 subRun: 0 event: 224770 at 10-Nov-2025 21:53:07 UTC
Begin processing the 971st record. run: 8106 subRun: 0 event: 224771 at 10-Nov-2025 21:53:07 UTC
Begin processing the 972nd record. run: 8106 subRun: 0 event: 224772 at 10-Nov-2025 21:53:07 UTC
Begin processing the 973rd record. run: 8106 subRun: 0 event: 224773 at 10-Nov-2025 21:53:07 UTC
Begin processing the 974th record. run: 8106 subRun: 0 event: 224774 at 10-Nov-2025 21:53:07 UTC
Begin processing the 975th record. run: 8106 subRun: 0 event: 224775 at 10-Nov-2025 21:53:07 UTC
Begin processing the 976th record. run: 8106 subRun: 0 event: 224776 at 10-Nov-2025 21:53:07 UTC
Begin processing the 977th record. run: 8106 subRun: 0 event: 224777 at 10-Nov-2025 21:53:07 UTC
Begin processing the 978th record. run: 8106 subRun: 0 event: 224778 at 10-Nov-2025 21:53:07 UTC
Begin processing the 979th record. run: 8106 subRun: 0 event: 224779 at 10-Nov-2025 21:53:07 UTC
Begin processing the 980th record. run: 8106 subRun: 0 event: 224780 at 10-Nov-2025 21:53:07 UTC
Begin processing the 981st record. run: 8106 subRun: 0 event: 224781 at 10-Nov-2025 21:53:07 UTC
Begin processing the 982nd record. run: 8106 subRun: 0 event: 224782 at 10-Nov-2025 21:53:07 UTC
Begin processing the 983rd record. run: 8106 subRun: 0 event: 224783 at 10-Nov-2025 21:53:07 UTC
Begin processing the 984th record. run: 8106 subRun: 0 event: 224784 at 10-Nov-2025 21:53:07 UTC
Begin processing the 985th record. run: 8106 subRun: 0 event: 224785 at 10-Nov-2025 21:53:07 UTC
Begin processing the 986th record. run: 8106 subRun: 0 event: 224786 at 10-Nov-2025 21:53:07 UTC
Begin processing the 987th record. run: 8106 subRun: 0 event: 224787 at 10-Nov-2025 21:53:07 UTC
Begin processing the 988th record. run: 8106 subRun: 0 event: 224788 at 10-Nov-2025 21:53:07 UTC
Begin processing the 989th record. run: 8106 subRun: 0 event: 224789 at 10-Nov-2025 21:53:07 UTC
Begin processing the 990th record. run: 8106 subRun: 0 event: 224790 at 10-Nov-2025 21:53:07 UTC
Begin processing the 991st record. run: 8106 subRun: 0 event: 224791 at 10-Nov-2025 21:53:07 UTC
Begin processing the 992nd record. run: 8106 subRun: 0 event: 224792 at 10-Nov-2025 21:53:07 UTC
Begin processing the 993rd record. run: 8106 subRun: 0 event: 224793 at 10-Nov-2025 21:53:07 UTC
Begin processing the 994th record. run: 8106 subRun: 0 event: 224794 at 10-Nov-2025 21:53:07 UTC
Begin processing the 995th record. run: 8106 subRun: 0 event: 224795 at 10-Nov-2025 21:53:07 UTC
Begin processing the 996th record. run: 8106 subRun: 0 event: 224796 at 10-Nov-2025 21:53:07 UTC
Begin processing the 997th record. run: 8106 subRun: 0 event: 224797 at 10-Nov-2025 21:53:07 UTC
Begin processing the 998th record. run: 8106 subRun: 0 event: 224798 at 10-Nov-2025 21:53:07 UTC
Begin processing the 999th record. run: 8106 subRun: 0 event: 224799 at 10-Nov-2025 21:53:07 UTC
Begin processing the 1000th record. run: 8106 subRun: 0 event: 224800 at 10-Nov-2025 21:53:07 UTC
10-Nov-2025 21:53:07 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/5b/5e/prodmarley_nue_flat_es_dune10kt_1x2x2_20250927T164340Z_gen_004496_supernova_g4_detsim.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.000859098   0.00250435     0.022936     0.00161659    0.00241382      1000    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                7.7257e-05    0.000202469    0.0213762    9.28555e-05   0.00107877      1000    
end_path:triggerAna:TriggerAnaTree    0.00069662    0.00218454     0.011853     0.00135448    0.00212015      1000    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1267.12 MB
  Peak resident set size usage (VmHWM): 608.387 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Merging 20 files into ./prodmarley_nue_flat_es_dune10kt_1x2x2.fcl_trg_mc_2025a_detsim_dune10kt_1x2x2_notpcsigproc.fcl_detsim_v10_06_00d01_triggerana_dune10kt_1x2x2.fcl_s0_l20_hists_merged_20251110T214512.root using method lar
lar -c /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/362c01740f3c2747249e13f4fb8ac86b7a3b7c09/triggerana_dune10kt_1x2x2.fcl -n 10000000 -s root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/c2/90/prodmarley_nue_flat_es_dune10kt_1x2x2_20250826T083554Z_gen_000433_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/31/f9/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T162134Z_gen_000183_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ea/50/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T163509Z_gen_000520_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ae/5a/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T163750Z_gen_000585_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/f0/b7/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164516Z_gen_000843_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e5/50/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/83/6f/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/13/ab/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/d3/e7/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T170754Z_gen_002072_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/cd/40/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171033Z_gen_002281_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/d6/ec/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171038Z_gen_002292_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/77/98/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171109Z_gen_002345_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/f7/27/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T172443Z_gen_003065_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/e3/7f/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T173701Z_gen_003770_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ba/c1/prodmarley_nue_flat_es_dune10kt_1x2x2_20250927T061509Z_gen_000071_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ab/db/prodmarley_nue_flat_es_dune10kt_1x2x2_20250927T063210Z_gen_000253_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/97/87/prodmarley_nue_flat_es_dune10kt_1x2x2_20250927T065320Z_gen_000543_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/5c/5f/prodmarley_nue_flat_es_dune10kt_1x2x2_20250927T163202Z_gen_003253_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/a0/cd/prodmarley_nue_flat_es_dune10kt_1x2x2_20250927T163809Z_gen_003556_supernova_g4_detsim.root root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/5b/5e/prodmarley_nue_flat_es_dune10kt_1x2x2_20250927T164340Z_gen_004496_supernova_g4_detsim.root && mv triggerAna.ntuple.root ./prodmarley_nue_flat_es_dune10kt_1x2x2.fcl_trg_mc_2025a_detsim_dune10kt_1x2x2_notpcsigproc.fcl_detsim_v10_06_00d01_triggerana_dune10kt_1x2x2.fcl_s0_l20_hists_merged_20251110T214512.root
merged pass1_US_FNAL-FermiGrid_000001.json
justIN time: 2026-02-10 11:44:04 UTC       justIN version: 01.06.00