justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236602.0@dunegpschedd02.fnal.gov

Jobsub ID236602.0@dunegpschedd02.fnal.gov
Workflow ID9541
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 10:46:43
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_gk02_condce_mcore
Last heartbeat2025-11-03 11:14:45
From worker nodeHostnameuct2-c613.mwt2.org
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 10:48:13
Input filesfardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim.root
fardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171033Z_gen_002281_supernova_g4_detsim.root
fardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim.root
JobscriptExit code0
Real time19m (1163s)
CPU time17m (1046s = 89%)
Max RSS bytes797732864 (760 MiB)
Outputting started2025-11-03 11:07:37
Output filesusertests:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171033Z_gen_002281_supernova_g4_detsim_20251103T105439Z_reco.root
usertests:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T110103Z_reco.root
usertests:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T104820Z_reco.root
Finished2025-11-03 11:14:45
Saved logsjustin-logs:236602.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ng in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
03-Nov-2025 05:01:16 CST  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/83/6f/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim.root"
03-Nov-2025 05:01:18 CST  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/83/6f/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim.root"
Begin processing the 1st record. run: 8106 subRun: 0 event: 61651 at 03-Nov-2025 05:01:18 CST
03-Nov-2025 05:01:25 CST  Opened output file with pattern "prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T110103Z_reco.root"
Begin processing the 2nd record. run: 8106 subRun: 0 event: 61652 at 03-Nov-2025 05:02:02 CST
Begin processing the 3rd record. run: 8106 subRun: 0 event: 61653 at 03-Nov-2025 05:02:09 CST
Begin processing the 4th record. run: 8106 subRun: 0 event: 61654 at 03-Nov-2025 05:02:16 CST
Begin processing the 5th record. run: 8106 subRun: 0 event: 61655 at 03-Nov-2025 05:02:23 CST
Begin processing the 6th record. run: 8106 subRun: 0 event: 61656 at 03-Nov-2025 05:02:30 CST
Begin processing the 7th record. run: 8106 subRun: 0 event: 61657 at 03-Nov-2025 05:02:37 CST
Begin processing the 8th record. run: 8106 subRun: 0 event: 61658 at 03-Nov-2025 05:02:44 CST
Begin processing the 9th record. run: 8106 subRun: 0 event: 61659 at 03-Nov-2025 05:02:50 CST
Begin processing the 10th record. run: 8106 subRun: 0 event: 61660 at 03-Nov-2025 05:02:56 CST
Begin processing the 11th record. run: 8106 subRun: 0 event: 61661 at 03-Nov-2025 05:03:03 CST
Begin processing the 12th record. run: 8106 subRun: 0 event: 61662 at 03-Nov-2025 05:03:10 CST
Begin processing the 13th record. run: 8106 subRun: 0 event: 61663 at 03-Nov-2025 05:03:17 CST
Begin processing the 14th record. run: 8106 subRun: 0 event: 61664 at 03-Nov-2025 05:03:23 CST
Begin processing the 15th record. run: 8106 subRun: 0 event: 61665 at 03-Nov-2025 05:03:30 CST
Begin processing the 16th record. run: 8106 subRun: 0 event: 61666 at 03-Nov-2025 05:03:37 CST
Begin processing the 17th record. run: 8106 subRun: 0 event: 61667 at 03-Nov-2025 05:03:44 CST
Begin processing the 18th record. run: 8106 subRun: 0 event: 61668 at 03-Nov-2025 05:03:51 CST
Begin processing the 19th record. run: 8106 subRun: 0 event: 61669 at 03-Nov-2025 05:03:58 CST
Begin processing the 20th record. run: 8106 subRun: 0 event: 61670 at 03-Nov-2025 05:04:04 CST
Begin processing the 21st record. run: 8106 subRun: 0 event: 61671 at 03-Nov-2025 05:04:11 CST
Begin processing the 22nd record. run: 8106 subRun: 0 event: 61672 at 03-Nov-2025 05:04:17 CST
Begin processing the 23rd record. run: 8106 subRun: 0 event: 61673 at 03-Nov-2025 05:04:24 CST
Begin processing the 24th record. run: 8106 subRun: 0 event: 61674 at 03-Nov-2025 05:04:30 CST
Begin processing the 25th record. run: 8106 subRun: 0 event: 61675 at 03-Nov-2025 05:04:37 CST
Begin processing the 26th record. run: 8106 subRun: 0 event: 61676 at 03-Nov-2025 05:04:43 CST
Begin processing the 27th record. run: 8106 subRun: 0 event: 61677 at 03-Nov-2025 05:04:50 CST
Begin processing the 28th record. run: 8106 subRun: 0 event: 61678 at 03-Nov-2025 05:04:56 CST
Begin processing the 29th record. run: 8106 subRun: 0 event: 61679 at 03-Nov-2025 05:05:03 CST
Begin processing the 30th record. run: 8106 subRun: 0 event: 61680 at 03-Nov-2025 05:05:09 CST
Begin processing the 31st record. run: 8106 subRun: 0 event: 61681 at 03-Nov-2025 05:05:15 CST
Begin processing the 32nd record. run: 8106 subRun: 0 event: 61682 at 03-Nov-2025 05:05:22 CST
Begin processing the 33rd record. run: 8106 subRun: 0 event: 61683 at 03-Nov-2025 05:05:28 CST
Begin processing the 34th record. run: 8106 subRun: 0 event: 61684 at 03-Nov-2025 05:05:35 CST
Begin processing the 35th record. run: 8106 subRun: 0 event: 61685 at 03-Nov-2025 05:05:41 CST
Begin processing the 36th record. run: 8106 subRun: 0 event: 61686 at 03-Nov-2025 05:05:48 CST
Begin processing the 37th record. run: 8106 subRun: 0 event: 61687 at 03-Nov-2025 05:05:55 CST
Begin processing the 38th record. run: 8106 subRun: 0 event: 61688 at 03-Nov-2025 05:06:02 CST
Begin processing the 39th record. run: 8106 subRun: 0 event: 61689 at 03-Nov-2025 05:06:08 CST
Begin processing the 40th record. run: 8106 subRun: 0 event: 61690 at 03-Nov-2025 05:06:16 CST
Begin processing the 41st record. run: 8106 subRun: 0 event: 61691 at 03-Nov-2025 05:06:23 CST
Begin processing the 42nd record. run: 8106 subRun: 0 event: 61692 at 03-Nov-2025 05:06:30 CST
Begin processing the 43rd record. run: 8106 subRun: 0 event: 61693 at 03-Nov-2025 05:06:37 CST
Begin processing the 44th record. run: 8106 subRun: 0 event: 61694 at 03-Nov-2025 05:06:43 CST
Begin processing the 45th record. run: 8106 subRun: 0 event: 61695 at 03-Nov-2025 05:06:50 CST
Begin processing the 46th record. run: 8106 subRun: 0 event: 61696 at 03-Nov-2025 05:06:57 CST
Begin processing the 47th record. run: 8106 subRun: 0 event: 61697 at 03-Nov-2025 05:07:03 CST
Begin processing the 48th record. run: 8106 subRun: 0 event: 61698 at 03-Nov-2025 05:07:10 CST
Begin processing the 49th record. run: 8106 subRun: 0 event: 61699 at 03-Nov-2025 05:07:17 CST
Begin processing the 50th record. run: 8106 subRun: 0 event: 61700 at 03-Nov-2025 05:07:25 CST
03-Nov-2025 05:07:33 CST  Closed output file "prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T110103Z_reco.root"
03-Nov-2025 05:07:33 CST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/83/6f/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim.root"

=========================================================================================================================================
TimeTracker printout (sec)                                 Min           Avg           Max         Median          RMS         nEvts   
=========================================================================================================================================
Full event                                               5.93556       6.65628       7.30587       6.65601      0.328579        50     
-----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.00030458    0.000939254   0.00631623    0.000474258   0.00119475       50     
makers:tpmakerTPCsimpleThr:TriggerPrimitiveMakerTPC      3.0201        3.60399       4.01927       3.64099      0.217369        50     
makers:tpmakerTPCabsRS:TriggerPrimitiveMakerTPC          2.57879       3.04743       3.32394       3.07973      0.181466        50     
[art]:TriggerResults:TriggerResultInserter             1.7774e-05    3.72131e-05   9.7423e-05    3.35325e-05   1.60605e-05      50     
end_path:out1:RootOutput                                3.426e-06    9.02068e-06   3.1018e-05     7.93e-06     4.71641e-06      50     
end_path:out1:RootOutput(write)                        0.00139551    0.00308306    0.00643979    0.00236957    0.00142548       50     
=========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1537.15 MB
  Peak resident set size usage (VmHWM): 797.733 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
mv: cannot stat 'triggerAnaTree_hist.root': No such file or directory
============OUTPUT FILE:  prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T110103Z_reco.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
did.list
file.list
jobscript.log
justin-processed-pfns.txt
mem.db
old_md.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T104820Z_reco.root
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T104820Z_reco.root.ext.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T104820Z_reco.root.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T104820Z_reco.root.temp.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T110103Z_reco.root
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T110103Z_reco.root.ext.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T110103Z_reco.root.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T110103Z_reco.root.temp.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171033Z_gen_002281_supernova_g4_detsim_20251103T105439Z_reco.root
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171033Z_gen_002281_supernova_g4_detsim_20251103T105439Z_reco.root.ext.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171033Z_gen_002281_supernova_g4_detsim_20251103T105439Z_reco.root.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171033Z_gen_002281_supernova_g4_detsim_20251103T105439Z_reco.root.temp.json
time.db
triggersim_hist.root
justIN time: 2025-11-03 18:30:11 UTC       justIN version: 01.05.01