Jobsub ID 242356.1@dunegpschedd01.fnal.gov

Jobsub ID	242356.1@dunegpschedd01.fnal.gov
Workflow ID	9552
Stage ID	1
User name	epennacc@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-03 12:58:54
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat	2025-11-03 13:44:04
From worker node	Hostname	mwt2-c171.campuscluster.illinois.edu
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-03 12:59:26
Input files	fardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim.root fardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim.root fardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171109Z_gen_002345_supernova_g4_detsim.root fardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim.root
Jobscript	Exit code	0
	Real time	38m (2339s)
	CPU time	34m (2066s = 88%)
	Max RSS bytes	794374144 (757 MiB)
Outputting started	2025-11-03 13:38:25
Output files	usertests:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T130818Z_reco.root usertests:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T125934Z_reco.root usertests:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim_20251103T132810Z_reco.root usertests:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171109Z_gen_002345_supernova_g4_detsim_20251103T131801Z_reco.root
Finished	2025-11-03 13:44:04
Saved logs	justin-logs:242356.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/13/ab/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim.root"
Begin processing the 1st record. run: 8106 subRun: 0 event: 66201 at 03-Nov-2025 07:28:42 CST
03-Nov-2025 07:28:53 CST  Opened output file with pattern "prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim_20251103T132810Z_reco.root"
Begin processing the 2nd record. run: 8106 subRun: 0 event: 66202 at 03-Nov-2025 07:29:34 CST
Begin processing the 3rd record. run: 8106 subRun: 0 event: 66203 at 03-Nov-2025 07:29:44 CST
Begin processing the 4th record. run: 8106 subRun: 0 event: 66204 at 03-Nov-2025 07:29:55 CST
Begin processing the 5th record. run: 8106 subRun: 0 event: 66205 at 03-Nov-2025 07:30:05 CST
Begin processing the 6th record. run: 8106 subRun: 0 event: 66206 at 03-Nov-2025 07:30:15 CST
Begin processing the 7th record. run: 8106 subRun: 0 event: 66207 at 03-Nov-2025 07:30:26 CST
Begin processing the 8th record. run: 8106 subRun: 0 event: 66208 at 03-Nov-2025 07:30:36 CST
Begin processing the 9th record. run: 8106 subRun: 0 event: 66209 at 03-Nov-2025 07:30:46 CST
Begin processing the 10th record. run: 8106 subRun: 0 event: 66210 at 03-Nov-2025 07:30:56 CST
Begin processing the 11th record. run: 8106 subRun: 0 event: 66211 at 03-Nov-2025 07:31:07 CST
Begin processing the 12th record. run: 8106 subRun: 0 event: 66212 at 03-Nov-2025 07:31:17 CST
Begin processing the 13th record. run: 8106 subRun: 0 event: 66213 at 03-Nov-2025 07:31:28 CST
Begin processing the 14th record. run: 8106 subRun: 0 event: 66214 at 03-Nov-2025 07:31:38 CST
Begin processing the 15th record. run: 8106 subRun: 0 event: 66215 at 03-Nov-2025 07:31:48 CST
Begin processing the 16th record. run: 8106 subRun: 0 event: 66216 at 03-Nov-2025 07:31:59 CST
Begin processing the 17th record. run: 8106 subRun: 0 event: 66217 at 03-Nov-2025 07:32:11 CST
Begin processing the 18th record. run: 8106 subRun: 0 event: 66218 at 03-Nov-2025 07:32:21 CST
Begin processing the 19th record. run: 8106 subRun: 0 event: 66219 at 03-Nov-2025 07:32:33 CST
Begin processing the 20th record. run: 8106 subRun: 0 event: 66220 at 03-Nov-2025 07:32:44 CST
Begin processing the 21st record. run: 8106 subRun: 0 event: 66221 at 03-Nov-2025 07:32:55 CST
Begin processing the 22nd record. run: 8106 subRun: 0 event: 66222 at 03-Nov-2025 07:33:06 CST
Begin processing the 23rd record. run: 8106 subRun: 0 event: 66223 at 03-Nov-2025 07:33:17 CST
Begin processing the 24th record. run: 8106 subRun: 0 event: 66224 at 03-Nov-2025 07:33:29 CST
Begin processing the 25th record. run: 8106 subRun: 0 event: 66225 at 03-Nov-2025 07:33:40 CST
Begin processing the 26th record. run: 8106 subRun: 0 event: 66226 at 03-Nov-2025 07:33:50 CST
Begin processing the 27th record. run: 8106 subRun: 0 event: 66227 at 03-Nov-2025 07:34:01 CST
Begin processing the 28th record. run: 8106 subRun: 0 event: 66228 at 03-Nov-2025 07:34:12 CST
Begin processing the 29th record. run: 8106 subRun: 0 event: 66229 at 03-Nov-2025 07:34:23 CST
Begin processing the 30th record. run: 8106 subRun: 0 event: 66230 at 03-Nov-2025 07:34:34 CST
Begin processing the 31st record. run: 8106 subRun: 0 event: 66231 at 03-Nov-2025 07:34:45 CST
Begin processing the 32nd record. run: 8106 subRun: 0 event: 66232 at 03-Nov-2025 07:34:55 CST
Begin processing the 33rd record. run: 8106 subRun: 0 event: 66233 at 03-Nov-2025 07:35:06 CST
Begin processing the 34th record. run: 8106 subRun: 0 event: 66234 at 03-Nov-2025 07:35:17 CST
Begin processing the 35th record. run: 8106 subRun: 0 event: 66235 at 03-Nov-2025 07:35:27 CST
Begin processing the 36th record. run: 8106 subRun: 0 event: 66236 at 03-Nov-2025 07:35:38 CST
Begin processing the 37th record. run: 8106 subRun: 0 event: 66237 at 03-Nov-2025 07:35:49 CST
Begin processing the 38th record. run: 8106 subRun: 0 event: 66238 at 03-Nov-2025 07:36:00 CST
Begin processing the 39th record. run: 8106 subRun: 0 event: 66239 at 03-Nov-2025 07:36:10 CST
Begin processing the 40th record. run: 8106 subRun: 0 event: 66240 at 03-Nov-2025 07:36:21 CST
Begin processing the 41st record. run: 8106 subRun: 0 event: 66241 at 03-Nov-2025 07:36:31 CST
Begin processing the 42nd record. run: 8106 subRun: 0 event: 66242 at 03-Nov-2025 07:36:42 CST
Begin processing the 43rd record. run: 8106 subRun: 0 event: 66243 at 03-Nov-2025 07:36:53 CST
Begin processing the 44th record. run: 8106 subRun: 0 event: 66244 at 03-Nov-2025 07:37:04 CST
Begin processing the 45th record. run: 8106 subRun: 0 event: 66245 at 03-Nov-2025 07:37:14 CST
Begin processing the 46th record. run: 8106 subRun: 0 event: 66246 at 03-Nov-2025 07:37:25 CST
Begin processing the 47th record. run: 8106 subRun: 0 event: 66247 at 03-Nov-2025 07:37:36 CST
Begin processing the 48th record. run: 8106 subRun: 0 event: 66248 at 03-Nov-2025 07:37:47 CST
Begin processing the 49th record. run: 8106 subRun: 0 event: 66249 at 03-Nov-2025 07:37:58 CST
Begin processing the 50th record. run: 8106 subRun: 0 event: 66250 at 03-Nov-2025 07:38:09 CST
03-Nov-2025 07:38:21 CST  Closed output file "prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim_20251103T132810Z_reco.root"
03-Nov-2025 07:38:21 CST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/13/ab/prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim.root"

=========================================================================================================================================
TimeTracker printout (sec)                                 Min           Avg           Max         Median          RMS         nEvts   
=========================================================================================================================================
Full event                                               9.84578       10.6261       11.4221       10.6288      0.390845        50     
-----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.000323131   0.00155956     0.0227876    0.000533862   0.00344089       50     
makers:tpmakerTPCsimpleThr:TriggerPrimitiveMakerTPC      5.35144       5.74818       6.17527       5.76072      0.199826        50     
makers:tpmakerTPCabsRS:TriggerPrimitiveMakerTPC          4.43886       4.87143       5.29681       4.8652       0.231532        50     
[art]:TriggerResults:TriggerResultInserter             2.8093e-05    5.3827e-05    0.000160087   4.74515e-05   2.39862e-05      50     
end_path:out1:RootOutput                                4.473e-06    1.10321e-05   4.3086e-05    9.3995e-06    7.02653e-06      50     
end_path:out1:RootOutput(write)                        0.00245257    0.00398423     0.0126663    0.00352006    0.00154109       50     
=========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1536.88 MB
  Peak resident set size usage (VmHWM): 784.134 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
mv: cannot stat 'triggerAnaTree_hist.root': No such file or directory
============OUTPUT FILE:  prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim_20251103T132810Z_reco.root
Extracted metadata
checking file====== fardet-hd:prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
did.list
file.list
jobscript.log
mem.db
old_md.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T125934Z_reco.root
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T125934Z_reco.root.ext.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T125934Z_reco.root.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T164636Z_gen_000933_supernova_g4_detsim_20251103T125934Z_reco.root.temp.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T130818Z_reco.root
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T130818Z_reco.root.ext.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T130818Z_reco.root.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165427Z_gen_001234_supernova_g4_detsim_20251103T130818Z_reco.root.temp.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim_20251103T132810Z_reco.root
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim_20251103T132810Z_reco.root.ext.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim_20251103T132810Z_reco.root.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T165551Z_gen_001325_supernova_g4_detsim_20251103T132810Z_reco.root.temp.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171109Z_gen_002345_supernova_g4_detsim_20251103T131801Z_reco.root
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171109Z_gen_002345_supernova_g4_detsim_20251103T131801Z_reco.root.ext.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171109Z_gen_002345_supernova_g4_detsim_20251103T131801Z_reco.root.json
prodmarley_nue_flat_es_dune10kt_1x2x2_20250926T171109Z_gen_002345_supernova_g4_detsim_20251103T131801Z_reco.root.temp.json
temp_pfns.txt
time.db
triggersim_hist.root