justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253245.12@dunegpschedd02.fnal.gov

Jobsub ID253245.12@dunegpschedd02.fnal.gov
Workflow ID10810
Stage ID1
User nameschellma@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 18:20:55
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2025-11-26 22:44:05
From worker nodeHostnameb9p09p1619.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 22:15:04
Input filesmonte-carlo-010810-000014
JobscriptExit code0
Real time24m (1462s)
CPU time19m (1156s = 79%)
Max RSS bytes1411497984 (1346 MiB)
Outputting started2025-11-26 22:39:27
Output files
Finished2025-11-26 22:44:05
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gen_002390_supernova_g4_detsim_20251113T143757Z_reco.root"
Begin processing the 451st record. run: 8146 subRun: 0 event: 23891 at 26-Nov-2025 23:36:56 CET
Begin processing the 452nd record. run: 8146 subRun: 0 event: 23892 at 26-Nov-2025 23:36:59 CET
Begin processing the 453rd record. run: 8146 subRun: 0 event: 23893 at 26-Nov-2025 23:37:02 CET
Begin processing the 454th record. run: 8146 subRun: 0 event: 23894 at 26-Nov-2025 23:37:04 CET
Begin processing the 455th record. run: 8146 subRun: 0 event: 23895 at 26-Nov-2025 23:37:07 CET
Begin processing the 456th record. run: 8146 subRun: 0 event: 23896 at 26-Nov-2025 23:37:11 CET
Begin processing the 457th record. run: 8146 subRun: 0 event: 23897 at 26-Nov-2025 23:37:14 CET
Begin processing the 458th record. run: 8146 subRun: 0 event: 23898 at 26-Nov-2025 23:37:18 CET
Begin processing the 459th record. run: 8146 subRun: 0 event: 23899 at 26-Nov-2025 23:37:20 CET
Begin processing the 460th record. run: 8146 subRun: 0 event: 23900 at 26-Nov-2025 23:37:23 CET
26-Nov-2025 23:37:26 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6c/1f/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T084903Z_gen_002390_supernova_g4_detsim_20251113T143757Z_reco.root"
26-Nov-2025 23:37:26 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/53/6a/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085001Z_gen_002415_supernova_g4_detsim_20251113T161711Z_reco.root"
26-Nov-2025 23:37:26 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/53/6a/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085001Z_gen_002415_supernova_g4_detsim_20251113T161711Z_reco.root"
Begin processing the 461st record. run: 8146 subRun: 0 event: 24141 at 26-Nov-2025 23:37:27 CET
Begin processing the 462nd record. run: 8146 subRun: 0 event: 24142 at 26-Nov-2025 23:37:30 CET
Begin processing the 463rd record. run: 8146 subRun: 0 event: 24143 at 26-Nov-2025 23:37:32 CET
Begin processing the 464th record. run: 8146 subRun: 0 event: 24144 at 26-Nov-2025 23:37:34 CET
Begin processing the 465th record. run: 8146 subRun: 0 event: 24145 at 26-Nov-2025 23:37:36 CET
Begin processing the 466th record. run: 8146 subRun: 0 event: 24146 at 26-Nov-2025 23:37:39 CET
Begin processing the 467th record. run: 8146 subRun: 0 event: 24147 at 26-Nov-2025 23:37:42 CET
Begin processing the 468th record. run: 8146 subRun: 0 event: 24148 at 26-Nov-2025 23:37:46 CET
Begin processing the 469th record. run: 8146 subRun: 0 event: 24149 at 26-Nov-2025 23:37:48 CET
Begin processing the 470th record. run: 8146 subRun: 0 event: 24150 at 26-Nov-2025 23:37:52 CET
26-Nov-2025 23:37:54 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/53/6a/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085001Z_gen_002415_supernova_g4_detsim_20251113T161711Z_reco.root"
26-Nov-2025 23:37:54 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/06/4b/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085102Z_gen_002428_supernova_g4_detsim_20251114T091103Z_reco.root"
26-Nov-2025 23:37:55 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/06/4b/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085102Z_gen_002428_supernova_g4_detsim_20251114T091103Z_reco.root"
Begin processing the 471st record. run: 8146 subRun: 0 event: 24271 at 26-Nov-2025 23:37:55 CET
Begin processing the 472nd record. run: 8146 subRun: 0 event: 24272 at 26-Nov-2025 23:37:57 CET
Begin processing the 473rd record. run: 8146 subRun: 0 event: 24273 at 26-Nov-2025 23:38:00 CET
Begin processing the 474th record. run: 8146 subRun: 0 event: 24274 at 26-Nov-2025 23:38:02 CET
Begin processing the 475th record. run: 8146 subRun: 0 event: 24275 at 26-Nov-2025 23:38:05 CET
Begin processing the 476th record. run: 8146 subRun: 0 event: 24276 at 26-Nov-2025 23:38:07 CET
Begin processing the 477th record. run: 8146 subRun: 0 event: 24277 at 26-Nov-2025 23:38:11 CET
Begin processing the 478th record. run: 8146 subRun: 0 event: 24278 at 26-Nov-2025 23:38:14 CET
Begin processing the 479th record. run: 8146 subRun: 0 event: 24279 at 26-Nov-2025 23:38:16 CET
Begin processing the 480th record. run: 8146 subRun: 0 event: 24280 at 26-Nov-2025 23:38:19 CET
26-Nov-2025 23:38:21 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/06/4b/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085102Z_gen_002428_supernova_g4_detsim_20251114T091103Z_reco.root"
26-Nov-2025 23:38:21 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/de/36/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085110Z_gen_002437_supernova_g4_detsim_20251113T151851Z_reco.root"
26-Nov-2025 23:38:21 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/de/36/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085110Z_gen_002437_supernova_g4_detsim_20251113T151851Z_reco.root"
Begin processing the 481st record. run: 8146 subRun: 0 event: 24361 at 26-Nov-2025 23:38:22 CET
Begin processing the 482nd record. run: 8146 subRun: 0 event: 24362 at 26-Nov-2025 23:38:25 CET
Begin processing the 483rd record. run: 8146 subRun: 0 event: 24363 at 26-Nov-2025 23:38:29 CET
Begin processing the 484th record. run: 8146 subRun: 0 event: 24364 at 26-Nov-2025 23:38:31 CET
Begin processing the 485th record. run: 8146 subRun: 0 event: 24365 at 26-Nov-2025 23:38:34 CET
Begin processing the 486th record. run: 8146 subRun: 0 event: 24366 at 26-Nov-2025 23:38:37 CET
Begin processing the 487th record. run: 8146 subRun: 0 event: 24367 at 26-Nov-2025 23:38:39 CET
Begin processing the 488th record. run: 8146 subRun: 0 event: 24368 at 26-Nov-2025 23:38:42 CET
Begin processing the 489th record. run: 8146 subRun: 0 event: 24369 at 26-Nov-2025 23:38:45 CET
Begin processing the 490th record. run: 8146 subRun: 0 event: 24370 at 26-Nov-2025 23:38:47 CET
26-Nov-2025 23:38:49 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/de/36/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085110Z_gen_002437_supernova_g4_detsim_20251113T151851Z_reco.root"
26-Nov-2025 23:38:49 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/85/b4/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085114Z_gen_002439_supernova_g4_detsim_20251105T154438Z_reco.root"
26-Nov-2025 23:38:49 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/85/b4/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085114Z_gen_002439_supernova_g4_detsim_20251105T154438Z_reco.root"
Begin processing the 491st record. run: 8146 subRun: 0 event: 24381 at 26-Nov-2025 23:38:50 CET
Begin processing the 492nd record. run: 8146 subRun: 0 event: 24382 at 26-Nov-2025 23:38:53 CET
Begin processing the 493rd record. run: 8146 subRun: 0 event: 24383 at 26-Nov-2025 23:38:55 CET
Begin processing the 494th record. run: 8146 subRun: 0 event: 24384 at 26-Nov-2025 23:38:57 CET
Begin processing the 495th record. run: 8146 subRun: 0 event: 24385 at 26-Nov-2025 23:39:02 CET
Begin processing the 496th record. run: 8146 subRun: 0 event: 24386 at 26-Nov-2025 23:39:06 CET
Begin processing the 497th record. run: 8146 subRun: 0 event: 24387 at 26-Nov-2025 23:39:08 CET
Begin processing the 498th record. run: 8146 subRun: 0 event: 24388 at 26-Nov-2025 23:39:11 CET
Begin processing the 499th record. run: 8146 subRun: 0 event: 24389 at 26-Nov-2025 23:39:13 CET
Begin processing the 500th record. run: 8146 subRun: 0 event: 24390 at 26-Nov-2025 23:39:16 CET
26-Nov-2025 23:39:18 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/85/b4/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250927T085114Z_gen_002439_supernova_g4_detsim_20251105T154438Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                             0.946031       2.04367       5.48302       1.94608       0.66678        500    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00337181    0.00661067     0.0297163    0.00559899    0.00334199       500    
end_path:triggerAna:TriggerAnaTree     0.940269       2.03667       5.47584       1.94003      0.666582        500    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2179.56 MB
  Peak resident set size usage (VmHWM): 1411.5 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.

Processing output file prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA__trg_mc_2025a_tpg__tpg_dune10kt_1x2x6__reco__v10_12_01d01__triggerana_dune10kt_1x2x6__v10_12_01d01__ntuple__CENTRAL_prod_v2_s040000_l005000_20251126T181234_c13.root
merged pass1_CERN_000014.json
justIN time: 2025-12-21 10:15:54 UTC       justIN version: 01.05.03