justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253244.87@dunegpschedd02.fnal.gov

Jobsub ID253244.87@dunegpschedd02.fnal.gov
Workflow ID10809
Stage ID1
User nameschellma@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 18:20:53
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2025-11-26 22:16:46
From worker nodeHostnameb9p12p9960.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 21:48:01
Input filesmonte-carlo-010809-000090
JobscriptExit code0
Real time21m (1291s)
CPU time16m (983s = 76%)
Max RSS bytes1413328896 (1347 MiB)
Outputting started2025-11-26 22:09:33
Output files
Finished2025-11-26 22:16:46
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

en_006743_supernova_g4_detsim_20251114T113136Z_reco.root"
Begin processing the 451st record. run: 8191 subRun: 0 event: 67421 at 26-Nov-2025 23:07:19 CET
Begin processing the 452nd record. run: 8191 subRun: 0 event: 67422 at 26-Nov-2025 23:07:22 CET
Begin processing the 453rd record. run: 8191 subRun: 0 event: 67423 at 26-Nov-2025 23:07:24 CET
Begin processing the 454th record. run: 8191 subRun: 0 event: 67424 at 26-Nov-2025 23:07:26 CET
Begin processing the 455th record. run: 8191 subRun: 0 event: 67425 at 26-Nov-2025 23:07:28 CET
Begin processing the 456th record. run: 8191 subRun: 0 event: 67426 at 26-Nov-2025 23:07:31 CET
Begin processing the 457th record. run: 8191 subRun: 0 event: 67427 at 26-Nov-2025 23:07:33 CET
Begin processing the 458th record. run: 8191 subRun: 0 event: 67428 at 26-Nov-2025 23:07:36 CET
Begin processing the 459th record. run: 8191 subRun: 0 event: 67429 at 26-Nov-2025 23:07:38 CET
Begin processing the 460th record. run: 8191 subRun: 0 event: 67430 at 26-Nov-2025 23:07:40 CET
26-Nov-2025 23:07:42 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/69/ef/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202201Z_gen_006743_supernova_g4_detsim_20251114T113136Z_reco.root"
26-Nov-2025 23:07:42 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/fa/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202203Z_gen_006746_supernova_g4_detsim_20251114T102147Z_reco.root"
26-Nov-2025 23:07:42 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/fa/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202203Z_gen_006746_supernova_g4_detsim_20251114T102147Z_reco.root"
Begin processing the 461st record. run: 8191 subRun: 0 event: 67451 at 26-Nov-2025 23:07:42 CET
Begin processing the 462nd record. run: 8191 subRun: 0 event: 67452 at 26-Nov-2025 23:07:44 CET
Begin processing the 463rd record. run: 8191 subRun: 0 event: 67453 at 26-Nov-2025 23:07:47 CET
Begin processing the 464th record. run: 8191 subRun: 0 event: 67454 at 26-Nov-2025 23:07:51 CET
Begin processing the 465th record. run: 8191 subRun: 0 event: 67455 at 26-Nov-2025 23:07:53 CET
Begin processing the 466th record. run: 8191 subRun: 0 event: 67456 at 26-Nov-2025 23:07:55 CET
Begin processing the 467th record. run: 8191 subRun: 0 event: 67457 at 26-Nov-2025 23:07:59 CET
Begin processing the 468th record. run: 8191 subRun: 0 event: 67458 at 26-Nov-2025 23:08:01 CET
Begin processing the 469th record. run: 8191 subRun: 0 event: 67459 at 26-Nov-2025 23:08:05 CET
Begin processing the 470th record. run: 8191 subRun: 0 event: 67460 at 26-Nov-2025 23:08:06 CET
26-Nov-2025 23:08:09 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/fa/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202203Z_gen_006746_supernova_g4_detsim_20251114T102147Z_reco.root"
26-Nov-2025 23:08:09 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d0/d5/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202203Z_gen_006748_supernova_g4_detsim_20251105T151558Z_reco.root"
26-Nov-2025 23:08:09 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d0/d5/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202203Z_gen_006748_supernova_g4_detsim_20251105T151558Z_reco.root"
Begin processing the 471st record. run: 8191 subRun: 0 event: 67471 at 26-Nov-2025 23:08:10 CET
Begin processing the 472nd record. run: 8191 subRun: 0 event: 67472 at 26-Nov-2025 23:08:12 CET
Begin processing the 473rd record. run: 8191 subRun: 0 event: 67473 at 26-Nov-2025 23:08:14 CET
Begin processing the 474th record. run: 8191 subRun: 0 event: 67474 at 26-Nov-2025 23:08:16 CET
Begin processing the 475th record. run: 8191 subRun: 0 event: 67475 at 26-Nov-2025 23:08:18 CET
Begin processing the 476th record. run: 8191 subRun: 0 event: 67476 at 26-Nov-2025 23:08:20 CET
Begin processing the 477th record. run: 8191 subRun: 0 event: 67477 at 26-Nov-2025 23:08:23 CET
Begin processing the 478th record. run: 8191 subRun: 0 event: 67478 at 26-Nov-2025 23:08:26 CET
Begin processing the 479th record. run: 8191 subRun: 0 event: 67479 at 26-Nov-2025 23:08:28 CET
Begin processing the 480th record. run: 8191 subRun: 0 event: 67480 at 26-Nov-2025 23:08:29 CET
26-Nov-2025 23:08:32 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d0/d5/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202203Z_gen_006748_supernova_g4_detsim_20251105T151558Z_reco.root"
26-Nov-2025 23:08:32 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/f8/58/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202206Z_gen_006754_supernova_g4_detsim_20251113T185655Z_reco.root"
26-Nov-2025 23:08:32 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/f8/58/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202206Z_gen_006754_supernova_g4_detsim_20251113T185655Z_reco.root"
Begin processing the 481st record. run: 8191 subRun: 0 event: 67531 at 26-Nov-2025 23:08:32 CET
Begin processing the 482nd record. run: 8191 subRun: 0 event: 67532 at 26-Nov-2025 23:08:35 CET
Begin processing the 483rd record. run: 8191 subRun: 0 event: 67533 at 26-Nov-2025 23:08:39 CET
Begin processing the 484th record. run: 8191 subRun: 0 event: 67534 at 26-Nov-2025 23:08:41 CET
Begin processing the 485th record. run: 8191 subRun: 0 event: 67535 at 26-Nov-2025 23:08:43 CET
Begin processing the 486th record. run: 8191 subRun: 0 event: 67536 at 26-Nov-2025 23:08:47 CET
Begin processing the 487th record. run: 8191 subRun: 0 event: 67537 at 26-Nov-2025 23:08:49 CET
Begin processing the 488th record. run: 8191 subRun: 0 event: 67538 at 26-Nov-2025 23:08:51 CET
Begin processing the 489th record. run: 8191 subRun: 0 event: 67539 at 26-Nov-2025 23:08:53 CET
Begin processing the 490th record. run: 8191 subRun: 0 event: 67540 at 26-Nov-2025 23:08:55 CET
26-Nov-2025 23:08:57 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/f8/58/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202206Z_gen_006754_supernova_g4_detsim_20251113T185655Z_reco.root"
26-Nov-2025 23:08:57 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/0f/da/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202216Z_gen_006764_supernova_g4_detsim_20251114T093207Z_reco.root"
26-Nov-2025 23:08:57 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/0f/da/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202216Z_gen_006764_supernova_g4_detsim_20251114T093207Z_reco.root"
Begin processing the 491st record. run: 8191 subRun: 0 event: 67631 at 26-Nov-2025 23:08:57 CET
Begin processing the 492nd record. run: 8191 subRun: 0 event: 67632 at 26-Nov-2025 23:08:59 CET
Begin processing the 493rd record. run: 8191 subRun: 0 event: 67633 at 26-Nov-2025 23:09:02 CET
Begin processing the 494th record. run: 8191 subRun: 0 event: 67634 at 26-Nov-2025 23:09:04 CET
Begin processing the 495th record. run: 8191 subRun: 0 event: 67635 at 26-Nov-2025 23:09:08 CET
Begin processing the 496th record. run: 8191 subRun: 0 event: 67636 at 26-Nov-2025 23:09:12 CET
Begin processing the 497th record. run: 8191 subRun: 0 event: 67637 at 26-Nov-2025 23:09:15 CET
Begin processing the 498th record. run: 8191 subRun: 0 event: 67638 at 26-Nov-2025 23:09:17 CET
Begin processing the 499th record. run: 8191 subRun: 0 event: 67639 at 26-Nov-2025 23:09:20 CET
Begin processing the 500th record. run: 8191 subRun: 0 event: 67640 at 26-Nov-2025 23:09:22 CET
26-Nov-2025 23:09:25 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/0f/da/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T202216Z_gen_006764_supernova_g4_detsim_20251114T093207Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                             0.881392       1.79513       5.35088       1.59431      0.658891        500    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00333952    0.00661832     0.0429335    0.00535563    0.00427043       500    
end_path:triggerAna:TriggerAnaTree     0.872447       1.78814       5.3359        1.58503      0.658729        500    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2183.92 MB
  Peak resident set size usage (VmHWM): 1413.33 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.

Processing output file prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA__trg_mc_2025a_tpg__tpg_dune10kt_1x2x6__reco__v10_12_01d01__triggerana_dune10kt_1x2x6__v10_12_01d01__ntuple__CENTRAL_prod_v2_s035000_l005000_20251126T181127_c89.root
merged pass1_CERN_000090.json
justIN time: 2025-12-21 12:20:22 UTC       justIN version: 01.05.03