justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253245.60@dunegpschedd02.fnal.gov

Jobsub ID253245.60@dunegpschedd02.fnal.gov
Workflow ID10810
Stage ID1
User nameschellma@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 18:20:55
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2025-11-26 22:44:06
From worker nodeHostnameb9p27p6281.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 22:17:02
Input filesmonte-carlo-010810-000058
JobscriptExit code0
Real time22m (1338s)
CPU time17m (1055s = 78%)
Max RSS bytes1446862848 (1379 MiB)
Outputting started2025-11-26 22:39:21
Output files
Finished2025-11-26 22:44:06
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

8T140315Z_gen_000399_supernova_g4_detsim_20251113T180152Z_reco.root"
Begin processing the 451st record. run: 8190 subRun: 0 event: 3981 at 26-Nov-2025 23:36:53 CET
Begin processing the 452nd record. run: 8190 subRun: 0 event: 3982 at 26-Nov-2025 23:36:55 CET
Begin processing the 453rd record. run: 8190 subRun: 0 event: 3983 at 26-Nov-2025 23:36:58 CET
Begin processing the 454th record. run: 8190 subRun: 0 event: 3984 at 26-Nov-2025 23:37:00 CET
Begin processing the 455th record. run: 8190 subRun: 0 event: 3985 at 26-Nov-2025 23:37:02 CET
Begin processing the 456th record. run: 8190 subRun: 0 event: 3986 at 26-Nov-2025 23:37:06 CET
Begin processing the 457th record. run: 8190 subRun: 0 event: 3987 at 26-Nov-2025 23:37:08 CET
Begin processing the 458th record. run: 8190 subRun: 0 event: 3988 at 26-Nov-2025 23:37:12 CET
Begin processing the 459th record. run: 8190 subRun: 0 event: 3989 at 26-Nov-2025 23:37:14 CET
Begin processing the 460th record. run: 8190 subRun: 0 event: 3990 at 26-Nov-2025 23:37:16 CET
26-Nov-2025 23:37:18 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/be/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140315Z_gen_000399_supernova_g4_detsim_20251113T180152Z_reco.root"
26-Nov-2025 23:37:18 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7a/34/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140319Z_gen_009442_supernova_g4_detsim_20251114T092400Z_reco.root"
26-Nov-2025 23:37:18 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7a/34/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140319Z_gen_009442_supernova_g4_detsim_20251114T092400Z_reco.root"
Begin processing the 461st record. run: 8188 subRun: 0 event: 94411 at 26-Nov-2025 23:37:19 CET
Begin processing the 462nd record. run: 8188 subRun: 0 event: 94412 at 26-Nov-2025 23:37:23 CET
Begin processing the 463rd record. run: 8188 subRun: 0 event: 94413 at 26-Nov-2025 23:37:25 CET
Begin processing the 464th record. run: 8188 subRun: 0 event: 94414 at 26-Nov-2025 23:37:27 CET
Begin processing the 465th record. run: 8188 subRun: 0 event: 94415 at 26-Nov-2025 23:37:29 CET
Begin processing the 466th record. run: 8188 subRun: 0 event: 94416 at 26-Nov-2025 23:37:32 CET
Begin processing the 467th record. run: 8188 subRun: 0 event: 94417 at 26-Nov-2025 23:37:36 CET
Begin processing the 468th record. run: 8188 subRun: 0 event: 94418 at 26-Nov-2025 23:37:38 CET
Begin processing the 469th record. run: 8188 subRun: 0 event: 94419 at 26-Nov-2025 23:37:42 CET
Begin processing the 470th record. run: 8188 subRun: 0 event: 94420 at 26-Nov-2025 23:37:45 CET
26-Nov-2025 23:37:47 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7a/34/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140319Z_gen_009442_supernova_g4_detsim_20251114T092400Z_reco.root"
26-Nov-2025 23:37:47 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/ff/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140332Z_gen_009460_supernova_g4_detsim_20251113T140916Z_reco.root"
26-Nov-2025 23:37:47 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/ff/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140332Z_gen_009460_supernova_g4_detsim_20251113T140916Z_reco.root"
Begin processing the 471st record. run: 8188 subRun: 0 event: 94591 at 26-Nov-2025 23:37:48 CET
Begin processing the 472nd record. run: 8188 subRun: 0 event: 94592 at 26-Nov-2025 23:37:50 CET
Begin processing the 473rd record. run: 8188 subRun: 0 event: 94593 at 26-Nov-2025 23:37:52 CET
Begin processing the 474th record. run: 8188 subRun: 0 event: 94594 at 26-Nov-2025 23:37:54 CET
Begin processing the 475th record. run: 8188 subRun: 0 event: 94595 at 26-Nov-2025 23:37:57 CET
Begin processing the 476th record. run: 8188 subRun: 0 event: 94596 at 26-Nov-2025 23:37:59 CET
Begin processing the 477th record. run: 8188 subRun: 0 event: 94597 at 26-Nov-2025 23:38:03 CET
Begin processing the 478th record. run: 8188 subRun: 0 event: 94598 at 26-Nov-2025 23:38:06 CET
Begin processing the 479th record. run: 8188 subRun: 0 event: 94599 at 26-Nov-2025 23:38:07 CET
Begin processing the 480th record. run: 8188 subRun: 0 event: 94600 at 26-Nov-2025 23:38:10 CET
26-Nov-2025 23:38:13 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/ff/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140332Z_gen_009460_supernova_g4_detsim_20251113T140916Z_reco.root"
26-Nov-2025 23:38:13 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/71/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140338Z_gen_009470_supernova_g4_detsim_20251114T111112Z_reco.root"
26-Nov-2025 23:38:13 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/71/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140338Z_gen_009470_supernova_g4_detsim_20251114T111112Z_reco.root"
Begin processing the 481st record. run: 8188 subRun: 0 event: 94691 at 26-Nov-2025 23:38:14 CET
Begin processing the 482nd record. run: 8188 subRun: 0 event: 94692 at 26-Nov-2025 23:38:16 CET
Begin processing the 483rd record. run: 8188 subRun: 0 event: 94693 at 26-Nov-2025 23:38:21 CET
Begin processing the 484th record. run: 8188 subRun: 0 event: 94694 at 26-Nov-2025 23:38:24 CET
Begin processing the 485th record. run: 8188 subRun: 0 event: 94695 at 26-Nov-2025 23:38:26 CET
Begin processing the 486th record. run: 8188 subRun: 0 event: 94696 at 26-Nov-2025 23:38:29 CET
Begin processing the 487th record. run: 8188 subRun: 0 event: 94697 at 26-Nov-2025 23:38:32 CET
Begin processing the 488th record. run: 8188 subRun: 0 event: 94698 at 26-Nov-2025 23:38:34 CET
Begin processing the 489th record. run: 8188 subRun: 0 event: 94699 at 26-Nov-2025 23:38:37 CET
Begin processing the 490th record. run: 8188 subRun: 0 event: 94700 at 26-Nov-2025 23:38:39 CET
26-Nov-2025 23:38:41 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/71/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140338Z_gen_009470_supernova_g4_detsim_20251114T111112Z_reco.root"
26-Nov-2025 23:38:41 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/aa/7f/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140347Z_gen_009489_supernova_g4_detsim_20251114T102347Z_reco.root"
26-Nov-2025 23:38:41 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/aa/7f/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140347Z_gen_009489_supernova_g4_detsim_20251114T102347Z_reco.root"
Begin processing the 491st record. run: 8188 subRun: 0 event: 94881 at 26-Nov-2025 23:38:42 CET
Begin processing the 492nd record. run: 8188 subRun: 0 event: 94882 at 26-Nov-2025 23:38:45 CET
Begin processing the 493rd record. run: 8188 subRun: 0 event: 94883 at 26-Nov-2025 23:38:48 CET
Begin processing the 494th record. run: 8188 subRun: 0 event: 94884 at 26-Nov-2025 23:38:50 CET
Begin processing the 495th record. run: 8188 subRun: 0 event: 94885 at 26-Nov-2025 23:38:54 CET
Begin processing the 496th record. run: 8188 subRun: 0 event: 94886 at 26-Nov-2025 23:38:57 CET
Begin processing the 497th record. run: 8188 subRun: 0 event: 94887 at 26-Nov-2025 23:39:00 CET
Begin processing the 498th record. run: 8188 subRun: 0 event: 94888 at 26-Nov-2025 23:39:02 CET
Begin processing the 499th record. run: 8188 subRun: 0 event: 94889 at 26-Nov-2025 23:39:05 CET
Begin processing the 500th record. run: 8188 subRun: 0 event: 94890 at 26-Nov-2025 23:39:07 CET
26-Nov-2025 23:39:10 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/aa/7f/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T140347Z_gen_009489_supernova_g4_detsim_20251114T102347Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                             0.988068       1.8226        4.66202       1.6707        0.57422        500    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00326348    0.00619053     0.0466228    0.00517733    0.00414355       500    
end_path:triggerAna:TriggerAnaTree     0.983439       1.81606       4.65545       1.66512      0.573883        500    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2184.5 MB
  Peak resident set size usage (VmHWM): 1446.86 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.

Processing output file prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA__trg_mc_2025a_tpg__tpg_dune10kt_1x2x6__reco__v10_12_01d01__triggerana_dune10kt_1x2x6__v10_12_01d01__ntuple__CENTRAL_prod_v2_s040000_l005000_20251126T181234_c57.root
merged pass1_CERN_000058.json
justIN time: 2025-12-21 06:36:32 UTC       justIN version: 01.05.03