justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258139.51@dunegpschedd01.fnal.gov

Jobsub ID258139.51@dunegpschedd01.fnal.gov
Workflow ID10771
Stage ID1
User nameschellma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 02:28:04
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2025-11-26 04:13:55
From worker nodeHostnameb9p07p5284.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-26 04:10:13
Input filesmonte-carlo-010771-000037
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-26 04:13:55
Saved logsjustin-logs:258139.51-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

3 at 26-Nov-2025 05:11:33 CET
Begin processing the 24th record. run: 8190 subRun: 0 event: 83024 at 26-Nov-2025 05:11:36 CET
Begin processing the 25th record. run: 8190 subRun: 0 event: 83025 at 26-Nov-2025 05:11:39 CET
Begin processing the 26th record. run: 8190 subRun: 0 event: 83026 at 26-Nov-2025 05:11:41 CET
Begin processing the 27th record. run: 8190 subRun: 0 event: 83027 at 26-Nov-2025 05:11:44 CET
Begin processing the 28th record. run: 8190 subRun: 0 event: 83028 at 26-Nov-2025 05:11:47 CET
Begin processing the 29th record. run: 8190 subRun: 0 event: 83029 at 26-Nov-2025 05:11:51 CET
Begin processing the 30th record. run: 8190 subRun: 0 event: 83030 at 26-Nov-2025 05:11:53 CET
26-Nov-2025 05:11:56 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/34/26/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161850Z_gen_008303_supernova_g4_detsim_20251114T085929Z_reco.root"
26-Nov-2025 05:11:56 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 05:11:56 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
Begin processing the 31st record. run: 8190 subRun: 0 event: 83361 at 26-Nov-2025 05:11:56 CET
Begin processing the 32nd record. run: 8190 subRun: 0 event: 83362 at 26-Nov-2025 05:11:59 CET
Begin processing the 33rd record. run: 8190 subRun: 0 event: 83363 at 26-Nov-2025 05:12:02 CET
Begin processing the 34th record. run: 8190 subRun: 0 event: 83364 at 26-Nov-2025 05:12:04 CET
Begin processing the 35th record. run: 8190 subRun: 0 event: 83365 at 26-Nov-2025 05:12:06 CET
Begin processing the 36th record. run: 8190 subRun: 0 event: 83366 at 26-Nov-2025 05:12:09 CET
Begin processing the 37th record. run: 8190 subRun: 0 event: 83367 at 26-Nov-2025 05:12:11 CET
Begin processing the 38th record. run: 8190 subRun: 0 event: 83368 at 26-Nov-2025 05:12:15 CET
Begin processing the 39th record. run: 8190 subRun: 0 event: 83369 at 26-Nov-2025 05:12:18 CET
Begin processing the 40th record. run: 8190 subRun: 0 event: 83370 at 26-Nov-2025 05:12:21 CET
26-Nov-2025 05:12:24 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 05:12:24 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 05:12:24 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
Begin processing the 41st record. run: 8190 subRun: 0 event: 83381 at 26-Nov-2025 05:12:25 CET
Begin processing the 42nd record. run: 8190 subRun: 0 event: 83382 at 26-Nov-2025 05:12:28 CET
Begin processing the 43rd record. run: 8190 subRun: 0 event: 83383 at 26-Nov-2025 05:12:31 CET
Begin processing the 44th record. run: 8190 subRun: 0 event: 83384 at 26-Nov-2025 05:12:33 CET
Begin processing the 45th record. run: 8190 subRun: 0 event: 83385 at 26-Nov-2025 05:12:35 CET
Begin processing the 46th record. run: 8190 subRun: 0 event: 83386 at 26-Nov-2025 05:12:38 CET
Begin processing the 47th record. run: 8190 subRun: 0 event: 83387 at 26-Nov-2025 05:12:42 CET
Begin processing the 48th record. run: 8190 subRun: 0 event: 83388 at 26-Nov-2025 05:12:45 CET
Begin processing the 49th record. run: 8190 subRun: 0 event: 83389 at 26-Nov-2025 05:12:47 CET
Begin processing the 50th record. run: 8190 subRun: 0 event: 83390 at 26-Nov-2025 05:12:50 CET
26-Nov-2025 05:12:53 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 05:12:53 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 05:12:53 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
Begin processing the 51st record. run: 8190 subRun: 0 event: 83471 at 26-Nov-2025 05:12:53 CET
Begin processing the 52nd record. run: 8190 subRun: 0 event: 83472 at 26-Nov-2025 05:12:55 CET
Begin processing the 53rd record. run: 8190 subRun: 0 event: 83473 at 26-Nov-2025 05:12:58 CET
Begin processing the 54th record. run: 8190 subRun: 0 event: 83474 at 26-Nov-2025 05:13:02 CET
Begin processing the 55th record. run: 8190 subRun: 0 event: 83475 at 26-Nov-2025 05:13:05 CET
Begin processing the 56th record. run: 8190 subRun: 0 event: 83476 at 26-Nov-2025 05:13:08 CET
Begin processing the 57th record. run: 8190 subRun: 0 event: 83477 at 26-Nov-2025 05:13:11 CET
Begin processing the 58th record. run: 8190 subRun: 0 event: 83478 at 26-Nov-2025 05:13:14 CET
Begin processing the 59th record. run: 8190 subRun: 0 event: 83479 at 26-Nov-2025 05:13:16 CET
Begin processing the 60th record. run: 8190 subRun: 0 event: 83480 at 26-Nov-2025 05:13:19 CET
26-Nov-2025 05:13:21 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 05:13:21 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
26-Nov-2025 05:13:21 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
Begin processing the 61st record. run: 8190 subRun: 0 event: 83491 at 26-Nov-2025 05:13:22 CET
Begin processing the 62nd record. run: 8190 subRun: 0 event: 83492 at 26-Nov-2025 05:13:24 CET
Begin processing the 63rd record. run: 8190 subRun: 0 event: 83493 at 26-Nov-2025 05:13:26 CET
Begin processing the 64th record. run: 8190 subRun: 0 event: 83494 at 26-Nov-2025 05:13:29 CET
Begin processing the 65th record. run: 8190 subRun: 0 event: 83495 at 26-Nov-2025 05:13:32 CET
R__unzip: error -5 in inflate (zlib)
R__unzip: error -5 in inflate (zlib)
26-Nov-2025 05:13:33 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.00453849      1.97175       3.72905       1.95899      0.629172        65     
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                 0.0034443    0.00482893     0.0100021    0.00453854    0.00133441       65     
end_path:triggerAna:TriggerAnaTree       1.182        1.99732       3.72029       1.96187      0.583464        64     
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1777.01 MB
  Peak resident set size usage (VmHWM): 1017.11 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Nov-2025 05:13:35 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- FileReadError BEGIN
        ---- FatalRootError BEGIN
          Fatal Root Error: TBasket::ReadBasketBuffers
          fNbytes = 9005153, fKeylen = 119, fObjlen = 20163250, noutot = 0, nout=0, nin=7525521, nbuf=16777215
          ROOT severity: 3000
        ---- FatalRootError END
        
        The above exception was thrown while processing module TriggerAnaTree/triggerAna run: 8190 subRun: 0 event: 83495
      ---- FileReadError END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.

ERROR: Merging failed with return code 1
failed to merge pass1_CERN_000037.json 1
justIN time: 2026-02-08 10:30:22 UTC       justIN version: 01.06.00