justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252631.1@dunegpschedd02.fnal.gov

Jobsub ID252631.1@dunegpschedd02.fnal.gov
Workflow ID10771
Stage ID1
User nameschellma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 04:21:03
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2025-11-26 05:25:19
From worker nodeHostnameb9p04p6494.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-26 05:19:31
Input filesmonte-carlo-010771-000037
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-26 05:25:19
Saved logsjustin-logs:252631.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

3023 at 26-Nov-2025 06:20:53 CET
Begin processing the 24th record. run: 8190 subRun: 0 event: 83024 at 26-Nov-2025 06:20:56 CET
Begin processing the 25th record. run: 8190 subRun: 0 event: 83025 at 26-Nov-2025 06:20:58 CET
Begin processing the 26th record. run: 8190 subRun: 0 event: 83026 at 26-Nov-2025 06:21:01 CET
Begin processing the 27th record. run: 8190 subRun: 0 event: 83027 at 26-Nov-2025 06:21:04 CET
Begin processing the 28th record. run: 8190 subRun: 0 event: 83028 at 26-Nov-2025 06:21:07 CET
Begin processing the 29th record. run: 8190 subRun: 0 event: 83029 at 26-Nov-2025 06:21:11 CET
Begin processing the 30th record. run: 8190 subRun: 0 event: 83030 at 26-Nov-2025 06:21:13 CET
26-Nov-2025 06:21:16 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/34/26/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161850Z_gen_008303_supernova_g4_detsim_20251114T085929Z_reco.root"
26-Nov-2025 06:21:16 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 06:21:16 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
Begin processing the 31st record. run: 8190 subRun: 0 event: 83361 at 26-Nov-2025 06:21:16 CET
Begin processing the 32nd record. run: 8190 subRun: 0 event: 83362 at 26-Nov-2025 06:21:19 CET
Begin processing the 33rd record. run: 8190 subRun: 0 event: 83363 at 26-Nov-2025 06:21:22 CET
Begin processing the 34th record. run: 8190 subRun: 0 event: 83364 at 26-Nov-2025 06:21:24 CET
Begin processing the 35th record. run: 8190 subRun: 0 event: 83365 at 26-Nov-2025 06:21:27 CET
Begin processing the 36th record. run: 8190 subRun: 0 event: 83366 at 26-Nov-2025 06:21:29 CET
Begin processing the 37th record. run: 8190 subRun: 0 event: 83367 at 26-Nov-2025 06:21:31 CET
Begin processing the 38th record. run: 8190 subRun: 0 event: 83368 at 26-Nov-2025 06:21:35 CET
Begin processing the 39th record. run: 8190 subRun: 0 event: 83369 at 26-Nov-2025 06:21:38 CET
Begin processing the 40th record. run: 8190 subRun: 0 event: 83370 at 26-Nov-2025 06:21:41 CET
26-Nov-2025 06:21:44 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 06:21:44 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 06:23:47 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
Begin processing the 41st record. run: 8190 subRun: 0 event: 83381 at 26-Nov-2025 06:23:48 CET
Begin processing the 42nd record. run: 8190 subRun: 0 event: 83382 at 26-Nov-2025 06:23:51 CET
Begin processing the 43rd record. run: 8190 subRun: 0 event: 83383 at 26-Nov-2025 06:23:53 CET
Begin processing the 44th record. run: 8190 subRun: 0 event: 83384 at 26-Nov-2025 06:23:55 CET
Begin processing the 45th record. run: 8190 subRun: 0 event: 83385 at 26-Nov-2025 06:23:58 CET
Begin processing the 46th record. run: 8190 subRun: 0 event: 83386 at 26-Nov-2025 06:24:01 CET
Begin processing the 47th record. run: 8190 subRun: 0 event: 83387 at 26-Nov-2025 06:24:05 CET
Begin processing the 48th record. run: 8190 subRun: 0 event: 83388 at 26-Nov-2025 06:24:07 CET
Begin processing the 49th record. run: 8190 subRun: 0 event: 83389 at 26-Nov-2025 06:24:09 CET
Begin processing the 50th record. run: 8190 subRun: 0 event: 83390 at 26-Nov-2025 06:24:12 CET
26-Nov-2025 06:24:14 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 06:24:14 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 06:24:14 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
Begin processing the 51st record. run: 8190 subRun: 0 event: 83471 at 26-Nov-2025 06:24:15 CET
Begin processing the 52nd record. run: 8190 subRun: 0 event: 83472 at 26-Nov-2025 06:24:17 CET
Begin processing the 53rd record. run: 8190 subRun: 0 event: 83473 at 26-Nov-2025 06:24:19 CET
Begin processing the 54th record. run: 8190 subRun: 0 event: 83474 at 26-Nov-2025 06:24:23 CET
Begin processing the 55th record. run: 8190 subRun: 0 event: 83475 at 26-Nov-2025 06:24:26 CET
Begin processing the 56th record. run: 8190 subRun: 0 event: 83476 at 26-Nov-2025 06:24:30 CET
Begin processing the 57th record. run: 8190 subRun: 0 event: 83477 at 26-Nov-2025 06:24:32 CET
Begin processing the 58th record. run: 8190 subRun: 0 event: 83478 at 26-Nov-2025 06:24:35 CET
Begin processing the 59th record. run: 8190 subRun: 0 event: 83479 at 26-Nov-2025 06:24:37 CET
Begin processing the 60th record. run: 8190 subRun: 0 event: 83480 at 26-Nov-2025 06:24:40 CET
26-Nov-2025 06:24:42 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 06:24:42 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
26-Nov-2025 06:24:43 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
Begin processing the 61st record. run: 8190 subRun: 0 event: 83491 at 26-Nov-2025 06:24:43 CET
Begin processing the 62nd record. run: 8190 subRun: 0 event: 83492 at 26-Nov-2025 06:24:45 CET
Begin processing the 63rd record. run: 8190 subRun: 0 event: 83493 at 26-Nov-2025 06:24:48 CET
Begin processing the 64th record. run: 8190 subRun: 0 event: 83494 at 26-Nov-2025 06:24:51 CET
Begin processing the 65th record. run: 8190 subRun: 0 event: 83495 at 26-Nov-2025 06:24:54 CET
R__unzip: error -5 in inflate (zlib)
R__unzip: error -5 in inflate (zlib)
26-Nov-2025 06:24:55 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                             0.0049258      1.99309       3.70942       1.97787      0.619408        65     
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00314906     0.0042536    0.00838557    0.00381382    0.00109677       65     
end_path:triggerAna:TriggerAnaTree      1.04759       2.0196        3.70468       1.97478      0.571677        64     
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1772.22 MB
  Peak resident set size usage (VmHWM): 1019 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Nov-2025 06:24:57 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- FileReadError BEGIN
        ---- FatalRootError BEGIN
          Fatal Root Error: TBasket::ReadBasketBuffers
          fNbytes = 9005153, fKeylen = 119, fObjlen = 20163250, noutot = 0, nout=0, nin=7525521, nbuf=16777215
          ROOT severity: 3000
        ---- FatalRootError END
        
        The above exception was thrown while processing module TriggerAnaTree/triggerAna run: 8190 subRun: 0 event: 83495
      ---- FileReadError END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.

ERROR: Merging failed with return code 1
failed to merge pass1_CERN_000037.json 1
justIN time: 2026-02-08 10:27:41 UTC       justIN version: 01.06.00