justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258364.1@dunegpschedd01.fnal.gov

Jobsub ID258364.1@dunegpschedd01.fnal.gov
Workflow ID10771
Stage ID1
User nameschellma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 05:31:06
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2025-11-26 05:36:22
From worker nodeHostnameb9p09p9494.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-26 05:32:36
Input filesmonte-carlo-010771-000037
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-26 05:36:22
Saved logsjustin-logs:258364.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

3 at 26-Nov-2025 06:33:57 CET
Begin processing the 24th record. run: 8190 subRun: 0 event: 83024 at 26-Nov-2025 06:34:00 CET
Begin processing the 25th record. run: 8190 subRun: 0 event: 83025 at 26-Nov-2025 06:34:02 CET
Begin processing the 26th record. run: 8190 subRun: 0 event: 83026 at 26-Nov-2025 06:34:05 CET
Begin processing the 27th record. run: 8190 subRun: 0 event: 83027 at 26-Nov-2025 06:34:09 CET
Begin processing the 28th record. run: 8190 subRun: 0 event: 83028 at 26-Nov-2025 06:34:12 CET
Begin processing the 29th record. run: 8190 subRun: 0 event: 83029 at 26-Nov-2025 06:34:17 CET
Begin processing the 30th record. run: 8190 subRun: 0 event: 83030 at 26-Nov-2025 06:34:19 CET
26-Nov-2025 06:34:22 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/34/26/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161850Z_gen_008303_supernova_g4_detsim_20251114T085929Z_reco.root"
26-Nov-2025 06:34:22 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 06:34:22 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
Begin processing the 31st record. run: 8190 subRun: 0 event: 83361 at 26-Nov-2025 06:34:22 CET
Begin processing the 32nd record. run: 8190 subRun: 0 event: 83362 at 26-Nov-2025 06:34:25 CET
Begin processing the 33rd record. run: 8190 subRun: 0 event: 83363 at 26-Nov-2025 06:34:28 CET
Begin processing the 34th record. run: 8190 subRun: 0 event: 83364 at 26-Nov-2025 06:34:30 CET
Begin processing the 35th record. run: 8190 subRun: 0 event: 83365 at 26-Nov-2025 06:34:33 CET
Begin processing the 36th record. run: 8190 subRun: 0 event: 83366 at 26-Nov-2025 06:34:35 CET
Begin processing the 37th record. run: 8190 subRun: 0 event: 83367 at 26-Nov-2025 06:34:37 CET
Begin processing the 38th record. run: 8190 subRun: 0 event: 83368 at 26-Nov-2025 06:34:42 CET
Begin processing the 39th record. run: 8190 subRun: 0 event: 83369 at 26-Nov-2025 06:34:44 CET
Begin processing the 40th record. run: 8190 subRun: 0 event: 83370 at 26-Nov-2025 06:34:46 CET
26-Nov-2025 06:34:50 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 06:34:50 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 06:34:53 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
Begin processing the 41st record. run: 8190 subRun: 0 event: 83381 at 26-Nov-2025 06:34:53 CET
Begin processing the 42nd record. run: 8190 subRun: 0 event: 83382 at 26-Nov-2025 06:34:56 CET
Begin processing the 43rd record. run: 8190 subRun: 0 event: 83383 at 26-Nov-2025 06:34:59 CET
Begin processing the 44th record. run: 8190 subRun: 0 event: 83384 at 26-Nov-2025 06:35:01 CET
Begin processing the 45th record. run: 8190 subRun: 0 event: 83385 at 26-Nov-2025 06:35:03 CET
Begin processing the 46th record. run: 8190 subRun: 0 event: 83386 at 26-Nov-2025 06:35:05 CET
Begin processing the 47th record. run: 8190 subRun: 0 event: 83387 at 26-Nov-2025 06:35:09 CET
Begin processing the 48th record. run: 8190 subRun: 0 event: 83388 at 26-Nov-2025 06:35:11 CET
Begin processing the 49th record. run: 8190 subRun: 0 event: 83389 at 26-Nov-2025 06:35:14 CET
Begin processing the 50th record. run: 8190 subRun: 0 event: 83390 at 26-Nov-2025 06:35:16 CET
26-Nov-2025 06:35:18 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 06:35:18 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 06:35:18 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
Begin processing the 51st record. run: 8190 subRun: 0 event: 83471 at 26-Nov-2025 06:35:19 CET
Begin processing the 52nd record. run: 8190 subRun: 0 event: 83472 at 26-Nov-2025 06:35:21 CET
Begin processing the 53rd record. run: 8190 subRun: 0 event: 83473 at 26-Nov-2025 06:35:24 CET
Begin processing the 54th record. run: 8190 subRun: 0 event: 83474 at 26-Nov-2025 06:35:28 CET
Begin processing the 55th record. run: 8190 subRun: 0 event: 83475 at 26-Nov-2025 06:35:31 CET
Begin processing the 56th record. run: 8190 subRun: 0 event: 83476 at 26-Nov-2025 06:35:35 CET
Begin processing the 57th record. run: 8190 subRun: 0 event: 83477 at 26-Nov-2025 06:35:37 CET
Begin processing the 58th record. run: 8190 subRun: 0 event: 83478 at 26-Nov-2025 06:35:40 CET
Begin processing the 59th record. run: 8190 subRun: 0 event: 83479 at 26-Nov-2025 06:35:41 CET
Begin processing the 60th record. run: 8190 subRun: 0 event: 83480 at 26-Nov-2025 06:35:44 CET
26-Nov-2025 06:35:46 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 06:35:46 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
26-Nov-2025 06:35:46 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
Begin processing the 61st record. run: 8190 subRun: 0 event: 83491 at 26-Nov-2025 06:35:47 CET
Begin processing the 62nd record. run: 8190 subRun: 0 event: 83492 at 26-Nov-2025 06:35:49 CET
Begin processing the 63rd record. run: 8190 subRun: 0 event: 83493 at 26-Nov-2025 06:35:51 CET
Begin processing the 64th record. run: 8190 subRun: 0 event: 83494 at 26-Nov-2025 06:35:53 CET
Begin processing the 65th record. run: 8190 subRun: 0 event: 83495 at 26-Nov-2025 06:35:58 CET
R__unzip: error -5 in inflate (zlib)
R__unzip: error -5 in inflate (zlib)
26-Nov-2025 06:35:59 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.00469032      1.96682       4.40997       1.85121      0.697874        65     
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00321363    0.00455881    0.00960426     0.0043498    0.00118618       65     
end_path:triggerAna:TriggerAnaTree     0.966378       1.99265       4.40544       1.85247      0.658346        64     
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1788.81 MB
  Peak resident set size usage (VmHWM): 1034.35 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Nov-2025 06:36:01 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- FileReadError BEGIN
        ---- FatalRootError BEGIN
          Fatal Root Error: TBasket::ReadBasketBuffers
          fNbytes = 9005153, fKeylen = 119, fObjlen = 20163250, noutot = 0, nout=0, nin=7525521, nbuf=16777215
          ROOT severity: 3000
        ---- FatalRootError END
        
        The above exception was thrown while processing module TriggerAnaTree/triggerAna run: 8190 subRun: 0 event: 83495
      ---- FileReadError END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.

ERROR: Merging failed with return code 1
failed to merge pass1_CERN_000037.json 1
justIN time: 2026-02-08 08:47:49 UTC       justIN version: 01.06.00