justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 252687.0@dunegpschedd02.fnal.gov

Jobsub ID252687.0@dunegpschedd02.fnal.gov
Workflow ID10771
Stage ID1
User nameschellma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 05:25:06
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2025-11-26 05:29:46
From worker nodeHostnameb9p04p6494.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-26 05:25:42
Input filesmonte-carlo-010771-000037
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-26 05:29:46
Saved logsjustin-logs:252687.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

3 at 26-Nov-2025 06:26:58 CET
Begin processing the 24th record. run: 8190 subRun: 0 event: 83024 at 26-Nov-2025 06:27:00 CET
Begin processing the 25th record. run: 8190 subRun: 0 event: 83025 at 26-Nov-2025 06:27:03 CET
Begin processing the 26th record. run: 8190 subRun: 0 event: 83026 at 26-Nov-2025 06:27:05 CET
Begin processing the 27th record. run: 8190 subRun: 0 event: 83027 at 26-Nov-2025 06:27:09 CET
Begin processing the 28th record. run: 8190 subRun: 0 event: 83028 at 26-Nov-2025 06:27:12 CET
Begin processing the 29th record. run: 8190 subRun: 0 event: 83029 at 26-Nov-2025 06:27:16 CET
Begin processing the 30th record. run: 8190 subRun: 0 event: 83030 at 26-Nov-2025 06:27:18 CET
26-Nov-2025 06:27:21 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/34/26/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161850Z_gen_008303_supernova_g4_detsim_20251114T085929Z_reco.root"
26-Nov-2025 06:27:21 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 06:27:21 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
Begin processing the 31st record. run: 8190 subRun: 0 event: 83361 at 26-Nov-2025 06:27:21 CET
Begin processing the 32nd record. run: 8190 subRun: 0 event: 83362 at 26-Nov-2025 06:27:23 CET
Begin processing the 33rd record. run: 8190 subRun: 0 event: 83363 at 26-Nov-2025 06:27:26 CET
Begin processing the 34th record. run: 8190 subRun: 0 event: 83364 at 26-Nov-2025 06:27:28 CET
Begin processing the 35th record. run: 8190 subRun: 0 event: 83365 at 26-Nov-2025 06:27:30 CET
Begin processing the 36th record. run: 8190 subRun: 0 event: 83366 at 26-Nov-2025 06:27:33 CET
Begin processing the 37th record. run: 8190 subRun: 0 event: 83367 at 26-Nov-2025 06:27:35 CET
Begin processing the 38th record. run: 8190 subRun: 0 event: 83368 at 26-Nov-2025 06:27:38 CET
Begin processing the 39th record. run: 8190 subRun: 0 event: 83369 at 26-Nov-2025 06:27:41 CET
Begin processing the 40th record. run: 8190 subRun: 0 event: 83370 at 26-Nov-2025 06:27:43 CET
26-Nov-2025 06:27:46 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 06:27:46 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 06:27:47 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
Begin processing the 41st record. run: 8190 subRun: 0 event: 83381 at 26-Nov-2025 06:27:49 CET
Begin processing the 42nd record. run: 8190 subRun: 0 event: 83382 at 26-Nov-2025 06:27:52 CET
Begin processing the 43rd record. run: 8190 subRun: 0 event: 83383 at 26-Nov-2025 06:27:59 CET
Begin processing the 44th record. run: 8190 subRun: 0 event: 83384 at 26-Nov-2025 06:28:01 CET
Begin processing the 45th record. run: 8190 subRun: 0 event: 83385 at 26-Nov-2025 06:28:04 CET
Begin processing the 46th record. run: 8190 subRun: 0 event: 83386 at 26-Nov-2025 06:28:10 CET
Begin processing the 47th record. run: 8190 subRun: 0 event: 83387 at 26-Nov-2025 06:28:20 CET
Begin processing the 48th record. run: 8190 subRun: 0 event: 83388 at 26-Nov-2025 06:28:28 CET
Begin processing the 49th record. run: 8190 subRun: 0 event: 83389 at 26-Nov-2025 06:28:33 CET
Begin processing the 50th record. run: 8190 subRun: 0 event: 83390 at 26-Nov-2025 06:28:37 CET
26-Nov-2025 06:28:44 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 06:28:44 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 06:28:45 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
Begin processing the 51st record. run: 8190 subRun: 0 event: 83471 at 26-Nov-2025 06:28:45 CET
Begin processing the 52nd record. run: 8190 subRun: 0 event: 83472 at 26-Nov-2025 06:28:47 CET
Begin processing the 53rd record. run: 8190 subRun: 0 event: 83473 at 26-Nov-2025 06:28:50 CET
Begin processing the 54th record. run: 8190 subRun: 0 event: 83474 at 26-Nov-2025 06:28:54 CET
Begin processing the 55th record. run: 8190 subRun: 0 event: 83475 at 26-Nov-2025 06:28:56 CET
Begin processing the 56th record. run: 8190 subRun: 0 event: 83476 at 26-Nov-2025 06:28:59 CET
Begin processing the 57th record. run: 8190 subRun: 0 event: 83477 at 26-Nov-2025 06:29:02 CET
Begin processing the 58th record. run: 8190 subRun: 0 event: 83478 at 26-Nov-2025 06:29:05 CET
Begin processing the 59th record. run: 8190 subRun: 0 event: 83479 at 26-Nov-2025 06:29:07 CET
Begin processing the 60th record. run: 8190 subRun: 0 event: 83480 at 26-Nov-2025 06:29:09 CET
26-Nov-2025 06:29:11 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 06:29:11 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
26-Nov-2025 06:29:11 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
Begin processing the 61st record. run: 8190 subRun: 0 event: 83491 at 26-Nov-2025 06:29:12 CET
Begin processing the 62nd record. run: 8190 subRun: 0 event: 83492 at 26-Nov-2025 06:29:14 CET
Begin processing the 63rd record. run: 8190 subRun: 0 event: 83493 at 26-Nov-2025 06:29:16 CET
Begin processing the 64th record. run: 8190 subRun: 0 event: 83494 at 26-Nov-2025 06:29:19 CET
Begin processing the 65th record. run: 8190 subRun: 0 event: 83495 at 26-Nov-2025 06:29:23 CET
R__unzip: error -5 in inflate (zlib)
R__unzip: error -5 in inflate (zlib)
26-Nov-2025 06:29:24 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.00389554      2.19064       8.35424       1.8768        1.33402        65     
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00336218     0.0225613     0.641661     0.00416073     0.102302        65     
end_path:triggerAna:TriggerAnaTree      1.0112        2.20166       8.3488        1.87033       1.29976        64     
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1779.59 MB
  Peak resident set size usage (VmHWM): 1022.55 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Nov-2025 06:29:25 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- FileReadError BEGIN
        ---- FatalRootError BEGIN
          Fatal Root Error: TBasket::ReadBasketBuffers
          fNbytes = 9005153, fKeylen = 119, fObjlen = 20163250, noutot = 0, nout=0, nin=7525521, nbuf=16777215
          ROOT severity: 3000
        ---- FatalRootError END
        
        The above exception was thrown while processing module TriggerAnaTree/triggerAna run: 8190 subRun: 0 event: 83495
      ---- FileReadError END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.

ERROR: Merging failed with return code 1
failed to merge pass1_CERN_000037.json 1
justIN time: 2026-02-08 08:47:48 UTC       justIN version: 01.06.00