justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258139.28@dunegpschedd01.fnal.gov

Jobsub ID258139.28@dunegpschedd01.fnal.gov
Workflow ID10771
Stage ID1
User nameschellma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 02:28:04
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2025-11-26 04:10:33
From worker nodeHostnameb9p06p8389.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-26 04:06:00
Input filesmonte-carlo-010771-000037
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-26 04:10:33
Saved logsjustin-logs:258139.28-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

3023 at 26-Nov-2025 05:07:22 CET
Begin processing the 24th record. run: 8190 subRun: 0 event: 83024 at 26-Nov-2025 05:07:25 CET
Begin processing the 25th record. run: 8190 subRun: 0 event: 83025 at 26-Nov-2025 05:07:27 CET
Begin processing the 26th record. run: 8190 subRun: 0 event: 83026 at 26-Nov-2025 05:07:30 CET
Begin processing the 27th record. run: 8190 subRun: 0 event: 83027 at 26-Nov-2025 05:07:34 CET
Begin processing the 28th record. run: 8190 subRun: 0 event: 83028 at 26-Nov-2025 05:07:38 CET
Begin processing the 29th record. run: 8190 subRun: 0 event: 83029 at 26-Nov-2025 05:07:42 CET
Begin processing the 30th record. run: 8190 subRun: 0 event: 83030 at 26-Nov-2025 05:07:44 CET
26-Nov-2025 05:07:47 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/34/26/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161850Z_gen_008303_supernova_g4_detsim_20251114T085929Z_reco.root"
26-Nov-2025 05:07:47 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 05:07:47 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
Begin processing the 31st record. run: 8190 subRun: 0 event: 83361 at 26-Nov-2025 05:07:48 CET
Begin processing the 32nd record. run: 8190 subRun: 0 event: 83362 at 26-Nov-2025 05:07:50 CET
Begin processing the 33rd record. run: 8190 subRun: 0 event: 83363 at 26-Nov-2025 05:07:53 CET
Begin processing the 34th record. run: 8190 subRun: 0 event: 83364 at 26-Nov-2025 05:07:56 CET
Begin processing the 35th record. run: 8190 subRun: 0 event: 83365 at 26-Nov-2025 05:07:58 CET
Begin processing the 36th record. run: 8190 subRun: 0 event: 83366 at 26-Nov-2025 05:08:01 CET
Begin processing the 37th record. run: 8190 subRun: 0 event: 83367 at 26-Nov-2025 05:08:03 CET
Begin processing the 38th record. run: 8190 subRun: 0 event: 83368 at 26-Nov-2025 05:08:08 CET
Begin processing the 39th record. run: 8190 subRun: 0 event: 83369 at 26-Nov-2025 05:08:11 CET
Begin processing the 40th record. run: 8190 subRun: 0 event: 83370 at 26-Nov-2025 05:08:13 CET
26-Nov-2025 05:08:17 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/e3/50/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008337_supernova_g4_detsim_20251113T194738Z_reco.root"
26-Nov-2025 05:08:17 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 05:08:18 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
Begin processing the 41st record. run: 8190 subRun: 0 event: 83381 at 26-Nov-2025 05:08:19 CET
Begin processing the 42nd record. run: 8190 subRun: 0 event: 83382 at 26-Nov-2025 05:08:27 CET
Begin processing the 43rd record. run: 8190 subRun: 0 event: 83383 at 26-Nov-2025 05:08:34 CET
Begin processing the 44th record. run: 8190 subRun: 0 event: 83384 at 26-Nov-2025 05:08:40 CET
Begin processing the 45th record. run: 8190 subRun: 0 event: 83385 at 26-Nov-2025 05:08:45 CET
Begin processing the 46th record. run: 8190 subRun: 0 event: 83386 at 26-Nov-2025 05:08:51 CET
Begin processing the 47th record. run: 8190 subRun: 0 event: 83387 at 26-Nov-2025 05:08:59 CET
Begin processing the 48th record. run: 8190 subRun: 0 event: 83388 at 26-Nov-2025 05:09:05 CET
Begin processing the 49th record. run: 8190 subRun: 0 event: 83389 at 26-Nov-2025 05:09:12 CET
Begin processing the 50th record. run: 8190 subRun: 0 event: 83390 at 26-Nov-2025 05:09:19 CET
26-Nov-2025 05:09:29 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/99/ea/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161902Z_gen_008339_supernova_g4_detsim_20251114T095701Z_reco.root"
26-Nov-2025 05:09:29 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 05:09:29 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
Begin processing the 51st record. run: 8190 subRun: 0 event: 83471 at 26-Nov-2025 05:09:29 CET
Begin processing the 52nd record. run: 8190 subRun: 0 event: 83472 at 26-Nov-2025 05:09:31 CET
Begin processing the 53rd record. run: 8190 subRun: 0 event: 83473 at 26-Nov-2025 05:09:33 CET
Begin processing the 54th record. run: 8190 subRun: 0 event: 83474 at 26-Nov-2025 05:09:37 CET
Begin processing the 55th record. run: 8190 subRun: 0 event: 83475 at 26-Nov-2025 05:09:40 CET
Begin processing the 56th record. run: 8190 subRun: 0 event: 83476 at 26-Nov-2025 05:09:44 CET
Begin processing the 57th record. run: 8190 subRun: 0 event: 83477 at 26-Nov-2025 05:09:47 CET
Begin processing the 58th record. run: 8190 subRun: 0 event: 83478 at 26-Nov-2025 05:09:49 CET
Begin processing the 59th record. run: 8190 subRun: 0 event: 83479 at 26-Nov-2025 05:09:52 CET
Begin processing the 60th record. run: 8190 subRun: 0 event: 83480 at 26-Nov-2025 05:09:54 CET
26-Nov-2025 05:09:57 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/54/fc/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161904Z_gen_008348_supernova_g4_detsim_20251114T113251Z_reco.root"
26-Nov-2025 05:09:57 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
26-Nov-2025 05:09:57 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"
Begin processing the 61st record. run: 8190 subRun: 0 event: 83491 at 26-Nov-2025 05:09:57 CET
Begin processing the 62nd record. run: 8190 subRun: 0 event: 83492 at 26-Nov-2025 05:10:00 CET
Begin processing the 63rd record. run: 8190 subRun: 0 event: 83493 at 26-Nov-2025 05:10:03 CET
Begin processing the 64th record. run: 8190 subRun: 0 event: 83494 at 26-Nov-2025 05:10:06 CET
Begin processing the 65th record. run: 8190 subRun: 0 event: 83495 at 26-Nov-2025 05:10:10 CET
R__unzip: error -5 in inflate (zlib)
R__unzip: error -5 in inflate (zlib)
26-Nov-2025 05:10:11 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/47/7c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T161905Z_gen_008350_supernova_g4_detsim_20251114T102453Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.00468364      2.06427       4.18465       2.00417      0.736145        65     
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00339082     0.0167825     0.458372     0.00446089     0.0600176       65     
end_path:triggerAna:TriggerAnaTree      1.17998       2.07917       3.99885       2.00024      0.684623        64     
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1790 MB
  Peak resident set size usage (VmHWM): 1040.54 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Nov-2025 05:10:13 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- FileReadError BEGIN
        ---- FatalRootError BEGIN
          Fatal Root Error: TBasket::ReadBasketBuffers
          fNbytes = 9005153, fKeylen = 119, fObjlen = 20163250, noutot = 0, nout=0, nin=7525521, nbuf=16777215
          ROOT severity: 3000
        ---- FatalRootError END
        
        The above exception was thrown while processing module TriggerAnaTree/triggerAna run: 8190 subRun: 0 event: 83495
      ---- FileReadError END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.

ERROR: Merging failed with return code 1
failed to merge pass1_CERN_000037.json 1
justIN time: 2026-02-08 10:30:24 UTC       justIN version: 01.06.00