justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253468.0@dunegpschedd02.fnal.gov

Jobsub ID253468.0@dunegpschedd02.fnal.gov
Workflow ID10813
Stage ID1
User nameschellma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 23:40:01
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2025-11-26 23:59:56
From worker nodeHostnameb9p19p7414.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-26 23:43:42
Input filesmonte-carlo-010813-000049
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-26 23:59:56
Saved logsjustin-logs:253468.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

188 subRun: 0 event: 31814 at 27-Nov-2025 00:56:51 CET
Begin processing the 185th record. run: 8188 subRun: 0 event: 31815 at 27-Nov-2025 00:56:55 CET
Begin processing the 186th record. run: 8188 subRun: 0 event: 31816 at 27-Nov-2025 00:56:59 CET
Begin processing the 187th record. run: 8188 subRun: 0 event: 31817 at 27-Nov-2025 00:57:02 CET
Begin processing the 188th record. run: 8188 subRun: 0 event: 31818 at 27-Nov-2025 00:57:07 CET
Begin processing the 189th record. run: 8188 subRun: 0 event: 31819 at 27-Nov-2025 00:57:11 CET
Begin processing the 190th record. run: 8188 subRun: 0 event: 31820 at 27-Nov-2025 00:57:14 CET
27-Nov-2025 00:57:17 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/5c/f4/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123634Z_gen_003182_supernova_g4_detsim_20251113T161752Z_reco.root"
27-Nov-2025 00:57:17 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/4b/ec/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123635Z_gen_003183_supernova_g4_detsim_20251114T093219Z_reco.root"
27-Nov-2025 00:57:17 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/4b/ec/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123635Z_gen_003183_supernova_g4_detsim_20251114T093219Z_reco.root"
Begin processing the 191st record. run: 8188 subRun: 0 event: 31821 at 27-Nov-2025 00:57:18 CET
Begin processing the 192nd record. run: 8188 subRun: 0 event: 31822 at 27-Nov-2025 00:57:21 CET
Begin processing the 193rd record. run: 8188 subRun: 0 event: 31823 at 27-Nov-2025 00:57:26 CET
Begin processing the 194th record. run: 8188 subRun: 0 event: 31824 at 27-Nov-2025 00:57:29 CET
Begin processing the 195th record. run: 8188 subRun: 0 event: 31825 at 27-Nov-2025 00:57:33 CET
Begin processing the 196th record. run: 8188 subRun: 0 event: 31826 at 27-Nov-2025 00:57:37 CET
Begin processing the 197th record. run: 8188 subRun: 0 event: 31827 at 27-Nov-2025 00:57:40 CET
Begin processing the 198th record. run: 8188 subRun: 0 event: 31828 at 27-Nov-2025 00:57:43 CET
Begin processing the 199th record. run: 8188 subRun: 0 event: 31829 at 27-Nov-2025 00:57:46 CET
Begin processing the 200th record. run: 8188 subRun: 0 event: 31830 at 27-Nov-2025 00:57:50 CET
27-Nov-2025 00:57:54 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/4b/ec/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123635Z_gen_003183_supernova_g4_detsim_20251114T093219Z_reco.root"
27-Nov-2025 00:57:54 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/f1/e0/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123636Z_gen_003185_supernova_g4_detsim_20251113T194953Z_reco.root"
27-Nov-2025 00:57:54 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/f1/e0/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123636Z_gen_003185_supernova_g4_detsim_20251113T194953Z_reco.root"
Begin processing the 201st record. run: 8188 subRun: 0 event: 31841 at 27-Nov-2025 00:57:55 CET
Begin processing the 202nd record. run: 8188 subRun: 0 event: 31842 at 27-Nov-2025 00:57:58 CET
Begin processing the 203rd record. run: 8188 subRun: 0 event: 31843 at 27-Nov-2025 00:58:05 CET
Begin processing the 204th record. run: 8188 subRun: 0 event: 31844 at 27-Nov-2025 00:58:08 CET
Begin processing the 205th record. run: 8188 subRun: 0 event: 31845 at 27-Nov-2025 00:58:11 CET
Begin processing the 206th record. run: 8188 subRun: 0 event: 31846 at 27-Nov-2025 00:58:14 CET
Begin processing the 207th record. run: 8188 subRun: 0 event: 31847 at 27-Nov-2025 00:58:17 CET
Begin processing the 208th record. run: 8188 subRun: 0 event: 31848 at 27-Nov-2025 00:58:20 CET
Begin processing the 209th record. run: 8188 subRun: 0 event: 31849 at 27-Nov-2025 00:58:24 CET
Begin processing the 210th record. run: 8188 subRun: 0 event: 31850 at 27-Nov-2025 00:58:27 CET
27-Nov-2025 00:58:31 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/f1/e0/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123636Z_gen_003185_supernova_g4_detsim_20251113T194953Z_reco.root"
27-Nov-2025 00:58:31 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/a2/74/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123647Z_gen_003194_supernova_g4_detsim_20251114T104850Z_reco.root"
27-Nov-2025 00:58:31 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/a2/74/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123647Z_gen_003194_supernova_g4_detsim_20251114T104850Z_reco.root"
Begin processing the 211th record. run: 8188 subRun: 0 event: 31931 at 27-Nov-2025 00:58:31 CET
Begin processing the 212th record. run: 8188 subRun: 0 event: 31932 at 27-Nov-2025 00:58:36 CET
Begin processing the 213th record. run: 8188 subRun: 0 event: 31933 at 27-Nov-2025 00:58:39 CET
Begin processing the 214th record. run: 8188 subRun: 0 event: 31934 at 27-Nov-2025 00:58:43 CET
Begin processing the 215th record. run: 8188 subRun: 0 event: 31935 at 27-Nov-2025 00:58:48 CET
Begin processing the 216th record. run: 8188 subRun: 0 event: 31936 at 27-Nov-2025 00:58:51 CET
Begin processing the 217th record. run: 8188 subRun: 0 event: 31937 at 27-Nov-2025 00:58:55 CET
Begin processing the 218th record. run: 8188 subRun: 0 event: 31938 at 27-Nov-2025 00:58:58 CET
Begin processing the 219th record. run: 8188 subRun: 0 event: 31939 at 27-Nov-2025 00:59:01 CET
Begin processing the 220th record. run: 8188 subRun: 0 event: 31940 at 27-Nov-2025 00:59:04 CET
27-Nov-2025 00:59:10 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/a2/74/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123647Z_gen_003194_supernova_g4_detsim_20251114T104850Z_reco.root"
27-Nov-2025 00:59:10 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/0f/47/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123651Z_gen_003198_supernova_g4_detsim_20251114T102733Z_reco.root"
27-Nov-2025 00:59:10 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/0f/47/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123651Z_gen_003198_supernova_g4_detsim_20251114T102733Z_reco.root"
Begin processing the 221st record. run: 8188 subRun: 0 event: 31971 at 27-Nov-2025 00:59:10 CET
Begin processing the 222nd record. run: 8188 subRun: 0 event: 31972 at 27-Nov-2025 00:59:14 CET
Begin processing the 223rd record. run: 8188 subRun: 0 event: 31973 at 27-Nov-2025 00:59:17 CET
Begin processing the 224th record. run: 8188 subRun: 0 event: 31974 at 27-Nov-2025 00:59:21 CET
Begin processing the 225th record. run: 8188 subRun: 0 event: 31975 at 27-Nov-2025 00:59:27 CET
Begin processing the 226th record. run: 8188 subRun: 0 event: 31976 at 27-Nov-2025 00:59:31 CET
27-Nov-2025 00:59:32 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/0f/47/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T123651Z_gen_003198_supernova_g4_detsim_20251114T102733Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.00455513      2.83219       6.3785        2.67015       1.00395        226    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00352662    0.00515673     0.0168931    0.00463101    0.00197377       226    
end_path:triggerAna:TriggerAnaTree      1.34847       2.83933       6.37363       2.6659       0.988026        225    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2185.32 MB
  Peak resident set size usage (VmHWM): 1448.74 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 27-Nov-2025 00:59:36 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- FileReadError BEGIN
        ---- FatalRootError BEGIN
          Fatal Root Error: TBasket::ReadBasketBuffers
          basket: has fNevBuf=0 but fEntryOffset=0, pos=108438429, len=114006, fNbytes=0, fObjlen=0, trying to repair
          ROOT severity: 2000
        ---- FatalRootError END
        
        The above exception was thrown while processing module TriggerAnaTree/triggerAna run: 8188 subRun: 0 event: 31976
      ---- FileReadError END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.

ERROR: Merging failed with return code 1
failed to merge pass1_CERN_000049.json 1
justIN time: 2026-02-08 07:15:27 UTC       justIN version: 01.06.00