justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253242.73@dunegpschedd02.fnal.gov

Jobsub ID253242.73@dunegpschedd02.fnal.gov
Workflow ID10807
Stage ID1
User nameschellma@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-26 18:20:50
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2025-11-26 20:05:38
From worker nodeHostnameb9p19p4258.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-26 19:41:47
Input filesmonte-carlo-010807-000074
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-26 20:05:38
Saved logsjustin-logs:253242.73-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0 subRun: 0 event: 92794 at 26-Nov-2025 21:01:37 CET
Begin processing the 245th record. run: 8190 subRun: 0 event: 92795 at 26-Nov-2025 21:01:41 CET
Begin processing the 246th record. run: 8190 subRun: 0 event: 92796 at 26-Nov-2025 21:01:45 CET
Begin processing the 247th record. run: 8190 subRun: 0 event: 92797 at 26-Nov-2025 21:01:49 CET
Begin processing the 248th record. run: 8190 subRun: 0 event: 92798 at 26-Nov-2025 21:01:55 CET
Begin processing the 249th record. run: 8190 subRun: 0 event: 92799 at 26-Nov-2025 21:01:59 CET
Begin processing the 250th record. run: 8190 subRun: 0 event: 92800 at 26-Nov-2025 21:02:03 CET
26-Nov-2025 21:02:07 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/4a/7d/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162804Z_gen_009280_supernova_g4_detsim_20251113T142022Z_reco.root"
26-Nov-2025 21:02:07 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/18/9d/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162805Z_gen_009284_supernova_g4_detsim_20251114T110410Z_reco.root"
26-Nov-2025 21:02:07 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/18/9d/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162805Z_gen_009284_supernova_g4_detsim_20251114T110410Z_reco.root"
Begin processing the 251st record. run: 8190 subRun: 0 event: 92831 at 26-Nov-2025 21:02:08 CET
Begin processing the 252nd record. run: 8190 subRun: 0 event: 92832 at 26-Nov-2025 21:02:13 CET
Begin processing the 253rd record. run: 8190 subRun: 0 event: 92833 at 26-Nov-2025 21:02:16 CET
Begin processing the 254th record. run: 8190 subRun: 0 event: 92834 at 26-Nov-2025 21:02:20 CET
Begin processing the 255th record. run: 8190 subRun: 0 event: 92835 at 26-Nov-2025 21:02:26 CET
Begin processing the 256th record. run: 8190 subRun: 0 event: 92836 at 26-Nov-2025 21:02:31 CET
Begin processing the 257th record. run: 8190 subRun: 0 event: 92837 at 26-Nov-2025 21:02:36 CET
Begin processing the 258th record. run: 8190 subRun: 0 event: 92838 at 26-Nov-2025 21:02:41 CET
Begin processing the 259th record. run: 8190 subRun: 0 event: 92839 at 26-Nov-2025 21:02:45 CET
Begin processing the 260th record. run: 8190 subRun: 0 event: 92840 at 26-Nov-2025 21:02:48 CET
26-Nov-2025 21:02:52 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/18/9d/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162805Z_gen_009284_supernova_g4_detsim_20251114T110410Z_reco.root"
26-Nov-2025 21:02:52 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/ae/a5/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162806Z_gen_009289_supernova_g4_detsim_20251105T151040Z_reco.root"
26-Nov-2025 21:02:52 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/ae/a5/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162806Z_gen_009289_supernova_g4_detsim_20251105T151040Z_reco.root"
Begin processing the 261st record. run: 8190 subRun: 0 event: 92881 at 26-Nov-2025 21:02:53 CET
Begin processing the 262nd record. run: 8190 subRun: 0 event: 92882 at 26-Nov-2025 21:02:57 CET
Begin processing the 263rd record. run: 8190 subRun: 0 event: 92883 at 26-Nov-2025 21:03:01 CET
Begin processing the 264th record. run: 8190 subRun: 0 event: 92884 at 26-Nov-2025 21:03:04 CET
Begin processing the 265th record. run: 8190 subRun: 0 event: 92885 at 26-Nov-2025 21:03:09 CET
Begin processing the 266th record. run: 8190 subRun: 0 event: 92886 at 26-Nov-2025 21:03:15 CET
Begin processing the 267th record. run: 8190 subRun: 0 event: 92887 at 26-Nov-2025 21:03:19 CET
Begin processing the 268th record. run: 8190 subRun: 0 event: 92888 at 26-Nov-2025 21:03:25 CET
Begin processing the 269th record. run: 8190 subRun: 0 event: 92889 at 26-Nov-2025 21:03:30 CET
Begin processing the 270th record. run: 8190 subRun: 0 event: 92890 at 26-Nov-2025 21:03:34 CET
26-Nov-2025 21:03:37 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/ae/a5/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162806Z_gen_009289_supernova_g4_detsim_20251105T151040Z_reco.root"
26-Nov-2025 21:03:37 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6e/8c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162809Z_gen_009300_supernova_g4_detsim_20251114T110739Z_reco.root"
26-Nov-2025 21:03:37 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6e/8c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162809Z_gen_009300_supernova_g4_detsim_20251114T110739Z_reco.root"
Begin processing the 271st record. run: 8190 subRun: 0 event: 92991 at 26-Nov-2025 21:03:38 CET
Begin processing the 272nd record. run: 8190 subRun: 0 event: 92992 at 26-Nov-2025 21:03:42 CET
Begin processing the 273rd record. run: 8190 subRun: 0 event: 92993 at 26-Nov-2025 21:03:47 CET
Begin processing the 274th record. run: 8190 subRun: 0 event: 92994 at 26-Nov-2025 21:03:52 CET
Begin processing the 275th record. run: 8190 subRun: 0 event: 92995 at 26-Nov-2025 21:03:56 CET
Begin processing the 276th record. run: 8190 subRun: 0 event: 92996 at 26-Nov-2025 21:04:02 CET
Begin processing the 277th record. run: 8190 subRun: 0 event: 92997 at 26-Nov-2025 21:04:07 CET
Begin processing the 278th record. run: 8190 subRun: 0 event: 92998 at 26-Nov-2025 21:04:10 CET
Begin processing the 279th record. run: 8190 subRun: 0 event: 92999 at 26-Nov-2025 21:04:15 CET
Begin processing the 280th record. run: 8190 subRun: 0 event: 93000 at 26-Nov-2025 21:04:19 CET
26-Nov-2025 21:04:24 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6e/8c/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162809Z_gen_009300_supernova_g4_detsim_20251114T110739Z_reco.root"
26-Nov-2025 21:04:24 CET  Initiating request to open input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7d/c8/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162811Z_gen_009311_supernova_g4_detsim_20251114T091806Z_reco.root"
26-Nov-2025 21:04:24 CET  Opened input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7d/c8/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162811Z_gen_009311_supernova_g4_detsim_20251114T091806Z_reco.root"
Begin processing the 281st record. run: 8190 subRun: 0 event: 93101 at 26-Nov-2025 21:04:25 CET
Begin processing the 282nd record. run: 8190 subRun: 0 event: 93102 at 26-Nov-2025 21:04:30 CET
Begin processing the 283rd record. run: 8190 subRun: 0 event: 93103 at 26-Nov-2025 21:04:36 CET
Begin processing the 284th record. run: 8190 subRun: 0 event: 93104 at 26-Nov-2025 21:04:40 CET
Begin processing the 285th record. run: 8190 subRun: 0 event: 93105 at 26-Nov-2025 21:04:46 CET
Begin processing the 286th record. run: 8190 subRun: 0 event: 93106 at 26-Nov-2025 21:04:50 CET
26-Nov-2025 21:04:52 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7d/c8/prodbackground_radiological_decay0_dune10kt_1x2x6_centralAPA_20250928T162811Z_gen_009311_supernova_g4_detsim_20251114T091806Z_reco.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.00552017      3.25799       7.93396       3.05124       1.05664        286    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00365219    0.00949299     0.0522387    0.00602044     0.0071649       286    
end_path:triggerAna:TriggerAnaTree      1.85223       3.25951       7.92522       3.04566       1.04069        285    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2185.54 MB
  Peak resident set size usage (VmHWM): 1439.07 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Nov-2025 21:04:57 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- FileReadError BEGIN
        ---- FatalRootError BEGIN
          Fatal Root Error: TBasket::ReadBasketBuffers
          basket: has fNevBuf=0 but fEntryOffset=0, pos=1268626895, len=9048833, fNbytes=0, fObjlen=0, trying to repair
          ROOT severity: 2000
        ---- FatalRootError END
        
        The above exception was thrown while processing module TriggerAnaTree/triggerAna run: 8190 subRun: 0 event: 93106
      ---- FileReadError END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.

ERROR: Merging failed with return code 1
failed to merge pass1_CERN_000074.json 1
justIN time: 2026-02-08 08:53:08 UTC       justIN version: 01.06.00