justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.184@dunegpschedd01.fnal.gov

Jobsub ID303652.184@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce514
Last heartbeat2026-02-04 00:33:59
From worker nodeHostnameb9p08p6650.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:36:21
Input filesvd-protodune:np02vd_raw_run042433_0599_df-s03-d1_dw_0_20260203T222947.hdf5
JobscriptExit code0
Real time55m (3313s)
CPU time53m (3184s = 96%)
Max RSS bytes3432316928 (3273 MiB)
Outputting started2026-02-04 00:31:35
Output files
Finished2026-02-04 00:33:59
Saved logsjustin-logs:303652.184-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ng up space point ptr... 0x514a0920, 0x66fa0c50
Caught exception in looking up space point ptr... 0x66fa0c50, 0x6de02640
Caught exception in looking up space point ptr... 0x6de02640, 0x592e4560
Caught exception in looking up space point ptr... 0x3bc8b620, 0x515c5bf0
Caught exception in looking up space point ptr... 0x515c5bf0, 0x6688eeb0
Caught exception in looking up space point ptr... 0x45c5aa70, 0x607d1420
Caught exception in looking up space point ptr... 0x607d1420, 0x45c5aa70
Caught exception in looking up space point ptr... 0x2b91de50, 0x2b91dfa0
Caught exception in looking up space point ptr... 0x2b91dfa0, 0x2b91dd50
Caught exception in looking up space point ptr... 0x4a024110, 0x4a023df0
Caught exception in looking up space point ptr... 0x4a023df0, 0x4a023940
Caught exception in looking up space point ptr... 0x10b25ad0, 0x1325f2d0
Caught exception in looking up space point ptr... 0x1325f2d0, 0x3661dae0
Caught exception in looking up space point ptr... 0x56b93160, 0x56b92df0
Caught exception in looking up space point ptr... 0x56b92df0, 0x56b91fa0
Caught exception in looking up space point ptr... 0x37508370, 0x627c2200
Caught exception in looking up space point ptr... 0x627c2200, 0x37508370
Caught exception in looking up space point ptr... 0x631925b0, 0x66f46b10
Caught exception in looking up space point ptr... 0x66f46b10, 0x631925b0
Caught exception in looking up space point ptr... 0x36713800, 0x53255e00
Caught exception in looking up space point ptr... 0x53255e00, 0x36713440
Caught exception in looking up space point ptr... 0x623d57d0, 0x536a5170
Caught exception in looking up space point ptr... 0x536a5170, 0x623d57d0
Caught exception in looking up space point ptr... 0x19908010, 0x5aa5a320
Caught exception in looking up space point ptr... 0x5aa5a320, 0x4dee43a0
Caught exception in looking up space point ptr... 0x5591a6a0, 0x359b7b50
Caught exception in looking up space point ptr... 0x359b7b50, 0x359ba4d0
Caught exception in looking up space point ptr... 0x5582a440, 0x52724d00
Caught exception in looking up space point ptr... 0x52724d00, 0x5582a440
Caught exception in looking up space point ptr... 0x447b1320, 0x447b14b0
Caught exception in looking up space point ptr... 0x447b14b0, 0x447b17d0
Caught exception in looking up space point ptr... 0xe13c4f0, 0xe1276d0
Caught exception in looking up space point ptr... 0xe1276d0, 0xe13c5e0
Caught exception in looking up space point ptr... 0x1904e880, 0xcf6a860
Caught exception in looking up space point ptr... 0x67cc7ab0, 0x1904e880
Caught exception in looking up space point ptr... 0x623b3590, 0xf1958d0
Caught exception in looking up space point ptr... 0xf1958d0, 0x623b3590
Caught exception in looking up space point ptr... 0x623b3590, 0x560c8c90
Caught exception in looking up space point ptr... 0x560c8c90, 0x623b3590
Caught exception in looking up space point ptr... 0x52309b10, 0x5230c540
Caught exception in looking up space point ptr... 0x5230c540, 0x5230bce0
Caught exception in looking up space point ptr... 0x406fa660, 0x406fa950
Caught exception in looking up space point ptr... 0x406fa950, 0x406fa7b0
Caught exception in looking up space point ptr... 0x406fa7b0, 0x406fa350
Caught exception in looking up space point ptr... 0x406fa350, 0x406fa4c0
Caught exception in looking up space point ptr... 0x4a3049c0, 0x4a304850
Caught exception in looking up space point ptr... 0x4a304850, 0x49200750
Caught exception in looking up space point ptr... 0xb31efa0, 0xb31f4f0
Caught exception in looking up space point ptr... 0xb31f4f0, 0xb31efa0
Caught exception in looking up space point ptr... 0xef4b510, 0xef4b360
Caught exception in looking up space point ptr... 0xef4b360, 0xef4b830
Caught exception in looking up space point ptr... 0x574a4850, 0x59adaa60
Caught exception in looking up space point ptr... 0x59adaa60, 0x574a4850
Caught exception in looking up space point ptr... 0x474c5e40, 0x474c6690
Caught exception in looking up space point ptr... 0x474c6690, 0x474c5e40
Caught exception in looking up space point ptr... 0x92cfb80, 0x92cfe90
Caught exception in looking up space point ptr... 0x92cfe90, 0x92d0020
Caught exception in looking up space point ptr... 0x5f288140, 0x5f287fb0
Caught exception in looking up space point ptr... 0x5f287fb0, 0x5f287e20
Caught exception in looking up space point ptr... 0x634f6970, 0x5a65f870
Caught exception in looking up space point ptr... 0x5a65f870, 0x30252310
Caught exception in looking up space point ptr... 0x32bc3b60, 0x32bc3a10
Caught exception in looking up space point ptr... 0x32bc3a10, 0x32bc3b60
Caught exception in looking up space point ptr... 0xe124f90, 0xe11fa50
Caught exception in looking up space point ptr... 0xe11fa50, 0xe124f90
Caught exception in looking up space point ptr... 0x447b61d0, 0x6af27b50
Caught exception in looking up space point ptr... 0x6af27b50, 0x447b61d0
++++>>>> total num hits: 91945, num free: 88117
04-Feb-2026 01:31:27 CET  Closed output file "np02vd_raw_run042433_0599_df-s03-d1_dw_0_20260203T222947_reco_stage0_20260204T003127_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         148.372       170.743       191.105       172.427       12.213         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.346e-05    0.000109157   0.000397995   8.6791e-05    7.30382e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                7.0838        7.58641       8.1469        7.62997      0.333088        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0107738     0.0235057     0.0701245     0.0156825     0.0169202       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000436556   0.000618268   0.00106535    0.000554686   0.00016605       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000441346   0.000608599    0.0010798    0.000523456   0.000187624      19     
produce:beamevent:BeamEvent                      7.4361e-05    0.000145622   0.000920651   0.000108501   0.000184135      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.44218       3.06208       3.53296       2.98737      0.277826        19     
produce:ophit:OpHitFinder                         0.0320677     0.0400976     0.0472332     0.0399276    0.00433438       19     
produce:wclsdatavd:WireCellToolkit                 108.89        122.698       135.819       123.95        8.09298        19     
produce:gaushit:GausHitFinder                      1.05642       1.59727       2.24257       1.55834      0.403235        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000293074   0.000421474   0.000739489   0.000404085   0.000106835      19     
produce:reco3d:SpacePointSolver                    17.828        25.9479       34.6615       26.5872       4.98597        19     
produce:hitpdune:DisambigFromSpacePoints          0.135004      0.242725      0.357855      0.240233      0.0633109       19     
produce:cluster3d:Cluster3D                        1.59876       3.51568       5.69332       3.4572       0.981999        19     
[art]:TriggerResults:TriggerResultInserter       2.8071e-05    4.30931e-05   0.000118562    3.644e-05    2.24853e-05      19     
end_path:out1:RootOutput                          3.54e-06     8.47168e-06   4.0201e-05     7.13e-06     7.6583e-06       19     
end_path:out1:RootOutput(write)                    4.62048       5.98896       7.44919       6.21804      0.777853        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5389.26 MB
  Peak resident set size usage (VmHWM): 3432.32 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0599_df-s03-d1_dw_0_20260203T222947_reco_stage0_20260204T003127_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0599_df-s03-d1_dw_0_20260203T222947_reco_stage0_20260204T003127_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0599_df-s03-d1_dw_0_20260203T222947_reco_stage0_20260204T003127_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161795.0,
    "core.end_time": 1770165088.0,
    "core.events": [
      68295,
      68301,
      68307,
      68313,
      68319,
      68325,
      68331,
      68337,
      68343,
      68349,
      68355,
      68361,
      68367,
      68373,
      68379,
      68385,
      68391,
      68397,
      68403
    ],
    "core.event_count": 19,
    "core.first_event_number": 68295,
    "core.last_event_number": 68403,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0599_df-s03-d1_dw_0_20260203T222947.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:42:54 UTC       justIN version: 01.06.00