justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.137@dunegpschedd01.fnal.gov

Jobsub ID303653.137@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod00
Last heartbeat2026-02-04 01:10:32
From worker nodeHostnamewl02.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:46:33
Input filesvd-protodune:np02vd_raw_run042421_0150_df-s04-d0_dw_0_20260203T145208.hdf5
JobscriptExit code0
Real time1h (4835s)
CPU time1h (4241s = 87%)
Max RSS bytes2735603712 (2608 MiB)
Outputting started2026-02-04 01:07:12
Output files
Finished2026-02-04 01:10:32
Saved logsjustin-logs:303653.137-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

8550
Caught exception in looking up space point ptr... 0x4e5d8550, 0x4e5d83c0
Caught exception in looking up space point ptr... 0x4854b770, 0x4854b130
Caught exception in looking up space point ptr... 0x4854b2c0, 0x4854b770
Caught exception in looking up space point ptr... 0xeb62cc0, 0x8c2d960
Caught exception in looking up space point ptr... 0x58bca330, 0xeb62cc0
Caught exception in looking up space point ptr... 0x478c84f0, 0x478c81d0
Caught exception in looking up space point ptr... 0x478c81d0, 0x478c7d20
Caught exception in looking up space point ptr... 0x4074c330, 0x4a3e1a90
Caught exception in looking up space point ptr... 0x3b21fc90, 0x4074c330
Caught exception in looking up space point ptr... 0x2136b560, 0x60485660
Caught exception in looking up space point ptr... 0x2136b6f0, 0x2136b560
Caught exception in looking up space point ptr... 0xeb6b0b0, 0x2a7508c0
Caught exception in looking up space point ptr... 0x2a7508c0, 0x58bd0f10
Caught exception in looking up space point ptr... 0x3751c8f0, 0x5b5b9a50
Caught exception in looking up space point ptr... 0x444c2720, 0x3751c8f0
Caught exception in looking up space point ptr... 0x61e22700, 0x61e102d0
Caught exception in looking up space point ptr... 0x5ebcae40, 0x61e22700
Caught exception in looking up space point ptr... 0xf6a2b90, 0x14def370
Caught exception in looking up space point ptr... 0x14def370, 0xf6a2a00
Caught exception in looking up space point ptr... 0x21e5ada0, 0x21e49dc0
Caught exception in looking up space point ptr... 0x21e6c230, 0x21e5ada0
Caught exception in looking up space point ptr... 0x3377a810, 0x42c4dc50
Caught exception in looking up space point ptr... 0x42c4dc50, 0x42c3b500
Caught exception in looking up space point ptr... 0x4935fbc0, 0x5ebe2a40
Caught exception in looking up space point ptr... 0x5ebe2a40, 0x5ebd02a0
Caught exception in looking up space point ptr... 0x5ef13d40, 0x540ce890
Caught exception in looking up space point ptr... 0x540ce890, 0x5ef13bb0
Caught exception in looking up space point ptr... 0x44498ae0, 0x4449d130
Caught exception in looking up space point ptr... 0x444989e0, 0x44498ae0
Caught exception in looking up space point ptr... 0x253256c0, 0x25325530
Caught exception in looking up space point ptr... 0x25325530, 0x253259e0
Caught exception in looking up space point ptr... 0x3ebcd740, 0x14cfd160
Caught exception in looking up space point ptr... 0x14cfd160, 0x3ebcd740
Caught exception in looking up space point ptr... 0x3eb72e90, 0x5ad52a20
Caught exception in looking up space point ptr... 0x5ad52a20, 0x3eb72e90
Caught exception in looking up space point ptr... 0x3eb72e90, 0x4202c2a0
Caught exception in looking up space point ptr... 0x4202c2a0, 0x3eb72e90
Caught exception in looking up space point ptr... 0x2a917d80, 0x3287c9d0
Caught exception in looking up space point ptr... 0x3287c9d0, 0x36e2e630
Caught exception in looking up space point ptr... 0x2531db10, 0x28f8f1d0
Caught exception in looking up space point ptr... 0x28f8f1d0, 0x3c711ef0
Caught exception in looking up space point ptr... 0x2d7671b0, 0x21ad66a0
Caught exception in looking up space point ptr... 0x21ad66a0, 0x2d7671b0
Caught exception in looking up space point ptr... 0x47790d90, 0x121efbd0
Caught exception in looking up space point ptr... 0x4970a510, 0x47790d90
Caught exception in looking up space point ptr... 0x32b49b40, 0x2d767010
Caught exception in looking up space point ptr... 0x2d767010, 0x32b49b40
Caught exception in looking up space point ptr... 0xeea4670, 0x4bbe8cb0
Caught exception in looking up space point ptr... 0x4bbe8cb0, 0xeea4670
Caught exception in looking up space point ptr... 0x4fe920a0, 0xf487be0
Caught exception in looking up space point ptr... 0xf487be0, 0x4fe920a0
Caught exception in looking up space point ptr... 0x4e5d18c0, 0x4e5d1a50
Caught exception in looking up space point ptr... 0x4e5d1a50, 0x4e5d18c0
Caught exception in looking up space point ptr... 0x2d894840, 0x2d8948e0
Caught exception in looking up space point ptr... 0x2d8948e0, 0x2d894840
Caught exception in looking up space point ptr... 0x63026c70, 0x4e5f0b70
Caught exception in looking up space point ptr... 0x4e5f0b70, 0x63026c70
Caught exception in looking up space point ptr... 0x27cb45d0, 0x27cb4770
Caught exception in looking up space point ptr... 0x27cb4770, 0x27cb4aa0
Caught exception in looking up space point ptr... 0xd597e50, 0x53759060
Caught exception in looking up space point ptr... 0x53759060, 0xd597e50
Caught exception in looking up space point ptr... 0x2c8a5fd0, 0x2c8a5800
Caught exception in looking up space point ptr... 0x2c8a5800, 0x2c8a5fd0
Caught exception in looking up space point ptr... 0x4fc17130, 0x4fc17680
Caught exception in looking up space point ptr... 0x4fc17680, 0x4fc17130
++++>>>> total num hits: 360948, num free: 353521
04-Feb-2026 01:07:00 GMT  Closed output file "np02vd_raw_run042421_0150_df-s04-d0_dw_0_20260203T145208_reco_stage0_20260204T010700_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         122.797       174.971       222.218       177.76        26.7405        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.5962e-05    0.000136976   0.000361362   0.000102733   6.92681e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                10.7636       11.5702       14.907        11.344       0.799776        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.486952      0.512983      0.595359      0.510283      0.0206014       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0162513     0.0214909     0.0569206     0.0169159     0.0104049       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0803214     0.0832108     0.0996045     0.0816812    0.00458198       27     
produce:beamevent:BeamEvent                      9.1123e-05    0.000182094   0.00139773    0.000123474   0.000242173      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.22737       3.9749        5.20792       3.81499      0.496086        27     
produce:ophit:OpHitFinder                         0.0299056     0.046456      0.0588621     0.0472204    0.00630948       27     
produce:wclsdatavd:WireCellToolkit                 79.1984       88.7095       118.012       85.6219       9.64792        27     
produce:gaushit:GausHitFinder                      1.10949       1.74956       2.78584       1.71015      0.430818        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00028165    0.000489786   0.000842877   0.000500356   0.000121887      27     
produce:reco3d:SpacePointSolver                    5.2363        13.4522       20.6165       13.6807       4.88321        27     
produce:hitpdune:DisambigFromSpacePoints          0.135802       0.28848      0.415623      0.305727      0.0902495       27     
produce:cluster3d:Cluster3D                        2.6004        46.4432       73.2343       53.3852        21.37         27     
[art]:TriggerResults:TriggerResultInserter       2.7161e-05    5.33258e-05   0.000280959   3.7001e-05    4.95091e-05      27     
end_path:out1:RootOutput                          5.481e-06    1.15267e-05   3.0231e-05     1.188e-05    4.72629e-06      27     
end_path:out1:RootOutput(write)                    6.2617        8.09007       10.3703       7.62853       1.23849        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4648.98 MB
  Peak resident set size usage (VmHWM): 2735.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0150_df-s04-d0_dw_0_20260203T145208_reco_stage0_20260204T010700_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0150_df-s04-d0_dw_0_20260203T145208_reco_stage0_20260204T010700_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0150_df-s04-d0_dw_0_20260203T145208_reco_stage0_20260204T010700_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162406.0,
    "core.end_time": 1770167222.0,
    "core.events": [
      24370,
      24376,
      24382,
      24388,
      24394,
      24400,
      24406,
      24412,
      24418,
      24424,
      24430,
      24436,
      24442,
      24448,
      24454,
      24460,
      24466,
      24472,
      24478,
      24484,
      24490,
      24496,
      24502,
      24508,
      24514,
      24520,
      24526
    ],
    "core.event_count": 27,
    "core.first_event_number": 24370,
    "core.last_event_number": 24526,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0150_df-s04-d0_dw_0_20260203T145208.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:04:26 UTC       justIN version: 01.06.00