justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297924.122@dunegpschedd02.fnal.gov

Jobsub ID297924.122@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:02:25
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce05
Last heartbeat2026-02-04 01:31:53
From worker nodeHostnamedune001-1844959.0-lcg2537.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:13:30
Input filesvd-protodune:np02vd_raw_run042433_0530_df-s04-d0_dw_0_20260203T220333.hdf5
JobscriptExit code0
Real time1h (4572s)
CPU time1h (4217s = 92%)
Max RSS bytes3366670336 (3210 MiB)
Outputting started2026-02-04 01:29:43
Output files
Finished2026-02-04 01:31:53
Saved logsjustin-logs:297924.122-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0xadf7ab00, 0xadf7a0c0
Caught exception in looking up space point ptr... 0xadf7a0c0, 0xadf7ab00
Caught exception in looking up space point ptr... 0x3837ea20, 0x3837eb90
Caught exception in looking up space point ptr... 0x3837eb90, 0x3837ea20
Caught exception in looking up space point ptr... 0x38ad84e0, 0x38acb790
Caught exception in looking up space point ptr... 0x38acb790, 0x38acb9e0
Caught exception in looking up space point ptr... 0x5d6c6370, 0x5d6c64c0
Caught exception in looking up space point ptr... 0x5d6c64c0, 0x5d6c6370
Caught exception in looking up space point ptr... 0xae046260, 0x885bc1e0
Caught exception in looking up space point ptr... 0x885bc1e0, 0x885bc430
Caught exception in looking up space point ptr... 0x885bc430, 0xae00d710
Caught exception in looking up space point ptr... 0x2ee78670, 0x2ee85050
Caught exception in looking up space point ptr... 0x2ee852a0, 0x2ee78670
Caught exception in looking up space point ptr... 0x88ba80e0, 0x6b0c2440
Caught exception in looking up space point ptr... 0x3af150b0, 0x88ba80e0
Caught exception in looking up space point ptr... 0xab55a4e0, 0x59b3bb10
Caught exception in looking up space point ptr... 0x59b3bb10, 0xab55a4e0
Caught exception in looking up space point ptr... 0x39782530, 0x397829e0
Caught exception in looking up space point ptr... 0x397829e0, 0x39782850
Caught exception in looking up space point ptr... 0x39782850, 0x39782b70
Caught exception in looking up space point ptr... 0x55d07c30, 0x55d07920
Caught exception in looking up space point ptr... 0x55cff400, 0x55d07c30
Caught exception in looking up space point ptr... 0x80b5b9b0, 0x80b5ffb0
Caught exception in looking up space point ptr... 0x80b5ffb0, 0x80b5dde0
Caught exception in looking up space point ptr... 0x80b5dde0, 0x80b5fa30
Caught exception in looking up space point ptr... 0x80b5fa30, 0x80b5b6e0
Caught exception in looking up space point ptr... 0xa6e44b20, 0xa6e426d0
Caught exception in looking up space point ptr... 0xa6e426d0, 0xa6e4a950
Caught exception in looking up space point ptr... 0x3a51b000, 0x9f4c1b10
Caught exception in looking up space point ptr... 0x9f4c1b10, 0x59dd2e20
Caught exception in looking up space point ptr... 0x686f3700, 0x5d6c3510
Caught exception in looking up space point ptr... 0x5d6c3510, 0x686f7d60
Caught exception in looking up space point ptr... 0x686f7d60, 0x5d6c37b0
Caught exception in looking up space point ptr... 0x885dbe60, 0x885db580
Caught exception in looking up space point ptr... 0x885db580, 0x885dbe60
Caught exception in looking up space point ptr... 0x5f7fbe70, 0x9f4bf110
Caught exception in looking up space point ptr... 0x9f4bf110, 0x8c536770
Caught exception in looking up space point ptr... 0x6001e9a0, 0x7e30aad0
Caught exception in looking up space point ptr... 0x7e30aad0, 0x600218a0
Caught exception in looking up space point ptr... 0xa7867cd0, 0xa7867b80
Caught exception in looking up space point ptr... 0xa7867b80, 0xa7867cd0
Caught exception in looking up space point ptr... 0x3ea1b470, 0x40e027b0
Caught exception in looking up space point ptr... 0x40e027b0, 0x3ea1b470
Caught exception in looking up space point ptr... 0x3ea1b470, 0x3ea1ba40
Caught exception in looking up space point ptr... 0x3ea1ba40, 0x3ea1b470
Caught exception in looking up space point ptr... 0x70b6c2c0, 0x6cd0b160
Caught exception in looking up space point ptr... 0x6cd0b160, 0x3c6705f0
Caught exception in looking up space point ptr... 0x885d97f0, 0x885d96c0
Caught exception in looking up space point ptr... 0x885d96c0, 0x885d97f0
Caught exception in looking up space point ptr... 0x622ad7b0, 0x617d4350
Caught exception in looking up space point ptr... 0x617d4350, 0xaba4f710
Caught exception in looking up space point ptr... 0x3d2dd8a0, 0x3d2ddaa0
Caught exception in looking up space point ptr... 0x3d2ddaa0, 0x3d2dd8a0
Caught exception in looking up space point ptr... 0x2ee5dd80, 0x2ee5e710
Caught exception in looking up space point ptr... 0x2ee5e710, 0x2ee5dd80
Caught exception in looking up space point ptr... 0x885e3e70, 0x885e4b70
Caught exception in looking up space point ptr... 0x885e4b70, 0x885e3e70
Caught exception in looking up space point ptr... 0x3945cb80, 0x3945c980
Caught exception in looking up space point ptr... 0x3945c980, 0x3945cb80
Caught exception in looking up space point ptr... 0xa786e070, 0xa786e5d0
Caught exception in looking up space point ptr... 0xa786e300, 0xa786e070
Caught exception in looking up space point ptr... 0x60027070, 0x60026f10
Caught exception in looking up space point ptr... 0x60026f10, 0x60027070
Caught exception in looking up space point ptr... 0x80b73d70, 0x80b73ad0
Caught exception in looking up space point ptr... 0x80b74210, 0x80b73d70
Caught exception in looking up space point ptr... 0xab2abe80, 0xab29f2b0
Caught exception in looking up space point ptr... 0xab29f2b0, 0xab2abe80
++++>>>> total num hits: 102695, num free: 99380
04-Feb-2026 01:29:30 UTC  Closed output file "np02vd_raw_run042433_0530_df-s04-d0_dw_0_20260203T220333_reco_stage0_20260204T012930_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         214.737       234.358       262.511       232.548       10.6431        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000112832   0.000150272   0.000343717   0.000133713   5.59758e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                19.7258       20.8583       23.6449       20.4134       1.13717        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.715028      0.773492      0.889373      0.768166      0.0420159       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.023519      0.0351466     0.103939      0.0270753     0.0203822       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0233154     0.023544      0.0243566     0.0235174    0.000215196      19     
produce:beamevent:BeamEvent                      0.000134143   0.000324533   0.00181234    0.000227355   0.00036082       19     
produce:pdvddaphne:DAPHNEReaderPDVD                4.07746       4.58437       5.08505       4.65771      0.317153        19     
produce:ophit:OpHitFinder                         0.043185      0.0570982     0.101105      0.0546215     0.0114753       19     
produce:wclsdatavd:WireCellToolkit                 153.859       165.363       187.33        162.479       8.70068        19     
produce:gaushit:GausHitFinder                      1.38903       2.03919       2.9544        1.92918      0.448936        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000408229   0.000657706   0.00193625    0.000559613   0.000342476      19     
produce:reco3d:SpacePointSolver                    18.6113       28.8433       36.0579       29.4621       4.76792        19     
produce:hitpdune:DisambigFromSpacePoints          0.203518      0.316247      0.423875      0.307191      0.0681876       19     
produce:cluster3d:Cluster3D                        1.94056       3.77448       5.2149        3.84537       1.06527        19     
[art]:TriggerResults:TriggerResultInserter       3.2801e-05    6.87146e-05   0.000248535   4.7631e-05    5.49727e-05      19     
end_path:out1:RootOutput                          7.881e-06    3.75346e-05   0.000509352    9.62e-06     0.000111329      19     
end_path:out1:RootOutput(write)                    6.06804       7.6081        9.29009       7.57694       1.0496         19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5551.96 MB
  Peak resident set size usage (VmHWM): 3366.67 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0530_df-s04-d0_dw_0_20260203T220333_reco_stage0_20260204T012930_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0530_df-s04-d0_dw_0_20260203T220333_reco_stage0_20260204T012930_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0530_df-s04-d0_dw_0_20260203T220333_reco_stage0_20260204T012930_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770164026.0,
    "core.end_time": 1770168573.0,
    "core.events": [
      60430,
      60436,
      60442,
      60448,
      60454,
      60460,
      60466,
      60472,
      60478,
      60484,
      60490,
      60496,
      60502,
      60508,
      60514,
      60520,
      60526,
      60532,
      60538
    ],
    "core.event_count": 19,
    "core.first_event_number": 60430,
    "core.last_event_number": 60538,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0530_df-s04-d0_dw_0_20260203T220333.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:16 UTC       justIN version: 01.06.00