justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.151@dunegpschedd02.fnal.gov

Jobsub ID297921.151@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 01:43:48
From worker nodeHostnamecn530.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:27
Input filesvd-protodune:np02vd_raw_run042433_0117_df-s02-d1_dw_0_20260203T192636.hdf5
JobscriptExit code0
Real time1h (6247s)
CPU time1h (6007s = 96%)
Max RSS bytes5550092288 (5292 MiB)
Outputting started2026-02-04 01:40:39
Output files
Finished2026-02-04 01:43:48
Saved logsjustin-logs:297921.151-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x82d89e00, 0x2ce811d0
Caught exception in looking up space point ptr... 0x5f0cf420, 0x5f0cf290
Caught exception in looking up space point ptr... 0x5f0cf290, 0x5f0cf420
Caught exception in looking up space point ptr... 0xa140790, 0x12f26bf0
Caught exception in looking up space point ptr... 0x12f26bf0, 0xa140790
Caught exception in looking up space point ptr... 0x6ada2cc0, 0x19f29bc0
Caught exception in looking up space point ptr... 0x19f29bc0, 0x2c3a7fc0
Caught exception in looking up space point ptr... 0xa6ab55c0, 0x2f622c00
Caught exception in looking up space point ptr... 0x2f622c00, 0xa6ab55c0
Caught exception in looking up space point ptr... 0xa6ab55c0, 0x60a7fb40
Caught exception in looking up space point ptr... 0x60a7fb40, 0xa6ab55c0
Caught exception in looking up space point ptr... 0x9ece7d70, 0x66d32580
Caught exception in looking up space point ptr... 0x66d32580, 0xa71573f0
Caught exception in looking up space point ptr... 0xa71573f0, 0x71643730
Caught exception in looking up space point ptr... 0x71643730, 0x876e0620
Caught exception in looking up space point ptr... 0x876e0620, 0x9ece7d70
Caught exception in looking up space point ptr... 0x56989550, 0x5696cfe0
Caught exception in looking up space point ptr... 0x5696cfe0, 0x56989550
Caught exception in looking up space point ptr... 0x56989550, 0x569700c0
Caught exception in looking up space point ptr... 0x569700c0, 0x56989550
Caught exception in looking up space point ptr... 0x56989550, 0x5697e4e0
Caught exception in looking up space point ptr... 0x5697e4e0, 0x56989550
Caught exception in looking up space point ptr... 0x59eaf940, 0x59eaf810
Caught exception in looking up space point ptr... 0x59eaf810, 0x59eaf940
Caught exception in looking up space point ptr... 0x56994770, 0x56993cc0
Caught exception in looking up space point ptr... 0x56993cc0, 0x56994770
Caught exception in looking up space point ptr... 0x56994770, 0x569842d0
Caught exception in looking up space point ptr... 0x569842d0, 0x56994770
Caught exception in looking up space point ptr... 0x56994770, 0x569966f0
Caught exception in looking up space point ptr... 0x569966f0, 0x56994770
Caught exception in looking up space point ptr... 0x56999010, 0x56999190
Caught exception in looking up space point ptr... 0x56999190, 0x56999010
Caught exception in looking up space point ptr... 0x56999010, 0x569a6b70
Caught exception in looking up space point ptr... 0x569a6b70, 0x56999010
Caught exception in looking up space point ptr... 0x56999010, 0x56996380
Caught exception in looking up space point ptr... 0x56996380, 0x56999010
Caught exception in looking up space point ptr... 0x56984dc0, 0x56983990
Caught exception in looking up space point ptr... 0x56983990, 0x56984dc0
Caught exception in looking up space point ptr... 0x56984dc0, 0x569ba100
Caught exception in looking up space point ptr... 0x569ba100, 0x56984dc0
Caught exception in looking up space point ptr... 0x56984dc0, 0x56993f20
Caught exception in looking up space point ptr... 0x56993f20, 0x56984dc0
Caught exception in looking up space point ptr... 0x4d918140, 0xf731f20
Caught exception in looking up space point ptr... 0xf731f20, 0x8e19f4a0
Caught exception in looking up space point ptr... 0x3ea28cc0, 0x153938c0
Caught exception in looking up space point ptr... 0x153938c0, 0x189606f0
Caught exception in looking up space point ptr... 0x5e4afbc0, 0x5e4ab8d0
Caught exception in looking up space point ptr... 0x5e4ab8d0, 0x5e3d9e70
Caught exception in looking up space point ptr... 0x9ec49780, 0x9e7cbf60
Caught exception in looking up space point ptr... 0x9e7cbf60, 0x680518d0
Caught exception in looking up space point ptr... 0x8ee39fb0, 0x7166ccf0
Caught exception in looking up space point ptr... 0x7166ccf0, 0x8ee39fb0
Caught exception in looking up space point ptr... 0x85a29d40, 0x35846c20
Caught exception in looking up space point ptr... 0x35846c20, 0x39dcaea0
Caught exception in looking up space point ptr... 0x6e273ec0, 0x7a7d7610
Caught exception in looking up space point ptr... 0x7a7d7610, 0x3359d7e0
Caught exception in looking up space point ptr... 0x8a3a2980, 0x6d348ce0
Caught exception in looking up space point ptr... 0x6d348ce0, 0x75f3dda0
Caught exception in looking up space point ptr... 0x4b9c27e0, 0x6e273ec0
Caught exception in looking up space point ptr... 0x3399c710, 0x3399c8a0
Caught exception in looking up space point ptr... 0x3399c8a0, 0x3399c710
Caught exception in looking up space point ptr... 0xa6dc5600, 0x6e4e6db0
Caught exception in looking up space point ptr... 0x6e4e70d0, 0xa6dc5600
Caught exception in looking up space point ptr... 0x339a4730, 0x339a48c0
Caught exception in looking up space point ptr... 0x339a48c0, 0x339a4730
Caught exception in looking up space point ptr... 0xa6eb7130, 0xa031d2b0
Caught exception in looking up space point ptr... 0xa031d2b0, 0xa6eb7130
++++>>>> total num hits: 2061765, num free: 2005462
04-Feb-2026 01:40:24 GMT  Closed output file "np02vd_raw_run042433_0117_df-s02-d1_dw_0_20260203T192636_reco_stage0_20260204T014024_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         211.966       323.653       827.08        292.44        146.799        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.9041e-05    0.000150846   0.000290565   0.000133467   5.88672e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.6316       16.4692       31.2547       15.5636       3.57096        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.493598      0.529431      0.644971      0.517507      0.0373223       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0166556     0.0180111     0.0239915     0.0169262    0.00229109       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0166502     0.0169783     0.0173028     0.0169301    0.000178533      19     
produce:beamevent:BeamEvent                      0.00014656    0.000255132   0.00113887    0.000204644   0.00021113       19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.78174       6.55813       7.59741       6.47251      0.438348        19     
produce:ophit:OpHitFinder                         0.0400123     0.0541014     0.0812757     0.0506798     0.0116476       19     
produce:wclsdatavd:WireCellToolkit                 159.962       175.056       210.628       170.152       13.2068        19     
produce:gaushit:GausHitFinder                      1.38152       3.90238       17.7163       2.92554       3.63774        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000538271   0.000786848   0.00253045    0.000638004   0.000437782      19     
produce:reco3d:SpacePointSolver                    10.529        22.4586       99.6561       16.8863       19.8954        19     
produce:hitpdune:DisambigFromSpacePoints           0.20499       1.16491       12.8116      0.392005       2.79592        19     
produce:cluster3d:Cluster3D                        2.1526        88.9589       470.54        70.8887       115.465        19     
[art]:TriggerResults:TriggerResultInserter       2.8649e-05    4.73812e-05    8.78e-05     4.2293e-05    1.67296e-05      19     
end_path:out1:RootOutput                          4.307e-06    9.98463e-06   2.1886e-05     8.758e-06    4.8468e-06       19     
end_path:out1:RootOutput(write)                    5.67009       8.4035        18.5735       7.77911       3.10899        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7785.25 MB
  Peak resident set size usage (VmHWM): 5550.09 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0117_df-s02-d1_dw_0_20260203T192636_reco_stage0_20260204T014024_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0117_df-s02-d1_dw_0_20260203T192636_reco_stage0_20260204T014024_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0117_df-s02-d1_dw_0_20260203T192636_reco_stage0_20260204T014024_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163000.0,
    "core.end_time": 1770169228.0,
    "core.events": [
      13345,
      13351,
      13357,
      13363,
      13369,
      13375,
      13381,
      13387,
      13393,
      13399,
      13405,
      13411,
      13417,
      13423,
      13429,
      13435,
      13441,
      13447,
      13453
    ],
    "core.event_count": 19,
    "core.first_event_number": 13345,
    "core.last_event_number": 13453,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0117_df-s02-d1_dw_0_20260203T192636.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:27:08 UTC       justIN version: 01.06.00