justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.75@dunegpschedd02.fnal.gov

Jobsub ID297923.75@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 01:24:32
From worker nodeHostnamecn053.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:37
Input filesvd-protodune:np02vd_raw_run042433_0009_df-s04-d0_dw_0_20260203T184531.hdf5
JobscriptExit code0
Real time1h (4870s)
CPU time1h (4581s = 94%)
Max RSS bytes3576504320 (3410 MiB)
Outputting started2026-02-04 01:21:05
Output files
Finished2026-02-04 01:24:32
Saved logsjustin-logs:297923.75-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ooking up space point ptr... 0x159e3d30, 0x159e41c0
Caught exception in looking up space point ptr... 0x159e41c0, 0x159e3d30
Caught exception in looking up space point ptr... 0x159e3d30, 0x159e3ea0
Caught exception in looking up space point ptr... 0x159e3ea0, 0x159e3d30
Caught exception in looking up space point ptr... 0x53d59bd0, 0x3d9342c0
Caught exception in looking up space point ptr... 0x417716f0, 0x53d59bd0
Caught exception in looking up space point ptr... 0x5c2a2120, 0x36748d30
Caught exception in looking up space point ptr... 0x36748d30, 0x5c2a2120
Caught exception in looking up space point ptr... 0x572c2af0, 0x572d5200
Caught exception in looking up space point ptr... 0x572c2c80, 0x572c2af0
Caught exception in looking up space point ptr... 0x82dc350, 0x1359d560
Caught exception in looking up space point ptr... 0x1359d560, 0x1359d6f0
Caught exception in looking up space point ptr... 0x80141010, 0x62f45d00
Caught exception in looking up space point ptr... 0x62f45d00, 0x801243f0
Caught exception in looking up space point ptr... 0x70f98fd0, 0x70f98e40
Caught exception in looking up space point ptr... 0x70f98e40, 0x62f51d20
Caught exception in looking up space point ptr... 0x5a4015a0, 0x3b2f5230
Caught exception in looking up space point ptr... 0x5a401730, 0x5a4015a0
Caught exception in looking up space point ptr... 0x85262b0, 0x8514650
Caught exception in looking up space point ptr... 0x8514650, 0x8526120
Caught exception in looking up space point ptr... 0x55f551a0, 0x43d73710
Caught exception in looking up space point ptr... 0x617ea880, 0x55f551a0
Caught exception in looking up space point ptr... 0x5664c630, 0x5663a200
Caught exception in looking up space point ptr... 0x5665ea60, 0x5664c630
Caught exception in looking up space point ptr... 0x566547e0, 0x56642540
Caught exception in looking up space point ptr... 0x56642540, 0x56630110
Caught exception in looking up space point ptr... 0x4a3197b0, 0x4a3071f0
Caught exception in looking up space point ptr... 0x4a328d00, 0x4a3197b0
Caught exception in looking up space point ptr... 0x6b4f09b0, 0x6b4f0b50
Caught exception in looking up space point ptr... 0x6b4f0b50, 0x6b4f09b0
Caught exception in looking up space point ptr... 0x107948c0, 0x10784240
Caught exception in looking up space point ptr... 0x35435fc0, 0x107948c0
Caught exception in looking up space point ptr... 0x4b8a6880, 0x4b894db0
Caught exception in looking up space point ptr... 0x4b8b8800, 0x4b8a6880
Caught exception in looking up space point ptr... 0x13d85c60, 0x13d86150
Caught exception in looking up space point ptr... 0x13d85eb0, 0x13d85c60
Caught exception in looking up space point ptr... 0x60d5af50, 0x19030150
Caught exception in looking up space point ptr... 0x19030150, 0x3ab3a0b0
Caught exception in looking up space point ptr... 0x3ab3a0b0, 0x61cf0660
Caught exception in looking up space point ptr... 0x3d49ffe0, 0x3d49c540
Caught exception in looking up space point ptr... 0x3d49c540, 0x3d49e9e0
Caught exception in looking up space point ptr... 0x81598fa0, 0x8c7c970
Caught exception in looking up space point ptr... 0x815991f0, 0x81598fa0
Caught exception in looking up space point ptr... 0x8325ca0, 0x3ad01a70
Caught exception in looking up space point ptr... 0x3ad01a70, 0x8997da90
Caught exception in looking up space point ptr... 0x3c1c5ca0, 0x4fd70450
Caught exception in looking up space point ptr... 0x4fd70450, 0x4fd70350
Caught exception in looking up space point ptr... 0x194549c0, 0x2ab884c0
Caught exception in looking up space point ptr... 0x2ab884c0, 0x1adf74c0
Caught exception in looking up space point ptr... 0x8b59a8c0, 0x8b59abe0
Caught exception in looking up space point ptr... 0x8b59abe0, 0x8b59a710
Caught exception in looking up space point ptr... 0x8b59a710, 0x31f82e90
Caught exception in looking up space point ptr... 0x31f82e90, 0x108dbea0
Caught exception in looking up space point ptr... 0x42e4c790, 0x687d6db0
Caught exception in looking up space point ptr... 0x687d6db0, 0x42e4c790
Caught exception in looking up space point ptr... 0x10e20560, 0x15b1ac10
Caught exception in looking up space point ptr... 0x15b1ac10, 0x10e20560
Caught exception in looking up space point ptr... 0x6b230dc0, 0x8993dea0
Caught exception in looking up space point ptr... 0x6b2310e0, 0x6b230dc0
Caught exception in looking up space point ptr... 0x4ab10a90, 0x4a35e5e0
Caught exception in looking up space point ptr... 0x4a35e5e0, 0x4ab10a90
Caught exception in looking up space point ptr... 0x2f512030, 0x2f511ea0
Caught exception in looking up space point ptr... 0x2f511ea0, 0x2f512030
Caught exception in looking up space point ptr... 0x4d7d2dc0, 0x4d7d30e0
Caught exception in looking up space point ptr... 0x4d7d30e0, 0x4d7d2dc0
Caught exception in looking up space point ptr... 0x33990920, 0x33990790
Caught exception in looking up space point ptr... 0x33990790, 0x33990c40
++++>>>> total num hits: 546736, num free: 537602
04-Feb-2026 01:20:08 GMT  Closed output file "np02vd_raw_run042433_0009_df-s04-d0_dw_0_20260203T184531_reco_stage0_20260204T012008_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         180.473       246.946       323.16        244.43        37.8984        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        7.607e-05    0.000132106    0.0002681    0.00012033    4.0981e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                13.4543       13.8759       15.4275       13.7887      0.434902        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.491683      0.499383      0.513207      0.496993     0.00649504       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165688     0.016919      0.0178848     0.0168318    0.000263778      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0165996     0.016866      0.0172007     0.0168244    0.000162326      19     
produce:beamevent:BeamEvent                      0.000128041   0.000232946   0.00115503    0.000183001   0.000220127      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.48914       6.70684       8.39824       6.66138      0.938938        19     
produce:ophit:OpHitFinder                         0.0390994     0.0480711     0.0711054     0.041971      0.0108816       19     
produce:wclsdatavd:WireCellToolkit                 134.179       156.47        196.379       156.604       20.4811        19     
produce:gaushit:GausHitFinder                      1.67885       2.45129       3.69205       2.50012      0.575823        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000627164   0.000747337   0.00107311    0.000707743   0.000110647      19     
produce:reco3d:SpacePointSolver                    12.7296       16.6989       23.1624       15.6139       2.76941        19     
produce:hitpdune:DisambigFromSpacePoints          0.245546      0.368536      0.716128      0.343303      0.108444        19     
produce:cluster3d:Cluster3D                        1.30591       43.2662       83.4514       60.7952       31.7806        19     
[art]:TriggerResults:TriggerResultInserter       2.7752e-05    4.13344e-05   8.0707e-05    3.8373e-05    1.18181e-05      19     
end_path:out1:RootOutput                          6.291e-06    9.54432e-06   2.2256e-05     8.584e-06    3.48528e-06      19     
end_path:out1:RootOutput(write)                    4.62985       6.46729       9.72567       6.2389        1.36448        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5514.22 MB
  Peak resident set size usage (VmHWM): 3576.5 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0009_df-s04-d0_dw_0_20260203T184531_reco_stage0_20260204T012008_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0009_df-s04-d0_dw_0_20260203T184531_reco_stage0_20260204T012008_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0009_df-s04-d0_dw_0_20260203T184531_reco_stage0_20260204T012008_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163188.0,
    "core.end_time": 1770168015.0,
    "core.events": [
      1036,
      1042,
      1048,
      1054,
      1060,
      1066,
      1072,
      1078,
      1084,
      1090,
      1096,
      1102,
      1108,
      1114,
      1120,
      1126,
      1132,
      1138,
      1144
    ],
    "core.event_count": 19,
    "core.first_event_number": 1036,
    "core.last_event_number": 1144,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0009_df-s04-d0_dw_0_20260203T184531.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:28 UTC       justIN version: 01.06.00