justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.49@dunegpschedd02.fnal.gov

Jobsub ID297923.49@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 02:58:07
From worker nodeHostnamewn1907251.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:37
Input filesvd-protodune:np02vd_raw_run042433_0276_df-s04-d1_dw_0_20260203T202701.hdf5
JobscriptExit code0
Real time2h (10540s)
CPU time1h (5698s = 54%)
Max RSS bytes3353759744 (3198 MiB)
Outputting started2026-02-04 02:55:19
Output files
Finished2026-02-04 02:58:07
Saved logsjustin-logs:297923.49-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g up space point ptr... 0x157584e0, 0x15765b10
Caught exception in looking up space point ptr... 0x66995fa0, 0x5d7b5590
Caught exception in looking up space point ptr... 0x5d7b5590, 0x66996140
Caught exception in looking up space point ptr... 0xc25e3a0, 0xc25ea90
Caught exception in looking up space point ptr... 0xc25ea90, 0xc25e3a0
Caught exception in looking up space point ptr... 0x12b36880, 0x12b36a60
Caught exception in looking up space point ptr... 0x12b36a60, 0x12b36dd0
Caught exception in looking up space point ptr... 0x12b10690, 0x12b180a0
Caught exception in looking up space point ptr... 0x12b180a0, 0x12b182f0
Caught exception in looking up space point ptr... 0xc2c9140, 0xc2c1540
Caught exception in looking up space point ptr... 0xc2c1540, 0xc2c8c20
Caught exception in looking up space point ptr... 0x5db9eae0, 0x5db9e950
Caught exception in looking up space point ptr... 0x5db9e950, 0x5db9ec70
Caught exception in looking up space point ptr... 0x37d94470, 0x37d94150
Caught exception in looking up space point ptr... 0x37d94150, 0x37d942e0
Caught exception in looking up space point ptr... 0x6a169cc0, 0x6a153550
Caught exception in looking up space point ptr... 0x5ea8f0d0, 0x6a169cc0
Caught exception in looking up space point ptr... 0x322c4eb0, 0x3fa56080
Caught exception in looking up space point ptr... 0x3fa56080, 0x322c4eb0
Caught exception in looking up space point ptr... 0x48455820, 0x48448c40
Caught exception in looking up space point ptr... 0x48448c40, 0x48455820
Caught exception in looking up space point ptr... 0x187c4a30, 0x187c7440
Caught exception in looking up space point ptr... 0x187c7440, 0x187c4a30
Caught exception in looking up space point ptr... 0x5cb05620, 0x13e6d200
Caught exception in looking up space point ptr... 0x13e6d200, 0x6a656b30
Caught exception in looking up space point ptr... 0x3807b990, 0x52600d50
Caught exception in looking up space point ptr... 0x52600d50, 0x380928c0
Caught exception in looking up space point ptr... 0x58afa390, 0x58afa6b0
Caught exception in looking up space point ptr... 0x58afa6b0, 0x58afa390
Caught exception in looking up space point ptr... 0x3ddbbae0, 0x3ddbbc70
Caught exception in looking up space point ptr... 0x3ddbbc70, 0x3ddbbae0
Caught exception in looking up space point ptr... 0x16530cb0, 0x16fdeb90
Caught exception in looking up space point ptr... 0x16fdeb90, 0x16530cb0
Caught exception in looking up space point ptr... 0x127d0260, 0x127d0a80
Caught exception in looking up space point ptr... 0x127d0a80, 0x127d0260
Caught exception in looking up space point ptr... 0x5d9d7660, 0x5d9d78b0
Caught exception in looking up space point ptr... 0x5d9d78b0, 0x5d9d7760
Caught exception in looking up space point ptr... 0x12b26a90, 0x12b26c70
Caught exception in looking up space point ptr... 0x12b26c70, 0x12b25120
Caught exception in looking up space point ptr... 0x15dd67f0, 0x15dd64d0
Caught exception in looking up space point ptr... 0x15dd64d0, 0x15dd67f0
Caught exception in looking up space point ptr... 0x15dd67f0, 0x15dd62f0
Caught exception in looking up space point ptr... 0x15dd62f0, 0x15dd67f0
Caught exception in looking up space point ptr... 0xf9b3230, 0xf9b9210
Caught exception in looking up space point ptr... 0xf9b9210, 0xf9b3230
Caught exception in looking up space point ptr... 0xf9b3230, 0xf9b9930
Caught exception in looking up space point ptr... 0xf9b9930, 0xf9b3230
Caught exception in looking up space point ptr... 0xd599680, 0x197771f0
Caught exception in looking up space point ptr... 0x197771f0, 0xd599680
Caught exception in looking up space point ptr... 0x67dc9b70, 0x5e971d20
Caught exception in looking up space point ptr... 0x5e971d20, 0x5e96ef30
Caught exception in looking up space point ptr... 0xa418000, 0x30a0e2d0
Caught exception in looking up space point ptr... 0x30a0e2d0, 0x30a14960
Caught exception in looking up space point ptr... 0xd80b320, 0x5eed7330
Caught exception in looking up space point ptr... 0x5eed7330, 0x41a10c80
Caught exception in looking up space point ptr... 0xd4c2ff0, 0xd4c2e60
Caught exception in looking up space point ptr... 0xd4c2e60, 0xd4c2ff0
Caught exception in looking up space point ptr... 0x674f1040, 0x674f0490
Caught exception in looking up space point ptr... 0x674f0490, 0x674f1040
Caught exception in looking up space point ptr... 0x85696070, 0x3aca3c90
Caught exception in looking up space point ptr... 0x3aca3c90, 0x4de6e740
Caught exception in looking up space point ptr... 0x6b0c5950, 0x6b0c2a70
Caught exception in looking up space point ptr... 0x6b0c2a70, 0x6b0c5950
Caught exception in looking up space point ptr... 0x6b0d91c0, 0x6b0c55b0
Caught exception in looking up space point ptr... 0x6b0c55b0, 0x6b0c2ba0
Caught exception in looking up space point ptr... 0x3ddd0680, 0x1189b460
Caught exception in looking up space point ptr... 0x1189b460, 0x3ddd0680
++++>>>> total num hits: 131833, num free: 126396
04-Feb-2026 02:54:59 GMT  Closed output file "np02vd_raw_run042433_0276_df-s04-d1_dw_0_20260203T202701_reco_stage0_20260204T025459_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         440.954       546.425       648.441       562.946       58.219         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000107496   0.00024629    0.000479918   0.000247153    9.757e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                155.863         237         338.63        231.326       41.9266        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.718564       1.13511       1.60127       1.09166      0.263821        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.022374      0.0308383     0.120632      0.0227324     0.0220221       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0222153     0.0231725     0.0335544     0.0225706    0.00245822       19     
produce:beamevent:BeamEvent                      0.000175279   0.000303314   0.00127252    0.000240069   0.000236754      19     
produce:pdvddaphne:DAPHNEReaderPDVD                18.1075       24.6245       28.6778       25.6779       3.29289        19     
produce:ophit:OpHitFinder                         0.0497454     0.0697011     0.0828304     0.0722618    0.00941802       19     
produce:wclsdatavd:WireCellToolkit                 189.103       228.604       272.291       224.618       23.6377        19     
produce:gaushit:GausHitFinder                      1.87428       2.99957       3.70415       2.95181      0.449248        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000643849   0.00104198    0.00268352    0.000958202   0.000421721      19     
produce:reco3d:SpacePointSolver                    24.0381       35.6643       49.8756       35.3096       7.11028        19     
produce:hitpdune:DisambigFromSpacePoints          0.347784      0.521132      0.716971      0.548089      0.112857        19     
produce:cluster3d:Cluster3D                        3.23808       5.44915       8.60725       5.4244        1.5526         19     
[art]:TriggerResults:TriggerResultInserter       3.4648e-05    6.35687e-05   0.000124093   5.3526e-05    2.19195e-05      19     
end_path:out1:RootOutput                          9.829e-06    1.64893e-05   3.6277e-05     1.386e-05    6.44941e-06      19     
end_path:out1:RootOutput(write)                    7.98162       10.2045       13.0712       10.1901       1.19946        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5358.53 MB
  Peak resident set size usage (VmHWM): 3353.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0276_df-s04-d1_dw_0_20260203T202701_reco_stage0_20260204T025459_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0276_df-s04-d1_dw_0_20260203T202701_reco_stage0_20260204T025459_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0276_df-s04-d1_dw_0_20260203T202701_reco_stage0_20260204T025459_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163192.0,
    "core.end_time": 1770173709.0,
    "core.events": [
      31475,
      31481,
      31487,
      31493,
      31499,
      31505,
      31511,
      31517,
      31523,
      31529,
      31535,
      31541,
      31547,
      31553,
      31559,
      31565,
      31571,
      31577,
      31583
    ],
    "core.event_count": 19,
    "core.first_event_number": 31475,
    "core.last_event_number": 31583,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0276_df-s04-d1_dw_0_20260203T202701.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:19:13 UTC       justIN version: 01.06.00