justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303728.5@dunegpschedd01.fnal.gov

Jobsub ID303728.5@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:20:30
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2026-02-04 03:19:56
From worker nodeHostnamewn-sate-052.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:20:59
Input filesvd-protodune:np02vd_raw_run042421_0107_df-s04-d0_dw_0_20260203T142852.hdf5
JobscriptExit code0
Real time1h (6955s)
CPU time49m (2982s = 42%)
Max RSS bytes2591019008 (2470 MiB)
Outputting started2026-02-04 03:16:55
Output files
Finished2026-02-04 03:19:56
Saved logsjustin-logs:303728.5-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x1cb31d30, 0x1cb351c0
Caught exception in looking up space point ptr... 0x1cb351c0, 0x1cb35260
Caught exception in looking up space point ptr... 0x42abf6f0, 0x26a5d660
Caught exception in looking up space point ptr... 0x26a5d660, 0x26a5d510
Caught exception in looking up space point ptr... 0x5803bb60, 0x695702b0
Caught exception in looking up space point ptr... 0x695702b0, 0x5803bb60
Caught exception in looking up space point ptr... 0x42fd7640, 0x29c07c90
Caught exception in looking up space point ptr... 0x29c07c90, 0x29c20910
Caught exception in looking up space point ptr... 0x28177c50, 0x5b74b550
Caught exception in looking up space point ptr... 0x5b74b550, 0x28177510
Caught exception in looking up space point ptr... 0x62931a40, 0x629340c0
Caught exception in looking up space point ptr... 0x629340c0, 0x62931a40
Caught exception in looking up space point ptr... 0x42fd3130, 0x42fd0b20
Caught exception in looking up space point ptr... 0x42fd0b20, 0x42fd0c70
Caught exception in looking up space point ptr... 0x61ce0df0, 0x21d982c0
Caught exception in looking up space point ptr... 0x21d982c0, 0x61ce8e00
Caught exception in looking up space point ptr... 0x4cc2f260, 0x4321e5a0
Caught exception in looking up space point ptr... 0x4321e5a0, 0x40aaac30
Caught exception in looking up space point ptr... 0x29f22990, 0x37771c40
Caught exception in looking up space point ptr... 0x5b986360, 0x29f22990
Caught exception in looking up space point ptr... 0x3b3adc90, 0x3b3b1790
Caught exception in looking up space point ptr... 0x3b3b1790, 0x3b3adc90
Caught exception in looking up space point ptr... 0x6b44fd30, 0x43b5c290
Caught exception in looking up space point ptr... 0x43b5c290, 0x29c12df0
Caught exception in looking up space point ptr... 0x2a2601f0, 0x2a2613a0
Caught exception in looking up space point ptr... 0x2a2613a0, 0x2a2601f0
Caught exception in looking up space point ptr... 0x29c211b0, 0x29c210b0
Caught exception in looking up space point ptr... 0x29c17470, 0x29c211b0
Caught exception in looking up space point ptr... 0x694b0d70, 0x694befe0
Caught exception in looking up space point ptr... 0x694b5610, 0x694b0d70
Caught exception in looking up space point ptr... 0x6438f560, 0x3fe85630
Caught exception in looking up space point ptr... 0x3fe85630, 0x6438f560
Caught exception in looking up space point ptr... 0x41f4f460, 0x41f525a0
Caught exception in looking up space point ptr... 0x41f54d70, 0x41f51a60
Caught exception in looking up space point ptr... 0x41f51a60, 0x41f4f460
Caught exception in looking up space point ptr... 0x77fafa20, 0x77faf8d0
Caught exception in looking up space point ptr... 0x77faf8d0, 0x77fafa20
Caught exception in looking up space point ptr... 0x66642f00, 0x21833360
Caught exception in looking up space point ptr... 0x513b7090, 0x66642f00
Caught exception in looking up space point ptr... 0x5c7e0aa0, 0x5c7e11d0
Caught exception in looking up space point ptr... 0x5c7e11d0, 0x674362c0
Caught exception in looking up space point ptr... 0x6abec4c0, 0x429b2380
Caught exception in looking up space point ptr... 0x6abec220, 0x6abec4c0
Caught exception in looking up space point ptr... 0x2698ffc0, 0x509ae610
Caught exception in looking up space point ptr... 0x509ae610, 0x2698ffc0
Caught exception in looking up space point ptr... 0x63e0e2d0, 0x6ceb3ec0
Caught exception in looking up space point ptr... 0x6ceb3ec0, 0x6d0c7a80
Caught exception in looking up space point ptr... 0x1cb330c0, 0x1cb32e20
Caught exception in looking up space point ptr... 0x1cb32e20, 0x1cb330c0
Caught exception in looking up space point ptr... 0x43b60ca0, 0x43b60a50
Caught exception in looking up space point ptr... 0x43b60a50, 0x43b60df0
Caught exception in looking up space point ptr... 0x5ab7b340, 0x6e6c66c0
Caught exception in looking up space point ptr... 0x6e6c66c0, 0x5ab7b340
Caught exception in looking up space point ptr... 0x45f9f910, 0x62033fd0
Caught exception in looking up space point ptr... 0x62033fd0, 0x45f9f910
Caught exception in looking up space point ptr... 0x53ac08f0, 0x53ac0650
Caught exception in looking up space point ptr... 0x53ac0650, 0x53ac0500
Caught exception in looking up space point ptr... 0x6b051ed0, 0x238dd9c0
Caught exception in looking up space point ptr... 0x238dd9c0, 0x6b051ed0
Caught exception in looking up space point ptr... 0x4945c030, 0x54e9c2e0
Caught exception in looking up space point ptr... 0x54e9c2e0, 0x61c796f0
Caught exception in looking up space point ptr... 0x5f008930, 0x39621830
Caught exception in looking up space point ptr... 0x39621830, 0x5f008930
Caught exception in looking up space point ptr... 0x2a208820, 0x722c65c0
Caught exception in looking up space point ptr... 0x722c65c0, 0x45fe2510
++++>>>> total num hits: 71003, num free: 67936
04-Feb-2026 04:16:46 CET  Closed output file "np02vd_raw_run042421_0107_df-s04-d0_dw_0_20260203T142852_reco_stage0_20260204T031646_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         213.85        254.423       284.713       252.44        19.9351        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.1745e-05    9.62427e-05   0.000231124   7.8438e-05    3.31057e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                92.0861       133.42        167.463       133.811       19.8507        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.580974      0.785252       1.49124      0.719699       0.22442        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0172489     0.0233801     0.0592716     0.0176588     0.0107738       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0854537     0.0893381     0.104574      0.0867263     0.0060894       27     
produce:beamevent:BeamEvent                      9.4768e-05    0.000156558   0.000949655   0.00012742     0.0001565       27     
produce:pdvddaphne:DAPHNEReaderPDVD                10.8063       16.9471       21.7702       17.5181       2.66097        27     
produce:ophit:OpHitFinder                         0.0312196     0.0472256     0.115748      0.0450788     0.0148767       27     
produce:wclsdatavd:WireCellToolkit                 74.7061       79.8777       87.1146       79.3019       3.29544        27     
produce:gaushit:GausHitFinder                      0.7644        1.26312       2.00836       1.25855      0.244044        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000264748   0.000354405   0.000637188   0.000346381   6.8815e-05       27     
produce:reco3d:SpacePointSolver                    9.64417       15.0949       22.7962       13.4801       3.86645        27     
produce:hitpdune:DisambigFromSpacePoints          0.0870844     0.134746      0.235188       0.1248       0.036078        27     
produce:cluster3d:Cluster3D                       0.854377       1.98249       5.08063       1.8684       0.801351        27     
[art]:TriggerResults:TriggerResultInserter       2.7462e-05    3.59706e-05    9.547e-05    3.1529e-05    1.4717e-05       27     
end_path:out1:RootOutput                          6.252e-06    9.11237e-06   2.5889e-05     8.406e-06     3.67e-06        27     
end_path:out1:RootOutput(write)                    4.30398       4.72406       6.75713       4.58786      0.453751        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4502.38 MB
  Peak resident set size usage (VmHWM): 2591.02 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0107_df-s04-d0_dw_0_20260203T142852_reco_stage0_20260204T031646_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0107_df-s04-d0_dw_0_20260203T142852_reco_stage0_20260204T031646_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0107_df-s04-d0_dw_0_20260203T142852_reco_stage0_20260204T031646_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168069.0,
    "core.end_time": 1770175008.0,
    "core.events": [
      17398,
      17404,
      17410,
      17416,
      17422,
      17428,
      17434,
      17440,
      17446,
      17452,
      17458,
      17464,
      17470,
      17476,
      17482,
      17488,
      17494,
      17500,
      17506,
      17512,
      17518,
      17524,
      17530,
      17536,
      17542,
      17548,
      17554
    ],
    "core.event_count": 27,
    "core.first_event_number": 17398,
    "core.last_event_number": 17554,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0107_df-s04-d0_dw_0_20260203T142852.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:47:59 UTC       justIN version: 01.06.00