justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.130@dunegpschedd01.fnal.gov

Jobsub ID303651.130@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 01:33:08
From worker nodeHostnamewn-snel-013.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:22:57
Input filesvd-protodune:np02vd_raw_run042421_0086_df-s04-d0_dw_0_20260203T141731.hdf5
JobscriptExit code0
Real time2h (7651s)
CPU time1h (4167s = 54%)
Max RSS bytes2735054848 (2608 MiB)
Outputting started2026-02-04 01:30:30
Output files
Finished2026-02-04 01:33:08
Saved logsjustin-logs:303651.130-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ht exception in looking up space point ptr... 0x50973490, 0x50973300
Caught exception in looking up space point ptr... 0x6613e3c0, 0x6613e1e0
Caught exception in looking up space point ptr... 0x6613e590, 0x6613e3c0
Caught exception in looking up space point ptr... 0x53801a00, 0x576e5c90
Caught exception in looking up space point ptr... 0x51a70f30, 0x53801a00
Caught exception in looking up space point ptr... 0x45e9b380, 0x4ba7a640
Caught exception in looking up space point ptr... 0x45e9b6a0, 0x45e9b380
Caught exception in looking up space point ptr... 0x36d1cdc0, 0x4d7d1e80
Caught exception in looking up space point ptr... 0x49bf2a40, 0x36d1cdc0
Caught exception in looking up space point ptr... 0x33ca5b90, 0x33ca9ec0
Caught exception in looking up space point ptr... 0x33ca9ec0, 0x36d0ba50
Caught exception in looking up space point ptr... 0x3f382610, 0x6990dcb0
Caught exception in looking up space point ptr... 0x3f3827a0, 0x3f382610
Caught exception in looking up space point ptr... 0x487368f0, 0x48728a40
Caught exception in looking up space point ptr... 0x48736b30, 0x487368f0
Caught exception in looking up space point ptr... 0x3c3308c0, 0x3c32dd00
Caught exception in looking up space point ptr... 0x3c3332f0, 0x3c3308c0
Caught exception in looking up space point ptr... 0x680a6360, 0x5f9bdae0
Caught exception in looking up space point ptr... 0x5f9bdae0, 0x680a61d0
Caught exception in looking up space point ptr... 0x58408c50, 0x583f9ed0
Caught exception in looking up space point ptr... 0x583f9ed0, 0x583eafc0
Caught exception in looking up space point ptr... 0x367b2a30, 0x35a0b900
Caught exception in looking up space point ptr... 0x35a0b900, 0x367b2a30
Caught exception in looking up space point ptr... 0x543e5530, 0x543d3100
Caught exception in looking up space point ptr... 0x543d3100, 0x40f07de0
Caught exception in looking up space point ptr... 0x2f346690, 0x58cc9610
Caught exception in looking up space point ptr... 0x58cc9610, 0x2f346690
Caught exception in looking up space point ptr... 0x43476bf0, 0x48500dd0
Caught exception in looking up space point ptr... 0x48500dd0, 0x1828e210
Caught exception in looking up space point ptr... 0x48ca6d30, 0x14128ca0
Caught exception in looking up space point ptr... 0x48ca6a20, 0x48ca6d30
Caught exception in looking up space point ptr... 0x6c05e060, 0x6c04e1b0
Caught exception in looking up space point ptr... 0x6c04e1b0, 0x46af98d0
Caught exception in looking up space point ptr... 0x3ebb63d0, 0x508ce5e0
Caught exception in looking up space point ptr... 0x508ce5e0, 0x3ebb63d0
Caught exception in looking up space point ptr... 0x68050620, 0x2ced1100
Caught exception in looking up space point ptr... 0x2ced1100, 0x68050620
Caught exception in looking up space point ptr... 0x611ede00, 0x611edf90
Caught exception in looking up space point ptr... 0x611edf90, 0x4684d020
Caught exception in looking up space point ptr... 0x5aebf930, 0x5aebf6d0
Caught exception in looking up space point ptr... 0x5aebf6d0, 0x5aebf930
Caught exception in looking up space point ptr... 0x5aebf2a0, 0x5aebf040
Caught exception in looking up space point ptr... 0x5aebf040, 0x5aebf2a0
Caught exception in looking up space point ptr... 0x2d7a7500, 0x2f56a480
Caught exception in looking up space point ptr... 0x2f56a480, 0x57ea3730
Caught exception in looking up space point ptr... 0xf33f180, 0x18196020
Caught exception in looking up space point ptr... 0x18196020, 0xf33f180
Caught exception in looking up space point ptr... 0x19b8dd50, 0x563e5660
Caught exception in looking up space point ptr... 0x563e5660, 0x19b8dd50
Caught exception in looking up space point ptr... 0x50d66140, 0x50b3b050
Caught exception in looking up space point ptr... 0x50b3b050, 0x4ff80a10
Caught exception in looking up space point ptr... 0x162cab00, 0x48916660
Caught exception in looking up space point ptr... 0x48916660, 0x162cab00
Caught exception in looking up space point ptr... 0x2b1c5280, 0x3e4b8a90
Caught exception in looking up space point ptr... 0x3e4b8a90, 0x2b1c5280
Caught exception in looking up space point ptr... 0x140516b0, 0x3bd5d150
Caught exception in looking up space point ptr... 0x3bd5d150, 0x140516b0
Caught exception in looking up space point ptr... 0x3ea76d60, 0x46fc5a30
Caught exception in looking up space point ptr... 0x46fc5a30, 0x3ea76d60
Caught exception in looking up space point ptr... 0x1432f1e0, 0x2d66b240
Caught exception in looking up space point ptr... 0x2d66b240, 0x2d66b340
Caught exception in looking up space point ptr... 0x42f579f0, 0x4788eaa0
Caught exception in looking up space point ptr... 0x4788eaa0, 0x42f579f0
Caught exception in looking up space point ptr... 0xdf2aa50, 0x3e8a1a00
Caught exception in looking up space point ptr... 0x3e8a1a00, 0xdf2aa50
++++>>>> total num hits: 319755, num free: 313939
04-Feb-2026 02:30:20 CET  Closed output file "np02vd_raw_run042421_0086_df-s04-d0_dw_0_20260203T141731_reco_stage0_20260204T013020_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         204.975       279.514       381.727        264.6        45.7223        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.4428e-05    0.000107696   0.000318227   0.000100558   4.16625e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                77.8367       120.076       185.64        116.393       27.293         27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.579586      0.748599       1.20482      0.713444      0.136491        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0171997     0.0410211     0.274651      0.0180257     0.066026        27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0855262     0.0928633     0.178487      0.0879228     0.0176314       27     
produce:beamevent:BeamEvent                      9.9546e-05    0.000178443   0.00140624    0.000130956   0.000241578      27     
produce:pdvddaphne:DAPHNEReaderPDVD                10.7257       15.5602       22.3913       14.9979       3.16752        27     
produce:ophit:OpHitFinder                         0.0319038     0.0424163     0.0508046     0.0435284    0.00527048       27     
produce:wclsdatavd:WireCellToolkit                 71.8254       84.4541       99.995        85.0666       7.43734        27     
produce:gaushit:GausHitFinder                     0.959339       1.4632        2.35693       1.41847      0.356596        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000336992   0.000559862   0.00099758    0.000535374   0.000125571      27     
produce:reco3d:SpacePointSolver                    5.05066       11.8375       22.1328       11.1776       4.31327        27     
produce:hitpdune:DisambigFromSpacePoints          0.0877359     0.237956      0.446012      0.253823      0.0782873       27     
produce:cluster3d:Cluster3D                        1.70574       39.0901       65.9721       49.3456       23.6742        27     
[art]:TriggerResults:TriggerResultInserter       2.4366e-05    3.6045e-05    0.000126226   3.0106e-05    1.9732e-05       27     
end_path:out1:RootOutput                          4.047e-06    6.43233e-06   3.1759e-05     4.689e-06    5.59653e-06      27     
end_path:out1:RootOutput(write)                    4.0472        5.83243       7.21477       5.71205      0.881204        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4697.43 MB
  Peak resident set size usage (VmHWM): 2735.05 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0086_df-s04-d0_dw_0_20260203T141731_reco_stage0_20260204T013020_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0086_df-s04-d0_dw_0_20260203T141731_reco_stage0_20260204T013020_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0086_df-s04-d0_dw_0_20260203T141731_reco_stage0_20260204T013020_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160991.0,
    "core.end_time": 1770168622.0,
    "core.events": [
      13990,
      13996,
      14002,
      14008,
      14014,
      14020,
      14026,
      14032,
      14038,
      14044,
      14050,
      14056,
      14062,
      14068,
      14074,
      14080,
      14086,
      14092,
      14098,
      14104,
      14110,
      14116,
      14122,
      14128,
      14134,
      14140,
      14146
    ],
    "core.event_count": 27,
    "core.first_event_number": 13990,
    "core.last_event_number": 14146,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0086_df-s04-d0_dw_0_20260203T141731.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:07 UTC       justIN version: 01.06.00