justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.184@dunegpschedd01.fnal.gov

Jobsub ID303651.184@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 00:05:27
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:23:22
Input filesvd-protodune:np02vd_raw_run042421_0004_df-s04-d1_dw_0_20260203T133303.hdf5
JobscriptExit code0
Real time39m (2379s)
CPU time37m (2248s = 94%)
Max RSS bytes4001329152 (3815 MiB)
Outputting started2026-02-04 00:03:03
Output files
Finished2026-02-04 00:05:27
Saved logsjustin-logs:303651.184-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

in looking up space point ptr... 0x4af9a090, 0x345b3a20
Caught exception in looking up space point ptr... 0x345b3a20, 0x4af9a090
Caught exception in looking up space point ptr... 0x103b5f30, 0x103b5da0
Caught exception in looking up space point ptr... 0x5fd092a0, 0x103b5f30
Caught exception in looking up space point ptr... 0x5fd1dbb0, 0x5fd1dd40
Caught exception in looking up space point ptr... 0x5fd1dd40, 0x28358350
Caught exception in looking up space point ptr... 0x34338620, 0x343387b0
Caught exception in looking up space point ptr... 0x343387b0, 0x433faa80
Caught exception in looking up space point ptr... 0x436efa50, 0x2e7bffb0
Caught exception in looking up space point ptr... 0x2e7bffb0, 0x53335730
Caught exception in looking up space point ptr... 0x53335730, 0x16e83f00
Caught exception in looking up space point ptr... 0x16e83f00, 0x16e83d60
Caught exception in looking up space point ptr... 0x4b07c9f0, 0x5bdbfdd0
Caught exception in looking up space point ptr... 0x5bdbfdd0, 0x4b07c9f0
Caught exception in looking up space point ptr... 0x454dc440, 0x454dbf90
Caught exception in looking up space point ptr... 0x454dbf90, 0x454dc440
Caught exception in looking up space point ptr... 0x454dc440, 0x454dbe00
Caught exception in looking up space point ptr... 0x454dbe00, 0x454dc440
Caught exception in looking up space point ptr... 0x454dc440, 0x454dc120
Caught exception in looking up space point ptr... 0x454dc120, 0x454dc440
Caught exception in looking up space point ptr... 0x454dc440, 0x454dbc20
Caught exception in looking up space point ptr... 0x454dbc20, 0x454dc440
Caught exception in looking up space point ptr... 0x5cf96500, 0x2db69c70
Caught exception in looking up space point ptr... 0x2db69c70, 0x2db69b70
Caught exception in looking up space point ptr... 0x3b66c6e0, 0x3b66c070
Caught exception in looking up space point ptr... 0x2a662560, 0x3b66c6e0
Caught exception in looking up space point ptr... 0x4d0fdf80, 0x4dc69800
Caught exception in looking up space point ptr... 0x4dc69800, 0x4d0fdf80
Caught exception in looking up space point ptr... 0x28372fc0, 0x28373480
Caught exception in looking up space point ptr... 0x28373480, 0x28372fc0
Caught exception in looking up space point ptr... 0x5e0a6c30, 0x29948300
Caught exception in looking up space point ptr... 0x29948300, 0x5e0a6c30
Caught exception in looking up space point ptr... 0x4f025a60, 0x1021ea90
Caught exception in looking up space point ptr... 0x1021ea90, 0x5a0aa800
Caught exception in looking up space point ptr... 0xe58e2a0, 0xe595520
Caught exception in looking up space point ptr... 0xe595520, 0xe594270
Caught exception in looking up space point ptr... 0x2db106a0, 0x2db10510
Caught exception in looking up space point ptr... 0x2db10510, 0x2db106a0
Caught exception in looking up space point ptr... 0x42a2c270, 0x139f74b0
Caught exception in looking up space point ptr... 0x139f74b0, 0x42a2c270
Caught exception in looking up space point ptr... 0x2a66e820, 0x303855b0
Caught exception in looking up space point ptr... 0x303855b0, 0x5cbdc000
Caught exception in looking up space point ptr... 0x192b0b60, 0x328a96d0
Caught exception in looking up space point ptr... 0x328a96d0, 0x328a95d0
Caught exception in looking up space point ptr... 0x37a8a060, 0x3d3ef270
Caught exception in looking up space point ptr... 0x3d3ef270, 0x4751b010
Caught exception in looking up space point ptr... 0x4751b010, 0x3d3ef270
Caught exception in looking up space point ptr... 0x3d3ef270, 0x4af9c200
Caught exception in looking up space point ptr... 0x5e7abe20, 0x15a01020
Caught exception in looking up space point ptr... 0x15a01020, 0x5e7abe20
Caught exception in looking up space point ptr... 0x309b94d0, 0x586a14b0
Caught exception in looking up space point ptr... 0x586a14b0, 0x309b94d0
Caught exception in looking up space point ptr... 0x15d90d30, 0x29a80270
Caught exception in looking up space point ptr... 0x29a80270, 0x332f8b80
Caught exception in looking up space point ptr... 0x45361780, 0x45364d30
Caught exception in looking up space point ptr... 0x45364d30, 0x45361780
Caught exception in looking up space point ptr... 0x13a12f30, 0x616f8c00
Caught exception in looking up space point ptr... 0x616f8c00, 0x13a12f30
Caught exception in looking up space point ptr... 0x571f61c0, 0x571f5ea0
Caught exception in looking up space point ptr... 0x571f5ea0, 0x571f5d10
Caught exception in looking up space point ptr... 0x5b11cda0, 0x5b118f50
Caught exception in looking up space point ptr... 0x5b118f50, 0x5b11dc40
Caught exception in looking up space point ptr... 0x139fc9d0, 0x139f6560
Caught exception in looking up space point ptr... 0x139f6560, 0x139fced0
Caught exception in looking up space point ptr... 0x139fced0, 0x139fcd40
Caught exception in looking up space point ptr... 0x139fcd40, 0x139fc9d0
++++>>>> total num hits: 64276, num free: 61390
04-Feb-2026 01:02:56 CET  Closed output file "np02vd_raw_run042421_0004_df-s04-d1_dw_0_20260203T133303_reco_stage0_20260204T000256_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         66.3523       85.7483       200.608       76.5209       29.913         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.258e-05    7.00217e-05   0.00017863     6.008e-05    3.29323e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.43252       4.5859        15.5377       4.12642       2.1876         27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0115893     0.0213397     0.0710602     0.0148214     0.0155632       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000415391   0.000898466   0.00601481    0.000489641   0.00130416       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00149429    0.00193492    0.00345339    0.00179733    0.000382869      27     
produce:beamevent:BeamEvent                       5.156e-05    0.000122632    0.0011031    7.3241e-05    0.000196375      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.8375        2.40243       3.11992       2.38411      0.244616        27     
produce:ophit:OpHitFinder                         0.0280502     0.0341572     0.0445415     0.0336107     0.0039953       27     
produce:wclsdatavd:WireCellToolkit                 47.5297       52.2731       63.5065       51.0537       3.86482        27     
produce:gaushit:GausHitFinder                     0.567136       1.00706       3.86107      0.793151      0.671561        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00013947    0.000245765   0.000439131   0.000240541   8.25673e-05      27     
produce:reco3d:SpacePointSolver                    4.76309       9.56893       20.8398       8.33068       3.76854        27     
produce:hitpdune:DisambigFromSpacePoints          0.0474677      0.13894      0.712319      0.0758195     0.155683        27     
produce:cluster3d:Cluster3D                       0.808881       12.3275       111.044       1.32522       26.7049        27     
[art]:TriggerResults:TriggerResultInserter        1.694e-05    2.82237e-05   0.00010043     2.571e-05    1.44995e-05      27     
end_path:out1:RootOutput                          4.88e-06     1.00545e-05    3.244e-05     9.74e-06     5.22227e-06      27     
end_path:out1:RootOutput(write)                    2.68139       3.3676        6.10046       3.01018       0.84296        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6047.41 MB
  Peak resident set size usage (VmHWM): 4001.33 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0004_df-s04-d1_dw_0_20260203T133303_reco_stage0_20260204T000256_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0004_df-s04-d1_dw_0_20260203T133303_reco_stage0_20260204T000256_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0004_df-s04-d1_dw_0_20260203T133303_reco_stage0_20260204T000256_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161017.0,
    "core.end_time": 1770163377.0,
    "core.events": [
      653,
      659,
      665,
      671,
      677,
      683,
      689,
      695,
      701,
      707,
      713,
      719,
      725,
      731,
      737,
      743,
      749,
      755,
      761,
      767,
      773,
      779,
      785,
      791,
      797,
      803,
      809
    ],
    "core.event_count": 27,
    "core.first_event_number": 653,
    "core.last_event_number": 809,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0004_df-s04-d1_dw_0_20260203T133303.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:24 UTC       justIN version: 01.06.00