justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.70@dunegpschedd01.fnal.gov

Jobsub ID303654.70@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 00:44:13
From worker nodeHostnameb9p12p0289.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:02:03
Input filesvd-protodune:np02vd_raw_run042433_0181_df-s04-d0_dw_0_20260203T195057.hdf5
JobscriptExit code0
Real time39m (2391s)
CPU time38m (2306s = 96%)
Max RSS bytes3368812544 (3212 MiB)
Outputting started2026-02-04 00:41:55
Output files
Finished2026-02-04 00:44:13
Saved logsjustin-logs:303654.70-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x4bbb9f10, 0x4bbb6cf0
Caught exception in looking up space point ptr... 0x621bba40, 0x621bb8f0
Caught exception in looking up space point ptr... 0x621bb8f0, 0x621bfd30
Caught exception in looking up space point ptr... 0x80ed9160, 0x80ed9290
Caught exception in looking up space point ptr... 0x80ed9290, 0x80ed9160
Caught exception in looking up space point ptr... 0x4d4e0b90, 0x4d4dd4a0
Caught exception in looking up space point ptr... 0x4d4dd4a0, 0x4d4e0b90
Caught exception in looking up space point ptr... 0x39e45bc0, 0x3d9fe5b0
Caught exception in looking up space point ptr... 0x3d9fe5b0, 0x3d9fe740
Caught exception in looking up space point ptr... 0xa7538c0, 0xa753410
Caught exception in looking up space point ptr... 0xa753410, 0xa7538c0
Caught exception in looking up space point ptr... 0x51e26b80, 0x51e26920
Caught exception in looking up space point ptr... 0x51e26920, 0x51e0a330
Caught exception in looking up space point ptr... 0x6565b9e0, 0x6565b850
Caught exception in looking up space point ptr... 0x6565b850, 0x6565b530
Caught exception in looking up space point ptr... 0x53997060, 0x53996f10
Caught exception in looking up space point ptr... 0x66fbf3f0, 0x53997060
Caught exception in looking up space point ptr... 0x2a3295c0, 0x2a314280
Caught exception in looking up space point ptr... 0x2a314280, 0x2a32b980
Caught exception in looking up space point ptr... 0x3eb7abe0, 0x3eb907b0
Caught exception in looking up space point ptr... 0x3eb907b0, 0x3eb7ad70
Caught exception in looking up space point ptr... 0x7c9f2540, 0x7c9ef3c0
Caught exception in looking up space point ptr... 0x7c9ef3c0, 0x7c9f2540
Caught exception in looking up space point ptr... 0x621a92d0, 0x621ab210
Caught exception in looking up space point ptr... 0x621ab210, 0x621a92d0
Caught exception in looking up space point ptr... 0x3eb79ac0, 0x3eb79930
Caught exception in looking up space point ptr... 0x3eb79930, 0x3eb79ac0
Caught exception in looking up space point ptr... 0x4f8687e0, 0x441197b0
Caught exception in looking up space point ptr... 0x441197b0, 0x4f8687e0
Caught exception in looking up space point ptr... 0x62d51af0, 0x62d3ef70
Caught exception in looking up space point ptr... 0x62d54a20, 0x62d51af0
Caught exception in looking up space point ptr... 0x1a4ff320, 0x1a4ea7f0
Caught exception in looking up space point ptr... 0x1a4ea7f0, 0x1a4ff320
Caught exception in looking up space point ptr... 0x5233ccd0, 0x5233ce60
Caught exception in looking up space point ptr... 0x5233ce60, 0x5233ccd0
Caught exception in looking up space point ptr... 0x81125610, 0x811219a0
Caught exception in looking up space point ptr... 0x811219a0, 0x81125490
Caught exception in looking up space point ptr... 0x6213c6c0, 0x6213c430
Caught exception in looking up space point ptr... 0x6213c990, 0x6213c6c0
Caught exception in looking up space point ptr... 0x3aa399a0, 0x6eb8de10
Caught exception in looking up space point ptr... 0x6eb8de10, 0x74262fc0
Caught exception in looking up space point ptr... 0x8ab3620, 0x8aae870
Caught exception in looking up space point ptr... 0x8aae870, 0x8ab3620
Caught exception in looking up space point ptr... 0x19b732f0, 0x63fc0060
Caught exception in looking up space point ptr... 0x63fc0060, 0x63fbd450
Caught exception in looking up space point ptr... 0x4bb917a0, 0x4bba9c80
Caught exception in looking up space point ptr... 0x4bba9c80, 0x4bb91400
Caught exception in looking up space point ptr... 0x5397e5b0, 0x5397e700
Caught exception in looking up space point ptr... 0x5397e700, 0x326a7700
Caught exception in looking up space point ptr... 0x141e8fa0, 0x141e9450
Caught exception in looking up space point ptr... 0x141e9450, 0x141e8fa0
Caught exception in looking up space point ptr... 0x17699af0, 0x18f709b0
Caught exception in looking up space point ptr... 0x18f709b0, 0x3a59e460
Caught exception in looking up space point ptr... 0x1473b030, 0x1473b3a0
Caught exception in looking up space point ptr... 0x1473b3a0, 0x1473b030
Caught exception in looking up space point ptr... 0x132ce790, 0x132ceb20
Caught exception in looking up space point ptr... 0x132ceb20, 0x132cc410
Caught exception in looking up space point ptr... 0x62fa0f00, 0x62fa1b70
Caught exception in looking up space point ptr... 0x62fa1b70, 0x62fa1030
Caught exception in looking up space point ptr... 0x372910c0, 0x6a944920
Caught exception in looking up space point ptr... 0x6a944920, 0x5fa20580
Caught exception in looking up space point ptr... 0x621d0b40, 0x5807b440
Caught exception in looking up space point ptr... 0x5807b440, 0x621d0b40
Caught exception in looking up space point ptr... 0x4baa4230, 0x4baa43c0
Caught exception in looking up space point ptr... 0x4baa43c0, 0x4baa4230
Caught exception in looking up space point ptr... 0x825883c0, 0x82588050
Caught exception in looking up space point ptr... 0x82588050, 0x825883c0
++++>>>> total num hits: 158965, num free: 153580
04-Feb-2026 01:41:46 CET  Closed output file "np02vd_raw_run042433_0181_df-s04-d0_dw_0_20260203T195057_reco_stage0_20260204T004146_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         108.58          123         143.038       121.717       7.54389        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.69e-05     8.67846e-05   0.000271561    7.83e-05     4.76555e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.14908       6.2694        10.8171       6.09929       1.12254        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.010392      0.0205298     0.0651827     0.0143659     0.0137766       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000387301   0.000522553   0.000812113   0.000510501   8.90215e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000409302   0.000499775   0.000671653   0.000476852   7.60473e-05      19     
produce:beamevent:BeamEvent                       5.691e-05    0.000127613   0.000708452    8.756e-05     0.0001398       19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.02982       2.55514       3.23204       2.4785       0.305264        19     
produce:ophit:OpHitFinder                         0.0269585     0.0349468     0.0432446     0.0348162    0.00438821       19     
produce:wclsdatavd:WireCellToolkit                 83.7532       93.6074       105.31        94.5369       4.81636        19     
produce:gaushit:GausHitFinder                     0.751558       1.04034       1.44291       1.02061      0.163217        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000220171   0.000286611   0.000457512   0.000264521   7.07747e-05      19     
produce:reco3d:SpacePointSolver                    9.22744       12.5612       18.6631       12.2909       2.27361        19     
produce:hitpdune:DisambigFromSpacePoints          0.110276      0.143105      0.215927      0.133197      0.0292364       19     
produce:cluster3d:Cluster3D                        1.15112       2.13685       3.87668       1.87007      0.717023        19     
[art]:TriggerResults:TriggerResultInserter        2.256e-05    3.16096e-05    7.789e-05    2.9551e-05    1.14049e-05      19     
end_path:out1:RootOutput                          3.36e-06     7.07011e-06    2.235e-05     4.81e-06     5.42968e-06      19     
end_path:out1:RootOutput(write)                    4.0742        4.59866       5.19423       4.58073      0.321883        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5426.85 MB
  Peak resident set size usage (VmHWM): 3368.81 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0181_df-s04-d0_dw_0_20260203T195057_reco_stage0_20260204T004146_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0181_df-s04-d0_dw_0_20260203T195057_reco_stage0_20260204T004146_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0181_df-s04-d0_dw_0_20260203T195057_reco_stage0_20260204T004146_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163333.0,
    "core.end_time": 1770165707.0,
    "core.events": [
      20644,
      20650,
      20656,
      20662,
      20668,
      20674,
      20680,
      20686,
      20692,
      20698,
      20704,
      20710,
      20716,
      20722,
      20728,
      20734,
      20740,
      20746,
      20752
    ],
    "core.event_count": 19,
    "core.first_event_number": 20644,
    "core.last_event_number": 20752,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0181_df-s04-d0_dw_0_20260203T195057.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:43:44 UTC       justIN version: 01.06.00