justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.174@dunegpschedd02.fnal.gov

Jobsub ID297920.174@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:31:20
From worker nodeHostnameb9p02p6824.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:43:09
Input filesvd-protodune:np02vd_raw_run042433_0112_df-s04-d1_dw_0_20260203T192441.hdf5
JobscriptExit code0
Real time45m (2745s)
CPU time44m (2648s = 96%)
Max RSS bytes3394367488 (3237 MiB)
Outputting started2026-02-04 00:28:55
Output files
Finished2026-02-04 00:31:20
Saved logsjustin-logs:297920.174-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x5e6b00b0, 0x15eeb3f0
Caught exception in looking up space point ptr... 0x65db4550, 0x65db4c30
Caught exception in looking up space point ptr... 0x65db4c30, 0x65db4550
Caught exception in looking up space point ptr... 0x7ccdc510, 0x7ccdc080
Caught exception in looking up space point ptr... 0x7ccdc080, 0x7ccdc510
Caught exception in looking up space point ptr... 0x3edcb170, 0x3edc9770
Caught exception in looking up space point ptr... 0x3edc9770, 0x3edcb170
Caught exception in looking up space point ptr... 0x319cb9d0, 0x319c7450
Caught exception in looking up space point ptr... 0x319c7450, 0x319cb9d0
Caught exception in looking up space point ptr... 0x352f79a0, 0x352f82a0
Caught exception in looking up space point ptr... 0x352f82a0, 0x352f5280
Caught exception in looking up space point ptr... 0x3b58d9e0, 0x3f9ce190
Caught exception in looking up space point ptr... 0x3f9ce190, 0x3a401c60
Caught exception in looking up space point ptr... 0x111c1ff0, 0x846b990
Caught exception in looking up space point ptr... 0x846b990, 0x111c1ff0
Caught exception in looking up space point ptr... 0x6310c1c0, 0x32830a90
Caught exception in looking up space point ptr... 0x32830a90, 0x6310c1c0
Caught exception in looking up space point ptr... 0x402a08a0, 0x402a0750
Caught exception in looking up space point ptr... 0x402a0750, 0x402a08a0
Caught exception in looking up space point ptr... 0x3b716040, 0x3b8bec90
Caught exception in looking up space point ptr... 0x3b8bec90, 0x4337fd80
Caught exception in looking up space point ptr... 0xadb6a40, 0x679082a0
Caught exception in looking up space point ptr... 0x679082a0, 0xadb6a40
Caught exception in looking up space point ptr... 0x139370a0, 0x13936d10
Caught exception in looking up space point ptr... 0x13936d10, 0x139370a0
Caught exception in looking up space point ptr... 0x15eeb8f0, 0x78f341f0
Caught exception in looking up space point ptr... 0x78f33de0, 0x15eeb8f0
Caught exception in looking up space point ptr... 0x34609e50, 0x34609c70
Caught exception in looking up space point ptr... 0x34609c70, 0x34609e50
Caught exception in looking up space point ptr... 0x4e1807d0, 0x4e1803f0
Caught exception in looking up space point ptr... 0x4e1803f0, 0x4e1807d0
Caught exception in looking up space point ptr... 0x191fcff0, 0x3aa74440
Caught exception in looking up space point ptr... 0x3aa74440, 0x3043aff0
Caught exception in looking up space point ptr... 0x65d1d6e0, 0x65d1ce70
Caught exception in looking up space point ptr... 0x65d1ce70, 0x65d1d6e0
Caught exception in looking up space point ptr... 0x30238dc0, 0x13933c70
Caught exception in looking up space point ptr... 0x13933c70, 0x30238dc0
Caught exception in looking up space point ptr... 0x30238dc0, 0x30202e70
Caught exception in looking up space point ptr... 0x30202e70, 0x30238dc0
Caught exception in looking up space point ptr... 0x676676e0, 0x67667550
Caught exception in looking up space point ptr... 0x67667550, 0x676676e0
Caught exception in looking up space point ptr... 0x2b007f30, 0x2afff740
Caught exception in looking up space point ptr... 0x2afff740, 0x2b007f30
Caught exception in looking up space point ptr... 0x2b007f30, 0x2b007da0
Caught exception in looking up space point ptr... 0x2b007da0, 0x2b007f30
Caught exception in looking up space point ptr... 0x2b007f30, 0x2afff530
Caught exception in looking up space point ptr... 0x2afff530, 0x2b007f30
Caught exception in looking up space point ptr... 0x2aff1490, 0x1483efb0
Caught exception in looking up space point ptr... 0x1483efb0, 0x2aff1490
Caught exception in looking up space point ptr... 0x10af8860, 0x336df100
Caught exception in looking up space point ptr... 0x336df100, 0x10af8860
Caught exception in looking up space point ptr... 0xcabcfa0, 0xcab4330
Caught exception in looking up space point ptr... 0xcab4330, 0xcabcfa0
Caught exception in looking up space point ptr... 0x2aff0540, 0x2aff73b0
Caught exception in looking up space point ptr... 0x2aff73b0, 0x2aff6ed0
Caught exception in looking up space point ptr... 0x40812cf0, 0x81b11260
Caught exception in looking up space point ptr... 0x81b11260, 0x5cb83ef0
Caught exception in looking up space point ptr... 0x352d3f10, 0x352d3830
Caught exception in looking up space point ptr... 0x352d3830, 0x352d3f10
Caught exception in looking up space point ptr... 0x198abd60, 0x17ed8bc0
Caught exception in looking up space point ptr... 0x367511f0, 0x198abd60
Caught exception in looking up space point ptr... 0x352d0590, 0x352d0290
Caught exception in looking up space point ptr... 0x352d0290, 0x352d0590
Caught exception in looking up space point ptr... 0x3354a320, 0x3f7e3080
Caught exception in looking up space point ptr... 0x3f7e3080, 0x116dd480
Caught exception in looking up space point ptr... 0x375c16f0, 0x3b766730
Caught exception in looking up space point ptr... 0x3b766730, 0x375c16f0
++++>>>> total num hits: 119673, num free: 115142
04-Feb-2026 01:28:47 CET  Closed output file "np02vd_raw_run042433_0112_df-s04-d1_dw_0_20260203T192441_reco_stage0_20260204T002847_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         122.614       141.125       163.821       140.23        11.0676        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.104e-05    0.000105688   0.000224232   9.1981e-05    4.02781e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.95054       6.97107       7.52747       7.0389       0.384147        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0128908     0.0268545     0.0743897     0.0217714     0.0165123       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000447204   0.00156441     0.0103083    0.000584415   0.00243243       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000464874   0.00061302    0.00100371    0.000542844   0.000139539      19     
produce:beamevent:BeamEvent                      7.0721e-05    0.000148978   0.00100414    0.000105711   0.000202477      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.2843        2.76826       3.32706       2.76835      0.261361        19     
produce:ophit:OpHitFinder                         0.0255373     0.036427      0.0416115     0.0372478    0.00444072       19     
produce:wclsdatavd:WireCellToolkit                 85.2982       99.5768       120.72        98.6068       9.14419        19     
produce:gaushit:GausHitFinder                     0.974917       1.58713       2.25195       1.57839      0.365468        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000312843   0.000424645   0.000675416   0.000440423   8.82393e-05      19     
produce:reco3d:SpacePointSolver                    18.1665       22.0977       29.1821       21.9117       3.08621        19     
produce:hitpdune:DisambigFromSpacePoints          0.124569      0.193098       0.27486      0.181158      0.0477566       19     
produce:cluster3d:Cluster3D                        1.61162       2.50266       3.96161       2.32244      0.614625        19     
[art]:TriggerResults:TriggerResultInserter        2.66e-05     3.91415e-05   0.000100441    3.366e-05    1.63104e-05      19     
end_path:out1:RootOutput                          3.59e-06     7.22526e-06    2.638e-05     6.44e-06     4.89827e-06      19     
end_path:out1:RootOutput(write)                    4.29844       5.33036       6.61342       4.88628      0.879887        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5383.79 MB
  Peak resident set size usage (VmHWM): 3394.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0112_df-s04-d1_dw_0_20260203T192441_reco_stage0_20260204T002847_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0112_df-s04-d1_dw_0_20260203T192441_reco_stage0_20260204T002847_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0112_df-s04-d1_dw_0_20260203T192441_reco_stage0_20260204T002847_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162201.0,
    "core.end_time": 1770164928.0,
    "core.events": [
      12779,
      12785,
      12791,
      12797,
      12803,
      12809,
      12815,
      12821,
      12827,
      12833,
      12839,
      12845,
      12851,
      12857,
      12863,
      12869,
      12875,
      12881,
      12887
    ],
    "core.event_count": 19,
    "core.first_event_number": 12779,
    "core.last_event_number": 12887,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0112_df-s04-d1_dw_0_20260203T192441.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:08 UTC       justIN version: 01.06.00