justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297916.54@dunegpschedd02.fnal.gov

Jobsub ID297916.54@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce505
Last heartbeat2026-02-03 23:21:44
From worker nodeHostnameb9p35p0031.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:36
Input filesvd-protodune:np02vd_raw_run042422_0110_df-s04-d1_dw_0_20260203T161415.hdf5
JobscriptExit code0
Real time32m (1920s)
CPU time30m (1851s = 96%)
Max RSS bytes2632777728 (2510 MiB)
Outputting started2026-02-03 23:19:36
Output files
Finished2026-02-03 23:21:44
Saved logsjustin-logs:297916.54-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

12392490
Caught exception in looking up space point ptr... 0x374925a0, 0x51860e40
Caught exception in looking up space point ptr... 0x51860e40, 0x37492350
Caught exception in looking up space point ptr... 0x37492350, 0x15c91690
Caught exception in looking up space point ptr... 0x15c91690, 0x374927f0
Caught exception in looking up space point ptr... 0x374927f0, 0x374925a0
Caught exception in looking up space point ptr... 0x3f2669e0, 0x5f55bc60
Caught exception in looking up space point ptr... 0x5f55bc60, 0x3f2669e0
Caught exception in looking up space point ptr... 0x150e7ce0, 0x4fa4adf0
Caught exception in looking up space point ptr... 0x4fa4adf0, 0x46b066b0
Caught exception in looking up space point ptr... 0x2d6dd1b0, 0x56614be0
Caught exception in looking up space point ptr... 0x56614be0, 0x28f0a1e0
Caught exception in looking up space point ptr... 0x28f0a1e0, 0x4f99ded0
Caught exception in looking up space point ptr... 0x4f99ded0, 0x2d6dd1b0
Caught exception in looking up space point ptr... 0x5bfbe110, 0xa702f50
Caught exception in looking up space point ptr... 0xa702f50, 0x5bfbf4c0
Caught exception in looking up space point ptr... 0x59408dc0, 0x2b0d48a0
Caught exception in looking up space point ptr... 0x2b0d48a0, 0x59408dc0
Caught exception in looking up space point ptr... 0x4f758e30, 0x4f758b10
Caught exception in looking up space point ptr... 0x4f758b10, 0x4f758e30
Caught exception in looking up space point ptr... 0x583e5d90, 0x574a2870
Caught exception in looking up space point ptr... 0x574a2870, 0x583e5d90
Caught exception in looking up space point ptr... 0x47c0f110, 0x47c0dbb0
Caught exception in looking up space point ptr... 0x47c0dbb0, 0x2dceb0c0
Caught exception in looking up space point ptr... 0x331e37e0, 0x45ba4b00
Caught exception in looking up space point ptr... 0x45ba4b00, 0x4cafc030
Caught exception in looking up space point ptr... 0x5edb53f0, 0x8b82e70
Caught exception in looking up space point ptr... 0x8b82e70, 0x5edb1600
Caught exception in looking up space point ptr... 0x958d930, 0x2dc66ad0
Caught exception in looking up space point ptr... 0x2dc66ad0, 0x25b93bb0
Caught exception in looking up space point ptr... 0x5870f1f0, 0x318afd70
Caught exception in looking up space point ptr... 0x318afd70, 0x5870f1f0
Caught exception in looking up space point ptr... 0x6025f110, 0x566dad50
Caught exception in looking up space point ptr... 0x566dad50, 0x2a6da890
Caught exception in looking up space point ptr... 0x4e340470, 0xad40130
Caught exception in looking up space point ptr... 0xad40130, 0x4e340470
Caught exception in looking up space point ptr... 0x2dbe2800, 0x2dbe24e0
Caught exception in looking up space point ptr... 0x2dbe24e0, 0x2dbe2350
Caught exception in looking up space point ptr... 0xacc6c70, 0xacc78b0
Caught exception in looking up space point ptr... 0xacc78b0, 0xacca320
Caught exception in looking up space point ptr... 0x145315f0, 0x3f2574a0
Caught exception in looking up space point ptr... 0x3f2574a0, 0x2995bc30
Caught exception in looking up space point ptr... 0x449686c0, 0x449683a0
Caught exception in looking up space point ptr... 0x449683a0, 0x449686c0
Caught exception in looking up space point ptr... 0x449686c0, 0x449681c0
Caught exception in looking up space point ptr... 0x449681c0, 0x449686c0
Caught exception in looking up space point ptr... 0x5bf4dd60, 0x5bf4def0
Caught exception in looking up space point ptr... 0x5bf4def0, 0x218ad3e0
Caught exception in looking up space point ptr... 0x11c50fc0, 0x11c511e0
Caught exception in looking up space point ptr... 0x11c511e0, 0x11c50fc0
Caught exception in looking up space point ptr... 0x11c50fc0, 0x33d53070
Caught exception in looking up space point ptr... 0x33d53070, 0x11c50fc0
Caught exception in looking up space point ptr... 0x15c9a4f0, 0x15c9a360
Caught exception in looking up space point ptr... 0x374a0180, 0x15c9a4f0
Caught exception in looking up space point ptr... 0x3748a9f0, 0x3748a6d0
Caught exception in looking up space point ptr... 0x3748a6d0, 0x3748a9f0
Caught exception in looking up space point ptr... 0x5bfaf3f0, 0x5bfaf320
Caught exception in looking up space point ptr... 0x5bfaf320, 0x5bfaf3f0
Caught exception in looking up space point ptr... 0xaac35e0, 0xaac3400
Caught exception in looking up space point ptr... 0xaac3400, 0xaac35e0
Caught exception in looking up space point ptr... 0x54b41fb0, 0x26243cf0
Caught exception in looking up space point ptr... 0x26243cf0, 0x54b41fb0
Caught exception in looking up space point ptr... 0x13225290, 0x496f04a0
Caught exception in looking up space point ptr... 0x496f04a0, 0x13225290
Caught exception in looking up space point ptr... 0x4ac14140, 0x1534dd30
Caught exception in looking up space point ptr... 0x1534dd30, 0x4ac14140
++++>>>> total num hits: 46309, num free: 43484
04-Feb-2026 00:19:30 CET  Closed output file "np02vd_raw_run042422_0110_df-s04-d1_dw_0_20260203T161415_reco_stage0_20260203T231930_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         62.1644       69.0343       86.2489       67.7595       5.76561        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.527e-05    7.24456e-05   0.00016839     7.163e-05    2.03517e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.53159       3.93843       5.99211       3.87073      0.447085        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00814468     0.0161713     0.0911379     0.0103118     0.0164027       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00030504    0.000508241   0.00075101    0.00046259    0.000120785      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00113774    0.00149164    0.00184038    0.00153091    0.000182479      27     
produce:beamevent:BeamEvent                       6.58e-05     0.000113081   0.00081574     8.58e-05     0.000138487      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.70466       2.06826       2.29493       2.08332      0.138803        27     
produce:ophit:OpHitFinder                         0.0240351     0.0327582     0.0409075     0.0332966    0.00389324       27     
produce:wclsdatavd:WireCellToolkit                 44.1394       48.3894       57.9922       47.9682       2.66804        27     
produce:gaushit:GausHitFinder                     0.488766      0.746511       1.42333      0.694137      0.219171        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00012197    0.000188939   0.000526501   0.00017624    7.47556e-05      27     
produce:reco3d:SpacePointSolver                    4.36521       8.18731       12.372        8.07529       2.27861        27     
produce:hitpdune:DisambigFromSpacePoints          0.0442648     0.0873108      0.17745      0.0806135     0.0340104       27     
produce:cluster3d:Cluster3D                       0.754398        2.017        17.9092       1.24592       3.16921        27     
[art]:TriggerResults:TriggerResultInserter        2.203e-05    3.43922e-05    8.908e-05     3.249e-05    1.2574e-05       27     
end_path:out1:RootOutput                          4.56e-06     5.97667e-06    2.152e-05     5.19e-06     3.10975e-06      27     
end_path:out1:RootOutput(write)                    2.74123       3.52311       4.37252       3.51195      0.389139        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4645.14 MB
  Peak resident set size usage (VmHWM): 2632.78 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0110_df-s04-d1_dw_0_20260203T161415_reco_stage0_20260203T231930_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0110_df-s04-d1_dw_0_20260203T161415_reco_stage0_20260203T231930_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0110_df-s04-d1_dw_0_20260203T161415_reco_stage0_20260203T231930_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158864.0,
    "core.end_time": 1770160772.0,
    "core.events": [
      17837,
      17843,
      17849,
      17855,
      17861,
      17867,
      17873,
      17879,
      17885,
      17891,
      17897,
      17903,
      17909,
      17915,
      17921,
      17927,
      17933,
      17939,
      17945,
      17951,
      17957,
      17963,
      17969,
      17975,
      17981,
      17987,
      17993
    ],
    "core.event_count": 27,
    "core.first_event_number": 17837,
    "core.last_event_number": 17993,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0110_df-s04-d1_dw_0_20260203T161415.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:35:54 UTC       justIN version: 01.06.00