justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.178@dunegpschedd01.fnal.gov

Jobsub ID303653.178@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2026-02-04 00:23:21
From worker nodeHostnameb9p35p0043.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:48:39
Input filesvd-protodune:np02vd_raw_run042421_0068_df-s04-d1_dw_0_20260203T140751.hdf5
JobscriptExit code0
Real time32m (1953s)
CPU time31m (1869s = 95%)
Max RSS bytes2582867968 (2463 MiB)
Outputting started2026-02-04 00:21:13
Output files
Finished2026-02-04 00:23:21
Saved logsjustin-logs:303653.178-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2e0, 0x10a25ef0
Caught exception in looking up space point ptr... 0x2da56d10, 0x2da638d0
Caught exception in looking up space point ptr... 0x44a6ee30, 0x44a6f080
Caught exception in looking up space point ptr... 0x44a6f080, 0x44a6f2d0
Caught exception in looking up space point ptr... 0x44a6f2d0, 0x2da56d10
Caught exception in looking up space point ptr... 0x608c06c0, 0x32a73ab0
Caught exception in looking up space point ptr... 0x32a73ab0, 0x608c06c0
Caught exception in looking up space point ptr... 0x2cffc2a0, 0x2cfff190
Caught exception in looking up space point ptr... 0x2cfff190, 0x2cfff030
Caught exception in looking up space point ptr... 0x2cfff030, 0x5d29eba0
Caught exception in looking up space point ptr... 0x163e6680, 0x5403c060
Caught exception in looking up space point ptr... 0x5403c060, 0x163e6580
Caught exception in looking up space point ptr... 0x44a63f30, 0x5e3bfad0
Caught exception in looking up space point ptr... 0x5e3bfad0, 0x44a63f30
Caught exception in looking up space point ptr... 0x44a63f30, 0x5e3ba9a0
Caught exception in looking up space point ptr... 0x5e3ba9a0, 0x44a63f30
Caught exception in looking up space point ptr... 0x36527fb0, 0x2a80f2a0
Caught exception in looking up space point ptr... 0x2a80f2a0, 0x36527fb0
Caught exception in looking up space point ptr... 0x8837990, 0x883ab60
Caught exception in looking up space point ptr... 0x883cd70, 0x8837990
Caught exception in looking up space point ptr... 0x20763280, 0x207633c0
Caught exception in looking up space point ptr... 0x207633c0, 0x20763500
Caught exception in looking up space point ptr... 0x20763780, 0x20763640
Caught exception in looking up space point ptr... 0x20763640, 0x20763280
Caught exception in looking up space point ptr... 0x165d20a0, 0x23d060e0
Caught exception in looking up space point ptr... 0x23d060e0, 0x165d20a0
Caught exception in looking up space point ptr... 0x4ae9b330, 0xb668160
Caught exception in looking up space point ptr... 0xb668160, 0x4ae9b330
Caught exception in looking up space point ptr... 0x2da49f60, 0x2da4a280
Caught exception in looking up space point ptr... 0x2da4a280, 0x2da4a0f0
Caught exception in looking up space point ptr... 0x227c3560, 0x227c37c0
Caught exception in looking up space point ptr... 0x227c37c0, 0x227c3560
Caught exception in looking up space point ptr... 0x2a2f71d0, 0x2a2f6e40
Caught exception in looking up space point ptr... 0x2a2f6e40, 0x2a2f6f70
Caught exception in looking up space point ptr... 0xac067d0, 0xac06320
Caught exception in looking up space point ptr... 0xac06320, 0xac067d0
Caught exception in looking up space point ptr... 0x396f72a0, 0x396f75c0
Caught exception in looking up space point ptr... 0x396f75c0, 0x396f7750
Caught exception in looking up space point ptr... 0x2397b550, 0x239884d0
Caught exception in looking up space point ptr... 0x23984d90, 0x2397b550
Caught exception in looking up space point ptr... 0x5e08ef60, 0x5e08f510
Caught exception in looking up space point ptr... 0x5e08f510, 0x5e08f770
Caught exception in looking up space point ptr... 0x2d7e1700, 0x2d7e1890
Caught exception in looking up space point ptr... 0x2d7e1890, 0x2d7e1a20
Caught exception in looking up space point ptr... 0x2a2a9ce0, 0x604e2f70
Caught exception in looking up space point ptr... 0x604e2f70, 0x2a29fe10
Caught exception in looking up space point ptr... 0x4adafef0, 0x4adb00f0
Caught exception in looking up space point ptr... 0x4adb00f0, 0x4adafef0
Caught exception in looking up space point ptr... 0x2dae7e10, 0x2dae75c0
Caught exception in looking up space point ptr... 0x2dae75c0, 0x2dae7230
Caught exception in looking up space point ptr... 0x2dae7230, 0x2dae7ce0
Caught exception in looking up space point ptr... 0x2dae7ce0, 0x2dae7e10
Caught exception in looking up space point ptr... 0x15bfffd0, 0x15bfeba0
Caught exception in looking up space point ptr... 0x15bfeba0, 0x15bffcb0
Caught exception in looking up space point ptr... 0xf9bf940, 0xf9bff20
Caught exception in looking up space point ptr... 0xf9bff20, 0xf9bf940
Caught exception in looking up space point ptr... 0xf9bf940, 0xf9bfdd0
Caught exception in looking up space point ptr... 0xf9bfdd0, 0xf9bf940
Caught exception in looking up space point ptr... 0xf9bf940, 0xf9bfae0
Caught exception in looking up space point ptr... 0xf9bfae0, 0xf9bf940
Caught exception in looking up space point ptr... 0x1607b520, 0x3a71ed70
Caught exception in looking up space point ptr... 0x3a71ed70, 0x1607b520
Caught exception in looking up space point ptr... 0x1607b520, 0x1607b670
Caught exception in looking up space point ptr... 0x1607b670, 0x1607b520
Caught exception in looking up space point ptr... 0x312c6fa0, 0x312c7130
Caught exception in looking up space point ptr... 0x312c7130, 0x312c6fa0
++++>>>> total num hits: 68834, num free: 65041
04-Feb-2026 01:21:07 CET  Closed output file "np02vd_raw_run042421_0068_df-s04-d1_dw_0_20260203T140751_reco_stage0_20260204T002107_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         60.5036       70.1421       82.3995       70.0148       4.88382        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.6741e-05    7.57586e-05   0.00016184     7.07e-05     1.85843e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.6916        4.28355       10.6148       4.0277        1.25878        27     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00835821     0.0151953     0.0376255     0.0109826    0.00819499       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0003734    0.000466506   0.000771351   0.00043024    9.97811e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00115813    0.00143289    0.00189262    0.00138889    0.000206757      27     
produce:beamevent:BeamEvent                       6.161e-05    0.000109979   0.00084726     8.231e-05    0.000145287      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.92862       2.11808       2.36758       2.10641      0.116328        27     
produce:ophit:OpHitFinder                         0.0266449     0.034053      0.0431704     0.0332769    0.00375226       27     
produce:wclsdatavd:WireCellToolkit                 45.3519       49.623        55.576        49.5011       2.92751        27     
produce:gaushit:GausHitFinder                     0.491501      0.790015       1.12115      0.788709      0.176858        27     
produce:nhitsfilter:NumberOfHitsFilter             0.00011     0.000189876   0.00047251    0.00017663    7.14882e-05      27     
produce:reco3d:SpacePointSolver                    4.00965       8.07882       13.8633       7.52261       2.62364        27     
produce:hitpdune:DisambigFromSpacePoints          0.0348812     0.0815075     0.122674      0.0891435     0.025188        27     
produce:cluster3d:Cluster3D                       0.529266       1.45395       2.9971        1.4165       0.530435        27     
[art]:TriggerResults:TriggerResultInserter        2.493e-05    3.50556e-05    8.97e-05      2.911e-05    1.49923e-05      27     
end_path:out1:RootOutput                          4.23e-06     5.79407e-06    2.193e-05     5.19e-06     3.20059e-06      27     
end_path:out1:RootOutput(write)                    3.0496        3.63521       4.30057       3.66251      0.320306        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4565.54 MB
  Peak resident set size usage (VmHWM): 2582.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0068_df-s04-d1_dw_0_20260203T140751_reco_stage0_20260204T002107_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0068_df-s04-d1_dw_0_20260203T140751_reco_stage0_20260204T002107_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0068_df-s04-d1_dw_0_20260203T140751_reco_stage0_20260204T002107_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162528.0,
    "core.end_time": 1770164468.0,
    "core.events": [
      11081,
      11087,
      11093,
      11099,
      11105,
      11111,
      11117,
      11123,
      11129,
      11135,
      11141,
      11147,
      11153,
      11159,
      11165,
      11171,
      11177,
      11183,
      11189,
      11195,
      11201,
      11207,
      11213,
      11219,
      11225,
      11231,
      11237
    ],
    "core.event_count": 27,
    "core.first_event_number": 11081,
    "core.last_event_number": 11237,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0068_df-s04-d1_dw_0_20260203T140751.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:41:51 UTC       justIN version: 01.06.00