justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.136@dunegpschedd01.fnal.gov

Jobsub ID303654.136@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:51:53
From worker nodeHostnameb9p05p0564.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:09:47
Input filesvd-protodune:np02vd_raw_run042433_0223_df-s04-d1_dw_0_20260203T200653.hdf5
JobscriptExit code0
Real time39m (2390s)
CPU time38m (2305s = 96%)
Max RSS bytes3403382784 (3245 MiB)
Outputting started2026-02-04 00:49:38
Output files
Finished2026-02-04 00:51:53
Saved logsjustin-logs:303654.136-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x53f577c0, 0x816d5880
Caught exception in looking up space point ptr... 0x816d5880, 0x53f577c0
Caught exception in looking up space point ptr... 0x57acad30, 0x56e79540
Caught exception in looking up space point ptr... 0x56e79540, 0x57acbba0
Caught exception in looking up space point ptr... 0x3ac1de40, 0x3ac1ea90
Caught exception in looking up space point ptr... 0x3ac1ea90, 0x3ac1ee40
Caught exception in looking up space point ptr... 0x3ac1ee40, 0x3ac1dc90
Caught exception in looking up space point ptr... 0x4f547fc0, 0x6e0cf930
Caught exception in looking up space point ptr... 0x6e0cf930, 0x684eef20
Caught exception in looking up space point ptr... 0x684eef20, 0x50142ef0
Caught exception in looking up space point ptr... 0x50142ef0, 0x4f547fc0
Caught exception in looking up space point ptr... 0x3572bbc0, 0x3572c080
Caught exception in looking up space point ptr... 0x3572c080, 0x3572bbc0
Caught exception in looking up space point ptr... 0x4ee27eb0, 0x184d1930
Caught exception in looking up space point ptr... 0x2fa45720, 0x4ee27eb0
Caught exception in looking up space point ptr... 0x57e734f0, 0x5ce12ae0
Caught exception in looking up space point ptr... 0x5ce12ae0, 0x57e734f0
Caught exception in looking up space point ptr... 0x83119d70, 0x4b90ebb0
Caught exception in looking up space point ptr... 0x4b910830, 0x83119d70
Caught exception in looking up space point ptr... 0x3799cf00, 0x3799d000
Caught exception in looking up space point ptr... 0x3799d000, 0x3799cf00
Caught exception in looking up space point ptr... 0x3799cf00, 0x49d26f20
Caught exception in looking up space point ptr... 0x49d26f20, 0x3799cf00
Caught exception in looking up space point ptr... 0x1274efa0, 0x40b58210
Caught exception in looking up space point ptr... 0x76eaad70, 0x636ae460
Caught exception in looking up space point ptr... 0x636ae460, 0x1a2fc2c0
Caught exception in looking up space point ptr... 0x1a2fc2c0, 0x1274efa0
Caught exception in looking up space point ptr... 0x372bd6f0, 0x6fca7820
Caught exception in looking up space point ptr... 0x6fca7820, 0x37c85890
Caught exception in looking up space point ptr... 0x816d2870, 0x816d2720
Caught exception in looking up space point ptr... 0x816d2720, 0x816d2870
Caught exception in looking up space point ptr... 0x537cf270, 0x537cef70
Caught exception in looking up space point ptr... 0x537cef70, 0x537cf270
Caught exception in looking up space point ptr... 0x502268d0, 0x50226c00
Caught exception in looking up space point ptr... 0x50226c00, 0x50226a70
Caught exception in looking up space point ptr... 0x195aedd0, 0x711592b0
Caught exception in looking up space point ptr... 0x711592b0, 0x2cdc8370
Caught exception in looking up space point ptr... 0x69744690, 0x69744820
Caught exception in looking up space point ptr... 0x69744820, 0x69744690
Caught exception in looking up space point ptr... 0x126909a0, 0x390fff20
Caught exception in looking up space point ptr... 0x390fff20, 0x126909a0
Caught exception in looking up space point ptr... 0x480c9ef0, 0x480ca140
Caught exception in looking up space point ptr... 0x480ca140, 0x480c9ef0
Caught exception in looking up space point ptr... 0x49b7a510, 0x49b7a410
Caught exception in looking up space point ptr... 0x49b7a410, 0x49b7a510
Caught exception in looking up space point ptr... 0x48c64fa0, 0x1a0c70c0
Caught exception in looking up space point ptr... 0x1a0c70c0, 0x48c64fa0
Caught exception in looking up space point ptr... 0x6bee2120, 0x79cb5390
Caught exception in looking up space point ptr... 0x79cb5390, 0x6bee2120
Caught exception in looking up space point ptr... 0x6becf9b0, 0x6bed0ae0
Caught exception in looking up space point ptr... 0x6bed0ae0, 0x6becf9b0
Caught exception in looking up space point ptr... 0x470ee730, 0x67edf0f0
Caught exception in looking up space point ptr... 0x67edf0f0, 0x470ecec0
Caught exception in looking up space point ptr... 0x3ac4cf50, 0x3ac44cb0
Caught exception in looking up space point ptr... 0x3ac44cb0, 0x3ac4cf50
Caught exception in looking up space point ptr... 0x53fea0f0, 0x53fe9e50
Caught exception in looking up space point ptr... 0x53fe9e50, 0x53fe9d00
Caught exception in looking up space point ptr... 0x3b10c620, 0xeba18a0
Caught exception in looking up space point ptr... 0xeba18a0, 0x3b10c620
Caught exception in looking up space point ptr... 0x36b154f0, 0x36b153f0
Caught exception in looking up space point ptr... 0x36b153f0, 0x36b155f0
Caught exception in looking up space point ptr... 0x3b3ba570, 0x3b3ba200
Caught exception in looking up space point ptr... 0x3b3ba200, 0x3b3ba3e0
Caught exception in looking up space point ptr... 0x816d4bd0, 0x53f56770
Caught exception in looking up space point ptr... 0x53f56770, 0x816d4bd0
Caught exception in looking up space point ptr... 0x816d4bd0, 0x53f56030
Caught exception in looking up space point ptr... 0x53f56030, 0x816d4bd0
++++>>>> total num hits: 80069, num free: 76339
04-Feb-2026 01:49:30 CET  Closed output file "np02vd_raw_run042433_0223_df-s04-d1_dw_0_20260203T200653_reco_stage0_20260204T004930_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         110.821       122.668        151.1        121.062       8.68813        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.306e-05    9.99413e-05   0.000187562   7.8791e-05    4.24716e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.77708       6.32155       7.07466       6.27976      0.384792        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0113788     0.0289483     0.100772      0.0163682     0.0228743       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000404335   0.000553952   0.000988312   0.000529366   0.000127195      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000374574   0.00052242    0.000983862   0.000505866   0.000131358      19     
produce:beamevent:BeamEvent                       5.172e-05    0.000132996   0.000682189   0.000105512   0.000131499      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.98262       2.58058       3.22828       2.52308      0.295338        19     
produce:ophit:OpHitFinder                         0.021725      0.0318438     0.0448577     0.0303893    0.00610003       19     
produce:wclsdatavd:WireCellToolkit                 87.0646       92.1193       110.282       91.3393       5.11414        19     
produce:gaushit:GausHitFinder                     0.775553       1.10832       1.85176       1.02025      0.269429        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000194223   0.000289538   0.000590578   0.000279463   8.2002e-05       19     
produce:reco3d:SpacePointSolver                    7.49235       13.701        22.2708       12.9929       3.60021        19     
produce:hitpdune:DisambigFromSpacePoints          0.0914936      0.16235      0.336781      0.142795      0.0532952       19     
produce:cluster3d:Cluster3D                       0.920716       2.10244       4.25108       1.76459      0.864789        19     
[art]:TriggerResults:TriggerResultInserter        2.197e-05    3.02783e-05   7.6561e-05     2.569e-05    1.19178e-05      19     
end_path:out1:RootOutput                          3.73e-06     8.74547e-06    2.068e-05     8.12e-06     3.70192e-06      19     
end_path:out1:RootOutput(write)                    3.90286       4.48265       6.23618       4.23361      0.605334        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5394.61 MB
  Peak resident set size usage (VmHWM): 3403.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0223_df-s04-d1_dw_0_20260203T200653_reco_stage0_20260204T004930_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0223_df-s04-d1_dw_0_20260203T200653_reco_stage0_20260204T004930_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0223_df-s04-d1_dw_0_20260203T200653_reco_stage0_20260204T004930_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163800.0,
    "core.end_time": 1770166171.0,
    "core.events": [
      25433,
      25439,
      25445,
      25451,
      25457,
      25463,
      25469,
      25475,
      25481,
      25487,
      25493,
      25499,
      25505,
      25511,
      25517,
      25523,
      25529,
      25535,
      25541
    ],
    "core.event_count": 19,
    "core.first_event_number": 25433,
    "core.last_event_number": 25541,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0223_df-s04-d1_dw_0_20260203T200653.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:56 UTC       justIN version: 01.06.00