justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.76@dunegpschedd02.fnal.gov

Jobsub ID297921.76@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 01:06:48
From worker nodeHostnameb9p06p4022.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:52:40
Input filesvd-protodune:np02vd_raw_run042433_0010_df-s04-d0_dw_0_20260203T184554.hdf5
JobscriptExit code0
Real time1h (4300s)
CPU time1h (4132s = 96%)
Max RSS bytes3424657408 (3266 MiB)
Outputting started2026-02-04 01:04:22
Output files
Finished2026-02-04 01:06:48
Saved logsjustin-logs:297921.76-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

n looking up space point ptr... 0xb89b3b0, 0xdce2cc0
Caught exception in looking up space point ptr... 0xdce2cc0, 0xdce2e50
Caught exception in looking up space point ptr... 0xdce2e50, 0xb89be80
Caught exception in looking up space point ptr... 0x710a6100, 0x30213f30
Caught exception in looking up space point ptr... 0x30213f30, 0x710a59c0
Caught exception in looking up space point ptr... 0x710a59c0, 0x710a5c10
Caught exception in looking up space point ptr... 0x710a5c10, 0x710a5e60
Caught exception in looking up space point ptr... 0x710a5e60, 0x710a6b50
Caught exception in looking up space point ptr... 0x30215170, 0x710a6100
Caught exception in looking up space point ptr... 0x40b69520, 0x586f0d10
Caught exception in looking up space point ptr... 0x586f0d10, 0x65c8f1e0
Caught exception in looking up space point ptr... 0x65c8f1e0, 0x1661f130
Caught exception in looking up space point ptr... 0x1661f130, 0x72504c70
Caught exception in looking up space point ptr... 0x72504c70, 0x72504f60
Caught exception in looking up space point ptr... 0x72504f60, 0x60f78ed0
Caught exception in looking up space point ptr... 0x116eff80, 0x375adaa0
Caught exception in looking up space point ptr... 0x375adaa0, 0x61dd4bc0
Caught exception in looking up space point ptr... 0x61dd4bc0, 0x158681b0
Caught exception in looking up space point ptr... 0x158681b0, 0x397890d0
Caught exception in looking up space point ptr... 0x32ee5cd0, 0x4082a680
Caught exception in looking up space point ptr... 0x4082a680, 0x32ee5cd0
Caught exception in looking up space point ptr... 0x1a81edb0, 0x32887ef0
Caught exception in looking up space point ptr... 0x32887ef0, 0x8fd00c0
Caught exception in looking up space point ptr... 0x40774200, 0x39099400
Caught exception in looking up space point ptr... 0x39099400, 0x40774200
Caught exception in looking up space point ptr... 0x52ec9f20, 0x4da11a30
Caught exception in looking up space point ptr... 0x4da11a30, 0x4da10f70
Caught exception in looking up space point ptr... 0x782b5f40, 0x782b60d0
Caught exception in looking up space point ptr... 0x782b8630, 0x782b5f40
Caught exception in looking up space point ptr... 0x5906dcf0, 0x5906e130
Caught exception in looking up space point ptr... 0x5906e130, 0x5906d450
Caught exception in looking up space point ptr... 0x4794c030, 0x4794c270
Caught exception in looking up space point ptr... 0x4794c270, 0x4e2ca930
Caught exception in looking up space point ptr... 0x5c6f4950, 0x60cfcc30
Caught exception in looking up space point ptr... 0x60cfcc30, 0x5c6f4950
Caught exception in looking up space point ptr... 0x3a10a390, 0x39a0f7a0
Caught exception in looking up space point ptr... 0x39a0f7a0, 0x2c010890
Caught exception in looking up space point ptr... 0x2b4accd0, 0x38cef950
Caught exception in looking up space point ptr... 0x38cef950, 0x39a145b0
Caught exception in looking up space point ptr... 0xf19b7a0, 0x399f0a20
Caught exception in looking up space point ptr... 0x399f0a20, 0x2b063490
Caught exception in looking up space point ptr... 0x58aebf60, 0x2cef6420
Caught exception in looking up space point ptr... 0x1658d6f0, 0x58aebf60
Caught exception in looking up space point ptr... 0x1b10d210, 0x1b10d310
Caught exception in looking up space point ptr... 0x1b10d310, 0x379c1930
Caught exception in looking up space point ptr... 0x6f0efff0, 0x6f0f1020
Caught exception in looking up space point ptr... 0x6f0f1020, 0x6f0f0990
Caught exception in looking up space point ptr... 0x63c76b50, 0x6584e340
Caught exception in looking up space point ptr... 0x6584e340, 0x63c76da0
Caught exception in looking up space point ptr... 0x190eb960, 0x4aa78b80
Caught exception in looking up space point ptr... 0x4aa78b80, 0x3a07c650
Caught exception in looking up space point ptr... 0x3c5fef80, 0x39148770
Caught exception in looking up space point ptr... 0x3de96d90, 0x3c5fef80
Caught exception in looking up space point ptr... 0x3a990770, 0x43085100
Caught exception in looking up space point ptr... 0x43085100, 0x3a990770
Caught exception in looking up space point ptr... 0x7e25d910, 0x3284f3d0
Caught exception in looking up space point ptr... 0x3284f3d0, 0x7e25d910
Caught exception in looking up space point ptr... 0x38397ae0, 0x397c4f20
Caught exception in looking up space point ptr... 0x397c4f20, 0x38397ae0
Caught exception in looking up space point ptr... 0x2b75f2d0, 0x2b75f5d0
Caught exception in looking up space point ptr... 0x2b75f5d0, 0x2b75f700
Caught exception in looking up space point ptr... 0x40710ef0, 0x3c4462d0
Caught exception in looking up space point ptr... 0x3c4462d0, 0x40710ef0
Caught exception in looking up space point ptr... 0x86b02c0, 0x5f171dc0
Caught exception in looking up space point ptr... 0x5f171dc0, 0x17296ea0
Caught exception in looking up space point ptr... 0x656a0d80, 0x656a0110
Caught exception in looking up space point ptr... 0x656a0110, 0x656a01b0
++++>>>> total num hits: 42164, num free: 40900
04-Feb-2026 02:04:14 CET  Closed output file "np02vd_raw_run042433_0010_df-s04-d0_dw_0_20260203T184554_reco_stage0_20260204T010414_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         126.398       220.156       269.94        233.716       42.0161        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.6761e-05    0.00025827     0.0006941    0.000206353   0.000154285      19     
produce:tpcrawdecoder:PDVDTPCReader                6.1508        9.65233       20.3199        9.253        2.97199        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0142267     0.0332308     0.0811199     0.0318255     0.0165402       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000477777   0.00172497     0.0102441    0.00113323    0.00211438       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000398836   0.00117666    0.00273298     0.0010659    0.00067204       19     
produce:beamevent:BeamEvent                      7.5732e-05    0.000361085   0.00189171    0.000196493   0.000432495      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.26075       3.80476       4.81156       3.86289      0.678144        19     
produce:ophit:OpHitFinder                         0.0323848     0.0528986     0.0895649     0.0478462     0.0166783       19     
produce:wclsdatavd:WireCellToolkit                 89.6691       166.529       203.626       177.053       35.6532        19     
produce:gaushit:GausHitFinder                     0.708667       2.09357       3.18895       2.0659       0.685778        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000217663   0.00154682    0.00412131    0.000610199   0.00141588       19     
produce:reco3d:SpacePointSolver                    7.98394       26.1875       43.0856       25.7677       8.41037        19     
produce:hitpdune:DisambigFromSpacePoints          0.0816974     0.349491      0.993665      0.312983      0.195135        19     
produce:cluster3d:Cluster3D                       0.912354       4.51114       10.4362       4.10984       2.32945        19     
[art]:TriggerResults:TriggerResultInserter       2.4401e-05    7.42406e-05   0.000270224   4.5351e-05    6.76273e-05      19     
end_path:out1:RootOutput                          3.22e-06     1.58018e-05   4.6971e-05     9.61e-06     1.37478e-05      19     
end_path:out1:RootOutput(write)                    3.98637       6.84605       8.77451       7.20531       1.48527        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5397.98 MB
  Peak resident set size usage (VmHWM): 3424.66 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0010_df-s04-d0_dw_0_20260203T184554_reco_stage0_20260204T010414_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0010_df-s04-d0_dw_0_20260203T184554_reco_stage0_20260204T010414_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0010_df-s04-d0_dw_0_20260203T184554_reco_stage0_20260204T010414_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162780.0,
    "core.end_time": 1770167056.0,
    "core.events": [
      1150,
      1156,
      1162,
      1168,
      1174,
      1180,
      1186,
      1192,
      1198,
      1204,
      1210,
      1216,
      1222,
      1228,
      1234,
      1240,
      1246,
      1252,
      1258
    ],
    "core.event_count": 19,
    "core.first_event_number": 1150,
    "core.last_event_number": 1258,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0010_df-s04-d0_dw_0_20260203T184554.hdf5"
    }
  ]
}
justIN time: 2026-02-04 10:45:16 UTC       justIN version: 01.06.00