justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.33@dunegpschedd02.fnal.gov

Jobsub ID297921.33@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod02
Last heartbeat2026-02-04 01:24:03
From worker nodeHostnamebf1770081312.novalocal
cpuinfoAMD EPYC-Genoa Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:47:03
Input filesvd-protodune:np02vd_raw_run042433_0049_df-s04-d0_dw_0_20260203T190044.hdf5
JobscriptExit code0
Real time1h (5619s)
CPU time38m (2318s = 41%)
Max RSS bytes3433390080 (3274 MiB)
Outputting started2026-02-04 01:20:42
Output files
Finished2026-02-04 01:24:03
Saved logsjustin-logs:297921.33-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ing up space point ptr... 0x5934f7b0, 0x5934f640
Caught exception in looking up space point ptr... 0x5934f640, 0x2a463a80
Caught exception in looking up space point ptr... 0x2a463a80, 0x3a8e3660
Caught exception in looking up space point ptr... 0x3bc3bce0, 0x3bc35f60
Caught exception in looking up space point ptr... 0x29d95f50, 0x3bc3bce0
Caught exception in looking up space point ptr... 0x422937f0, 0x47bbed40
Caught exception in looking up space point ptr... 0x47bbed40, 0x76bef590
Caught exception in looking up space point ptr... 0x36d050a0, 0x36d04ec0
Caught exception in looking up space point ptr... 0x36d04ec0, 0x36d050a0
Caught exception in looking up space point ptr... 0x36d050a0, 0x3b2c7970
Caught exception in looking up space point ptr... 0x3b2c7970, 0x36d050a0
Caught exception in looking up space point ptr... 0x70b31460, 0x70b68400
Caught exception in looking up space point ptr... 0x70b68400, 0x70b68270
Caught exception in looking up space point ptr... 0x70b68270, 0x70b6d810
Caught exception in looking up space point ptr... 0x70b6d810, 0x70b31460
Caught exception in looking up space point ptr... 0x7cca0b00, 0x1a095a90
Caught exception in looking up space point ptr... 0x5b733ea0, 0x7cca0b00
Caught exception in looking up space point ptr... 0x2ded45d0, 0x1814b770
Caught exception in looking up space point ptr... 0x1814b770, 0x2ded45d0
Caught exception in looking up space point ptr... 0x7b9c2130, 0x7b9c1d40
Caught exception in looking up space point ptr... 0x7b9c1d40, 0x7b9c2130
Caught exception in looking up space point ptr... 0x37230e00, 0x37230cb0
Caught exception in looking up space point ptr... 0x37230cb0, 0x37230e00
Caught exception in looking up space point ptr... 0x4c087b30, 0x4c0879a0
Caught exception in looking up space point ptr... 0x4c0879a0, 0x4c087b30
Caught exception in looking up space point ptr... 0x3d336fe0, 0x76ed24a0
Caught exception in looking up space point ptr... 0x76ed24a0, 0x3d336fe0
Caught exception in looking up space point ptr... 0xcd7a0b0, 0xcd7aa60
Caught exception in looking up space point ptr... 0x36c1c300, 0xcd7a0b0
Caught exception in looking up space point ptr... 0x17986610, 0x179872b0
Caught exception in looking up space point ptr... 0x179872b0, 0x17986b90
Caught exception in looking up space point ptr... 0x58886be0, 0x58887080
Caught exception in looking up space point ptr... 0x58887080, 0x58886e30
Caught exception in looking up space point ptr... 0x3723f750, 0x3723eb30
Caught exception in looking up space point ptr... 0x3723eb30, 0x3723f750
Caught exception in looking up space point ptr... 0x47bbb900, 0x47bbb770
Caught exception in looking up space point ptr... 0x47bbb770, 0x47bbbc20
Caught exception in looking up space point ptr... 0x29de15f0, 0x608d5300
Caught exception in looking up space point ptr... 0x608d5300, 0x608d38e0
Caught exception in looking up space point ptr... 0x608d38e0, 0x608d3790
Caught exception in looking up space point ptr... 0x608d57c0, 0x29de15f0
Caught exception in looking up space point ptr... 0x37fd8480, 0x610e75d0
Caught exception in looking up space point ptr... 0x610e75d0, 0x16698420
Caught exception in looking up space point ptr... 0x7cc9c6d0, 0x7cc7de00
Caught exception in looking up space point ptr... 0x7cc7de00, 0x7cc9c6d0
Caught exception in looking up space point ptr... 0x50bc61c0, 0x50bc6070
Caught exception in looking up space point ptr... 0x50bc6070, 0x50bc61c0
Caught exception in looking up space point ptr... 0x60d8f170, 0x36c29d60
Caught exception in looking up space point ptr... 0x36c29d60, 0x60d8f170
Caught exception in looking up space point ptr... 0x70d0e700, 0x70d0c740
Caught exception in looking up space point ptr... 0x70d0c740, 0x70d0e700
Caught exception in looking up space point ptr... 0x5b87af70, 0x62ec3ce0
Caught exception in looking up space point ptr... 0x62ec3ce0, 0x6cc34210
Caught exception in looking up space point ptr... 0x36c2b4e0, 0x6bb238e0
Caught exception in looking up space point ptr... 0x6bb238e0, 0x2c243b30
Caught exception in looking up space point ptr... 0x35f94270, 0x35f94340
Caught exception in looking up space point ptr... 0x35f94340, 0x35f94270
Caught exception in looking up space point ptr... 0x385f8710, 0x385f66d0
Caught exception in looking up space point ptr... 0x385f66d0, 0x385f8710
Caught exception in looking up space point ptr... 0x13a8b190, 0x52e8c740
Caught exception in looking up space point ptr... 0x52e8c740, 0x13a8b190
Caught exception in looking up space point ptr... 0x5b711ed0, 0x5b711cd0
Caught exception in looking up space point ptr... 0x5b711cd0, 0x5b711ed0
Caught exception in looking up space point ptr... 0x7b9b2990, 0x7b9b32d0
Caught exception in looking up space point ptr... 0x7b9b32d0, 0x7b9b3d40
Caught exception in looking up space point ptr... 0x7b9af050, 0x7b9b1990
Caught exception in looking up space point ptr... 0x7b9b1990, 0x7b9b1ab0
++++>>>> total num hits: 83041, num free: 80657
04-Feb-2026 01:20:33 GMT  Closed output file "np02vd_raw_run042433_0049_df-s04-d0_dw_0_20260203T190044_reco_stage0_20260204T012033_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         238.077       292.235       375.128       282.419       37.1463        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.1249e-05    0.000119396   0.000363757   9.3499e-05    7.54204e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                120.896       170.284       241.415       161.761       32.5716        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.576893      0.888121       1.54446      0.843646      0.288675        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0172303     0.0197489     0.0343736     0.0190242    0.00361613       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0171937     0.0314906     0.262735      0.0183229     0.0545162       19     
produce:beamevent:BeamEvent                      7.3929e-05    0.000193862   0.00107697    0.000124159   0.000219231      19     
produce:pdvddaphne:DAPHNEReaderPDVD                7.82995       14.0491       18.9947       14.3116       2.92421        19     
produce:ophit:OpHitFinder                         0.0216366     0.0344087     0.0711682     0.0341913    0.00966876       19     
produce:wclsdatavd:WireCellToolkit                 79.662        83.8397       87.4672       83.5086       2.32299        19     
produce:gaushit:GausHitFinder                     0.626637       1.0348        1.61249       1.00605      0.226247        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000151119   0.000285011   0.000773245   0.000233908   0.000142976      19     
produce:reco3d:SpacePointSolver                    8.80992       15.3037       25.8339       15.8255       4.72941        19     
produce:hitpdune:DisambigFromSpacePoints          0.0754667     0.153616      0.263716      0.150144      0.0394206       19     
produce:cluster3d:Cluster3D                        1.30213       2.32411       4.52836       2.1492       0.895932        19     
[art]:TriggerResults:TriggerResultInserter       1.8811e-05    3.58652e-05   0.00013897    2.1809e-05    3.58066e-05      19     
end_path:out1:RootOutput                          3.18e-06     1.86267e-05   0.000277049    3.72e-06     6.09694e-05      19     
end_path:out1:RootOutput(write)                    3.72687       4.22165       4.82747       4.21595      0.285169        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5341.87 MB
  Peak resident set size usage (VmHWM): 3433.39 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0049_df-s04-d0_dw_0_20260203T190044_reco_stage0_20260204T012033_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0049_df-s04-d0_dw_0_20260203T190044_reco_stage0_20260204T012033_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0049_df-s04-d0_dw_0_20260203T190044_reco_stage0_20260204T012033_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162434.0,
    "core.end_time": 1770168034.0,
    "core.events": [
      5596,
      5602,
      5608,
      5614,
      5620,
      5626,
      5632,
      5638,
      5644,
      5650,
      5656,
      5662,
      5668,
      5674,
      5680,
      5686,
      5692,
      5698,
      5704
    ],
    "core.event_count": 19,
    "core.first_event_number": 5596,
    "core.last_event_number": 5704,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0049_df-s04-d0_dw_0_20260203T190044.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:12 UTC       justIN version: 01.06.00