justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298045.2@dunegpschedd02.fnal.gov

Jobsub ID298045.2@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 02:22:33
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 05:16:50
From worker nodeHostnamewn2205241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 02:23:27
Input filesvd-protodune:np02vd_raw_run042433_0073_df-s03-d1_dw_0_20260203T190952.hdf5
JobscriptExit code0
Real time2h (10218s)
CPU time1h (4840s = 47%)
Max RSS bytes11983024128 (11427 MiB)
Outputting started2026-02-04 05:13:50
Output files
Finished2026-02-04 05:16:50
Saved logsjustin-logs:298045.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ng up space point ptr... 0x2c9e0500, 0x2cc893a0
Caught exception in looking up space point ptr... 0x2cc893a0, 0x2cc89530
Caught exception in looking up space point ptr... 0x642d3110, 0x642d32a0
Caught exception in looking up space point ptr... 0x642d32a0, 0x6427a340
Caught exception in looking up space point ptr... 0x978b1fe0, 0x5ca22620
Caught exception in looking up space point ptr... 0x5ca22620, 0x978b1fe0
Caught exception in looking up space point ptr... 0xade6ef50, 0xade6edc0
Caught exception in looking up space point ptr... 0xade6edc0, 0xade6ef50
Caught exception in looking up space point ptr... 0xa665ed70, 0xa6bfdaa0
Caught exception in looking up space point ptr... 0xa6bfdaa0, 0xa665f090
Caught exception in looking up space point ptr... 0xa665f090, 0xa6bfdc30
Caught exception in looking up space point ptr... 0xce493eb0, 0xce25a260
Caught exception in looking up space point ptr... 0xce25a260, 0xce493eb0
Caught exception in looking up space point ptr... 0xce946c20, 0xce906f40
Caught exception in looking up space point ptr... 0xce8b8b90, 0xce946c20
Caught exception in looking up space point ptr... 0x9ef18f90, 0xce263090
Caught exception in looking up space point ptr... 0xce263090, 0xce813fc0
Caught exception in looking up space point ptr... 0xb21f53d0, 0xb21f5560
Caught exception in looking up space point ptr... 0xb21f5560, 0x650b15b0
Caught exception in looking up space point ptr... 0x431cbee0, 0x431cbd50
Caught exception in looking up space point ptr... 0x431cbd50, 0x431cbbc0
Caught exception in looking up space point ptr... 0x431c33d0, 0x431c36f0
Caught exception in looking up space point ptr... 0x431c36f0, 0x431c3880
Caught exception in looking up space point ptr... 0x685def20, 0x685ded40
Caught exception in looking up space point ptr... 0x685ded40, 0x685def20
Caught exception in looking up space point ptr... 0x685def20, 0x683eb700
Caught exception in looking up space point ptr... 0x683eb700, 0x685def20
Caught exception in looking up space point ptr... 0x8047af00, 0x59e6ec90
Caught exception in looking up space point ptr... 0x59e6ec90, 0x8047af00
Caught exception in looking up space point ptr... 0xa99ca4c0, 0xeaec0df0
Caught exception in looking up space point ptr... 0xeaec0df0, 0xeae5f2a0
Caught exception in looking up space point ptr... 0xce404da0, 0xce292020
Caught exception in looking up space point ptr... 0xce292020, 0xce404da0
Caught exception in looking up space point ptr... 0x2c9fedd0, 0x2c9f3fc0
Caught exception in looking up space point ptr... 0x2c9f3fc0, 0x2c9fedd0
Caught exception in looking up space point ptr... 0x19a765c0, 0x3c4cb8b0
Caught exception in looking up space point ptr... 0x3c4cb8b0, 0x19a765c0
Caught exception in looking up space point ptr... 0x2ec0f8e0, 0x73aff4b0
Caught exception in looking up space point ptr... 0x73aff4b0, 0x2ec08d10
Caught exception in looking up space point ptr... 0x3cabea80, 0x818b1450
Caught exception in looking up space point ptr... 0x818b1450, 0x3cabea80
Caught exception in looking up space point ptr... 0x83c30840, 0x83994440
Caught exception in looking up space point ptr... 0x83994440, 0x83ac91f0
Caught exception in looking up space point ptr... 0x67db33f0, 0x9d22cbd0
Caught exception in looking up space point ptr... 0x67db2900, 0x67db33f0
Caught exception in looking up space point ptr... 0x67fed380, 0x676f6af0
Caught exception in looking up space point ptr... 0x676f6af0, 0x67fedce0
Caught exception in looking up space point ptr... 0x4dc8d0a0, 0xe1a5750
Caught exception in looking up space point ptr... 0xe1a5750, 0x4dc8d0a0
Caught exception in looking up space point ptr... 0x292fb4e0, 0x5334f550
Caught exception in looking up space point ptr... 0x5334f550, 0x292fb4e0
Caught exception in looking up space point ptr... 0x7e4b6770, 0xa16bf4c0
Caught exception in looking up space point ptr... 0xa16bf4c0, 0x7e4b6770
Caught exception in looking up space point ptr... 0x7c202780, 0x72f48be0
Caught exception in looking up space point ptr... 0x72f48be0, 0x7c202780
Caught exception in looking up space point ptr... 0xce4aa070, 0xce26bec0
Caught exception in looking up space point ptr... 0xce26bec0, 0xce4aa070
Caught exception in looking up space point ptr... 0x15b6aee0, 0x2f21db50
Caught exception in looking up space point ptr... 0x2f21db50, 0x14629750
Caught exception in looking up space point ptr... 0x8078fa20, 0x620d7910
Caught exception in looking up space point ptr... 0x620d7910, 0x8078fc20
Caught exception in looking up space point ptr... 0xa1b179e0, 0xa1c863e0
Caught exception in looking up space point ptr... 0xa1c863e0, 0xa1c86570
Caught exception in looking up space point ptr... 0x43168cf0, 0x43168b60
Caught exception in looking up space point ptr... 0x43168b60, 0x43168e80
Caught exception in looking up space point ptr... 0xa3afca20, 0xa1d0e4e0
Caught exception in looking up space point ptr... 0xa1d0e4e0, 0xa3afca20
++++>>>> total num hits: 5205795, num free: 5089003
04-Feb-2026 05:13:37 GMT  Closed output file "np02vd_raw_run042433_0073_df-s03-d1_dw_0_20260203T190952_reco_stage0_20260204T051337_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         342.861       532.036       1553.55       470.952       261.54         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.8737e-05    0.000179053   0.000326102   0.00015973    7.45638e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                157.792       263.339       354.751       262.803       62.4834        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.699516       1.11225       1.56335       1.1413       0.238935        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0224773     0.0277015     0.0738182     0.022968      0.0126877       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0222561     0.0226455     0.0233396     0.0225358    0.000311294      19     
produce:beamevent:BeamEvent                      0.000130405   0.000311924   0.00162621    0.000222177   0.000317261      19     
produce:pdvddaphne:DAPHNEReaderPDVD                14.1266       23.0008       33.4378       24.0298       5.37836        19     
produce:ophit:OpHitFinder                         0.0334397     0.043413      0.0526443     0.042872     0.00596757       19     
produce:wclsdatavd:WireCellToolkit                 103.798       132.693       157.133       135.72        15.5258        19     
produce:gaushit:GausHitFinder                     0.798328       2.35955       9.84542       1.68518       2.28213        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000265468   0.000539374   0.00181325    0.000447941   0.000340742      19     
produce:reco3d:SpacePointSolver                    8.32497       27.1163       108.505       19.1475       22.6861        19     
produce:hitpdune:DisambigFromSpacePoints          0.0996865      0.5874        4.62584      0.254987       1.06672        19     
produce:cluster3d:Cluster3D                        1.5892        74.0359       911.679       4.59379       218.888        19     
[art]:TriggerResults:TriggerResultInserter       3.0316e-05    5.26051e-05   0.000134332   4.3111e-05    2.45344e-05      19     
end_path:out1:RootOutput                          4.969e-06    1.24565e-05   4.2249e-05     9.097e-06    9.61533e-06      19     
end_path:out1:RootOutput(write)                    4.70316       7.63156       23.3355       6.64199       4.43273        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14148 MB
  Peak resident set size usage (VmHWM): 11983 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0073_df-s03-d1_dw_0_20260203T190952_reco_stage0_20260204T051337_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0073_df-s03-d1_dw_0_20260203T190952_reco_stage0_20260204T051337_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0073_df-s03-d1_dw_0_20260203T190952_reco_stage0_20260204T051337_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770171819.0,
    "core.end_time": 1770182019.0,
    "core.events": [
      8331,
      8337,
      8343,
      8349,
      8355,
      8361,
      8367,
      8373,
      8379,
      8385,
      8391,
      8397,
      8403,
      8409,
      8415,
      8421,
      8427,
      8433,
      8439
    ],
    "core.event_count": 19,
    "core.first_event_number": 8331,
    "core.last_event_number": 8439,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0073_df-s03-d1_dw_0_20260203T190952.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:44:00 UTC       justIN version: 01.06.00