justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.89@dunegpschedd02.fnal.gov

Jobsub ID297921.89@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:11:50
From worker nodeHostnamet2wn013.physics.ox.ac.uk
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:55:23
Input filesvd-protodune:np02vd_raw_run042433_0336_df-s04-d0_dw_0_20260203T204950.hdf5
JobscriptExit code0
Real time1h (4380s)
CPU time1h (3925s = 89%)
Max RSS bytes3339124736 (3184 MiB)
Outputting started2026-02-04 01:08:24
Output files
Finished2026-02-04 01:11:50
Saved logsjustin-logs:297921.89-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

p space point ptr... 0x455942b0, 0x45594560
Caught exception in looking up space point ptr... 0xa8995d0, 0x2a006f70
Caught exception in looking up space point ptr... 0x84fb5e10, 0xa8995d0
Caught exception in looking up space point ptr... 0x62776930, 0x4b88e270
Caught exception in looking up space point ptr... 0x4b88e270, 0x60be6c80
Caught exception in looking up space point ptr... 0xa8ab400, 0xa8c3100
Caught exception in looking up space point ptr... 0xa8c3100, 0xa8ab400
Caught exception in looking up space point ptr... 0x199cb6f0, 0x199cb7f0
Caught exception in looking up space point ptr... 0x199cb7f0, 0x5afdc020
Caught exception in looking up space point ptr... 0x3f1dd580, 0x10e081f0
Caught exception in looking up space point ptr... 0x10e081f0, 0x3f1dd580
Caught exception in looking up space point ptr... 0x626d1ae0, 0x626d49b0
Caught exception in looking up space point ptr... 0x626d49b0, 0x626d4ab0
Caught exception in looking up space point ptr... 0x9eeb470, 0x9ee0bb0
Caught exception in looking up space point ptr... 0x9ee0bb0, 0x9eeb470
Caught exception in looking up space point ptr... 0x6586fc60, 0x54c6d640
Caught exception in looking up space point ptr... 0x54c6d640, 0x47ab6ed0
Caught exception in looking up space point ptr... 0x3147ed10, 0x39659030
Caught exception in looking up space point ptr... 0x39659030, 0x3147ed10
Caught exception in looking up space point ptr... 0x3147ed10, 0x396404a0
Caught exception in looking up space point ptr... 0x396404a0, 0x3147ed10
Caught exception in looking up space point ptr... 0x4de2a7c0, 0x4de2a3e0
Caught exception in looking up space point ptr... 0x4de2a3e0, 0x4de2a7c0
Caught exception in looking up space point ptr... 0x16bdf6d0, 0x16bdf3b0
Caught exception in looking up space point ptr... 0x16bdf3b0, 0x16bdf220
Caught exception in looking up space point ptr... 0x6dec05b0, 0x6dec6970
Caught exception in looking up space point ptr... 0x6dec6970, 0x6dec05b0
Caught exception in looking up space point ptr... 0xdc9df80, 0x4aadba90
Caught exception in looking up space point ptr... 0x4aadba90, 0xdc9df80
Caught exception in looking up space point ptr... 0x4de33c70, 0x60831940
Caught exception in looking up space point ptr... 0x60831940, 0x4de33c70
Caught exception in looking up space point ptr... 0x8ddddf0, 0x28e50cc0
Caught exception in looking up space point ptr... 0x28e50cc0, 0x51730510
Caught exception in looking up space point ptr... 0x3957b390, 0x3957b700
Caught exception in looking up space point ptr... 0x3957a6f0, 0x3957b390
Caught exception in looking up space point ptr... 0x4de5f990, 0x84ea3dd0
Caught exception in looking up space point ptr... 0x84ea3dd0, 0x84ea21e0
Caught exception in looking up space point ptr... 0x36a5b600, 0x612df800
Caught exception in looking up space point ptr... 0x612df800, 0x5f74ad30
Caught exception in looking up space point ptr... 0x5debcd30, 0x5debd010
Caught exception in looking up space point ptr... 0x5debd010, 0x5debd2f0
Caught exception in looking up space point ptr... 0x79e8c350, 0x79e8a760
Caught exception in looking up space point ptr... 0x79e89e30, 0x79e8c350
Caught exception in looking up space point ptr... 0x37941a90, 0x37944550
Caught exception in looking up space point ptr... 0x37944550, 0x37944820
Caught exception in looking up space point ptr... 0x589cf6e0, 0x589cf370
Caught exception in looking up space point ptr... 0x589cf370, 0x589cf6e0
Caught exception in looking up space point ptr... 0x5899ad40, 0x17be6dc0
Caught exception in looking up space point ptr... 0x17be6dc0, 0x5899ad40
Caught exception in looking up space point ptr... 0x395791f0, 0x395790a0
Caught exception in looking up space point ptr... 0x395790a0, 0x395791f0
Caught exception in looking up space point ptr... 0x5b180a00, 0x5b180b20
Caught exception in looking up space point ptr... 0x5b180b20, 0x5b180a00
Caught exception in looking up space point ptr... 0x589c2530, 0x589b8010
Caught exception in looking up space point ptr... 0x589b8010, 0x589c2530
Caught exception in looking up space point ptr... 0x4bd054d0, 0x4bd051e0
Caught exception in looking up space point ptr... 0x4bd051e0, 0x4bd05380
Caught exception in looking up space point ptr... 0xeceb3a0, 0xeceb530
Caught exception in looking up space point ptr... 0xeceb530, 0xeceb3a0
Caught exception in looking up space point ptr... 0x613ebac0, 0x48eb4fa0
Caught exception in looking up space point ptr... 0x48eb4fa0, 0x39b50d50
Caught exception in looking up space point ptr... 0x9c166e0, 0x9c16a00
Caught exception in looking up space point ptr... 0x9c16a00, 0x9c166e0
Caught exception in looking up space point ptr... 0x185f77d0, 0x185f7c90
Caught exception in looking up space point ptr... 0x185f7c90, 0x185f8730
Caught exception in looking up space point ptr... 0x185d6f60, 0x5899c040
Caught exception in looking up space point ptr... 0x5899c040, 0x17c17680
++++>>>> total num hits: 130290, num free: 124951
04-Feb-2026 01:08:13 GMT  Closed output file "np02vd_raw_run042433_0336_df-s04-d0_dw_0_20260203T204950_reco_stage0_20260204T010813_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         198.494       225.109       275.009       224.505       20.7647        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.4515e-05    0.000109147   0.000284029   9.6337e-05    4.40652e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                18.4861       22.902        38.1016       22.1277       3.99921        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.599763      0.674556      0.869374      0.657372      0.070238        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0194235     0.0383497     0.317632      0.0201025     0.065975        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0193509     0.0195642     0.0198357      0.01955     0.000136473      19     
produce:beamevent:BeamEvent                      9.6316e-05    0.000174585   0.00109431    0.000122938   0.000217451      19     
produce:pdvddaphne:DAPHNEReaderPDVD                4.17528       4.8089        5.72897       4.61727      0.450773        19     
produce:ophit:OpHitFinder                         0.0290921     0.049867      0.057987      0.0533511    0.00749544       19     
produce:wclsdatavd:WireCellToolkit                 134.707       155.41        181.595       151.348       14.033         19     
produce:gaushit:GausHitFinder                      1.22354       2.13661       3.34942       2.08127      0.544321        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000275733   0.000502308   0.00082217    0.000483956   0.00013683       19     
produce:reco3d:SpacePointSolver                    16.4994       27.3158       54.052        25.6623       9.41863        19     
produce:hitpdune:DisambigFromSpacePoints          0.152289      0.284666      0.459345      0.284344      0.0760848       19     
produce:cluster3d:Cluster3D                        2.40163       4.02783       6.95403       3.44173       1.26974        19     
[art]:TriggerResults:TriggerResultInserter       2.4838e-05    3.66467e-05    9.309e-05    3.2793e-05    1.45822e-05      19     
end_path:out1:RootOutput                          3.957e-06    1.80948e-05   0.000232499    5.46e-06     5.07137e-05      19     
end_path:out1:RootOutput(write)                    5.3947        7.38765       9.13898       7.57758      0.986623        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5387.74 MB
  Peak resident set size usage (VmHWM): 3339.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0336_df-s04-d0_dw_0_20260203T204950_reco_stage0_20260204T010813_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0336_df-s04-d0_dw_0_20260203T204950_reco_stage0_20260204T010813_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0336_df-s04-d0_dw_0_20260203T204950_reco_stage0_20260204T010813_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162939.0,
    "core.end_time": 1770167295.0,
    "core.events": [
      38314,
      38320,
      38326,
      38332,
      38338,
      38344,
      38350,
      38356,
      38362,
      38368,
      38374,
      38380,
      38386,
      38392,
      38398,
      38404,
      38410,
      38416,
      38422
    ],
    "core.event_count": 19,
    "core.first_event_number": 38314,
    "core.last_event_number": 38422,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0336_df-s04-d0_dw_0_20260203T204950.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:04 UTC       justIN version: 01.06.00