justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.21@dunegpschedd02.fnal.gov

Jobsub ID297921.21@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:33:33
From worker nodeHostnamet2wn175.physics.ox.ac.uk
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:45:06
Input filesvd-protodune:np02vd_raw_run042433_0056_df-s04-d0_dw_0_20260203T190325.hdf5
JobscriptExit code0
Real time1h (6324s)
CPU time42m (2530s = 40%)
Max RSS bytes3278999552 (3127 MiB)
Outputting started2026-02-04 01:30:31
Output files
Finished2026-02-04 01:33:33
Saved logsjustin-logs:297921.21-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oking up space point ptr... 0x2b459fd0, 0x2b45a160
Caught exception in looking up space point ptr... 0x5598db30, 0x31440420
Caught exception in looking up space point ptr... 0x619f3610, 0x5598db30
Caught exception in looking up space point ptr... 0x127cce10, 0x127ccfa0
Caught exception in looking up space point ptr... 0x127ccfa0, 0x127cce10
Caught exception in looking up space point ptr... 0x5136cd40, 0x5137bcc0
Caught exception in looking up space point ptr... 0x5137bcc0, 0x5136cd40
Caught exception in looking up space point ptr... 0x4f7c0480, 0x1a5a4020
Caught exception in looking up space point ptr... 0x1a5a4020, 0x1a5a3dd0
Caught exception in looking up space point ptr... 0xd1c6100, 0xd1c6420
Caught exception in looking up space point ptr... 0xd1c6420, 0xd1c6100
Caught exception in looking up space point ptr... 0x635efe30, 0x384332c0
Caught exception in looking up space point ptr... 0x384332c0, 0x635efe30
Caught exception in looking up space point ptr... 0x171641b0, 0x1715bc30
Caught exception in looking up space point ptr... 0x1715bc30, 0x1715b8c0
Caught exception in looking up space point ptr... 0x7cd6e050, 0x77a89640
Caught exception in looking up space point ptr... 0x77a89640, 0x7cd6e050
Caught exception in looking up space point ptr... 0x4946c550, 0x49472990
Caught exception in looking up space point ptr... 0x49472990, 0x49470850
Caught exception in looking up space point ptr... 0x7cdbab70, 0x7cdb7200
Caught exception in looking up space point ptr... 0x7cdb7200, 0x7cdbab70
Caught exception in looking up space point ptr... 0x515983d0, 0x3aae8060
Caught exception in looking up space point ptr... 0x3aae8060, 0x515983d0
Caught exception in looking up space point ptr... 0x2e1028b0, 0x2e10a110
Caught exception in looking up space point ptr... 0x2e10a110, 0x2e1134d0
Caught exception in looking up space point ptr... 0x14defbd0, 0x14dee860
Caught exception in looking up space point ptr... 0x14dee860, 0x14defbd0
Caught exception in looking up space point ptr... 0x3dfa2130, 0x3dfa2bd0
Caught exception in looking up space point ptr... 0x3dfa2bd0, 0x3dfa2130
Caught exception in looking up space point ptr... 0x37563810, 0x37564150
Caught exception in looking up space point ptr... 0x37564150, 0x5d509010
Caught exception in looking up space point ptr... 0x178055c0, 0x17805750
Caught exception in looking up space point ptr... 0x5ec41f60, 0x178055c0
Caught exception in looking up space point ptr... 0x60cb4140, 0x60cb3e20
Caught exception in looking up space point ptr... 0x60cb3e20, 0x60cb4140
Caught exception in looking up space point ptr... 0x58f57b90, 0x58f58640
Caught exception in looking up space point ptr... 0x58f58640, 0x58f57b90
Caught exception in looking up space point ptr... 0x635e8050, 0x635ea570
Caught exception in looking up space point ptr... 0x635ea570, 0x635e8050
Caught exception in looking up space point ptr... 0x2ac37490, 0x3c5a4660
Caught exception in looking up space point ptr... 0x3c5a4660, 0x2ac37490
Caught exception in looking up space point ptr... 0x33ca5940, 0x33ca15e0
Caught exception in looking up space point ptr... 0x33ca15e0, 0x33ca5940
Caught exception in looking up space point ptr... 0x3144e6a0, 0x158e51c0
Caught exception in looking up space point ptr... 0x158e51c0, 0x3144e6a0
Caught exception in looking up space point ptr... 0x63bb6070, 0x63bb7e00
Caught exception in looking up space point ptr... 0x63bb7e00, 0x63bb6070
Caught exception in looking up space point ptr... 0x5d50edc0, 0x2ac23c80
Caught exception in looking up space point ptr... 0x2ac23c80, 0x5d50edc0
Caught exception in looking up space point ptr... 0x5309ed30, 0x530a0720
Caught exception in looking up space point ptr... 0x530a0720, 0x5309ed30
Caught exception in looking up space point ptr... 0x63ba2540, 0x63ba6090
Caught exception in looking up space point ptr... 0x63ba6090, 0x63bacc20
Caught exception in looking up space point ptr... 0x5f632b30, 0x600444c0
Caught exception in looking up space point ptr... 0x600444c0, 0x5f632b30
Caught exception in looking up space point ptr... 0x38083710, 0x46db93a0
Caught exception in looking up space point ptr... 0x46db93a0, 0x38083710
Caught exception in looking up space point ptr... 0x182917b0, 0x182915b0
Caught exception in looking up space point ptr... 0x182915b0, 0x182916b0
Caught exception in looking up space point ptr... 0x2eef3060, 0x2eef3290
Caught exception in looking up space point ptr... 0x2eef3290, 0x2eef3060
Caught exception in looking up space point ptr... 0x13f685b0, 0x13f661d0
Caught exception in looking up space point ptr... 0x13f661d0, 0x13f685b0
Caught exception in looking up space point ptr... 0x13f685b0, 0x13f6d330
Caught exception in looking up space point ptr... 0x13f6d330, 0x13f685b0
Caught exception in looking up space point ptr... 0x5f636300, 0x5f643600
Caught exception in looking up space point ptr... 0x5f643600, 0x5f63a470
++++>>>> total num hits: 126461, num free: 120261
04-Feb-2026 01:30:23 GMT  Closed output file "np02vd_raw_run042433_0056_df-s04-d0_dw_0_20260203T190325_reco_stage0_20260204T013023_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         271.656       328.95        411.351       319.572       41.4173        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.2009e-05    0.000107912   0.000234926    9.209e-05    3.62119e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                137.53        188.091       266.664       180.845       39.8432        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.687365       1.01684       1.7595       0.978407       0.24995        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0194294     0.0266792     0.0968374     0.0219077     0.0169962       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0193281     0.0196549     0.0203579     0.0196438    0.00022295       19     
produce:beamevent:BeamEvent                      0.000118661   0.000202621   0.000989662   0.000155416   0.000187454      19     
produce:pdvddaphne:DAPHNEReaderPDVD                12.0486       15.8188       23.5252       15.6239       3.25967        19     
produce:ophit:OpHitFinder                         0.0297237     0.0352733     0.044828      0.0360422     0.0038532       19     
produce:wclsdatavd:WireCellToolkit                 82.3939       93.9561       102.424       94.6727       5.11135        19     
produce:gaushit:GausHitFinder                     0.779977       1.34647       1.81387       1.37293      0.259247        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00022456    0.000339987   0.000563004   0.000317001   8.01175e-05      19     
produce:reco3d:SpacePointSolver                    13.5178       20.8043       27.8473       20.2136       3.84273        19     
produce:hitpdune:DisambigFromSpacePoints          0.106257      0.196868      0.280827      0.204527      0.0547889       19     
produce:cluster3d:Cluster3D                        1.22532       2.66118       3.73236       2.77884      0.689702        19     
[art]:TriggerResults:TriggerResultInserter       2.9685e-05    4.40001e-05   0.000112291   4.0482e-05    1.74027e-05      19     
end_path:out1:RootOutput                          5.068e-06    9.92611e-06   2.9525e-05     9.074e-06    5.16245e-06      19     
end_path:out1:RootOutput(write)                    4.29457       4.91605       5.60941       4.87082      0.359697        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5385.52 MB
  Peak resident set size usage (VmHWM): 3279 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0056_df-s04-d0_dw_0_20260203T190325_reco_stage0_20260204T013023_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0056_df-s04-d0_dw_0_20260203T190325_reco_stage0_20260204T013023_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0056_df-s04-d0_dw_0_20260203T190325_reco_stage0_20260204T013023_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162316.0,
    "core.end_time": 1770168625.0,
    "core.events": [
      6394,
      6400,
      6406,
      6412,
      6418,
      6424,
      6430,
      6436,
      6442,
      6448,
      6454,
      6460,
      6466,
      6472,
      6478,
      6484,
      6490,
      6496,
      6502
    ],
    "core.event_count": 19,
    "core.first_event_number": 6394,
    "core.last_event_number": 6502,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0056_df-s04-d0_dw_0_20260203T190325.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:47 UTC       justIN version: 01.06.00