justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.17@dunegpschedd02.fnal.gov

Jobsub ID297921.17@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 00:38:48
From worker nodeHostnamet2wn175.physics.ox.ac.uk
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:44:58
Input filesvd-protodune:np02vd_raw_run042433_0285_df-s04-d1_dw_0_20260203T203029.hdf5
JobscriptExit code0
Real time50m (3058s)
CPU time40m (2451s = 80%)
Max RSS bytes3383685120 (3226 MiB)
Outputting started2026-02-04 00:35:57
Output files
Finished2026-02-04 00:38:48
Saved logsjustin-logs:297921.17-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

pace point ptr... 0x5a6d1200, 0x5a6d1100
Caught exception in looking up space point ptr... 0x5a6d1100, 0x5a6d1200
Caught exception in looking up space point ptr... 0x5a6d1200, 0x5a6d1000
Caught exception in looking up space point ptr... 0x2a815b80, 0xf388ed0
Caught exception in looking up space point ptr... 0xf388ed0, 0x34238b90
Caught exception in looking up space point ptr... 0x3c18c480, 0x556ff7e0
Caught exception in looking up space point ptr... 0x556ff7e0, 0x3c18c480
Caught exception in looking up space point ptr... 0x2f325950, 0x2b5c38a0
Caught exception in looking up space point ptr... 0x2b5c38a0, 0x2b5c3a30
Caught exception in looking up space point ptr... 0xf30a4e0, 0xf309910
Caught exception in looking up space point ptr... 0xf309910, 0xf30a4e0
Caught exception in looking up space point ptr... 0xe2fd390, 0xe2f0730
Caught exception in looking up space point ptr... 0xe2f0730, 0xe2fd390
Caught exception in looking up space point ptr... 0x42861dc0, 0x4286b770
Caught exception in looking up space point ptr... 0x4286b770, 0x42861dc0
Caught exception in looking up space point ptr... 0x2af7abc0, 0x2af7aa30
Caught exception in looking up space point ptr... 0x2af7aa30, 0x2af7abc0
Caught exception in looking up space point ptr... 0x1aaf3280, 0x5cb86320
Caught exception in looking up space point ptr... 0x5cb86320, 0x1aaf3280
Caught exception in looking up space point ptr... 0x4283dc90, 0x4283e140
Caught exception in looking up space point ptr... 0x4283e140, 0x4283dc90
Caught exception in looking up space point ptr... 0x181ede60, 0x181edd60
Caught exception in looking up space point ptr... 0x181edd60, 0x181edc60
Caught exception in looking up space point ptr... 0x1aac6040, 0x1aac5e40
Caught exception in looking up space point ptr... 0x1aac5e40, 0x1aac5f40
Caught exception in looking up space point ptr... 0xe54b260, 0xe54b1c0
Caught exception in looking up space point ptr... 0xe54b1c0, 0xe57e570
Caught exception in looking up space point ptr... 0xe57e570, 0xe57e670
Caught exception in looking up space point ptr... 0xe57e770, 0xe54b260
Caught exception in looking up space point ptr... 0x61d5ad50, 0x61d5b070
Caught exception in looking up space point ptr... 0x61d5b070, 0x61d5b390
Caught exception in looking up space point ptr... 0x1647ae90, 0x44e8f580
Caught exception in looking up space point ptr... 0x44e90290, 0x1647ae90
Caught exception in looking up space point ptr... 0x13f19f80, 0x51286920
Caught exception in looking up space point ptr... 0x51286920, 0x5127b9e0
Caught exception in looking up space point ptr... 0x78b34fd0, 0x78b34e40
Caught exception in looking up space point ptr... 0x78b34e40, 0x78b34fd0
Caught exception in looking up space point ptr... 0x2eff1760, 0x3497d3b0
Caught exception in looking up space point ptr... 0x3497d3b0, 0x2ad527e0
Caught exception in looking up space point ptr... 0x13f18e30, 0x13f215d0
Caught exception in looking up space point ptr... 0x13f215d0, 0x13f21740
Caught exception in looking up space point ptr... 0x48a85d80, 0x48aad8e0
Caught exception in looking up space point ptr... 0x48aad8e0, 0x48aad9e0
Caught exception in looking up space point ptr... 0x401cff90, 0x401cfbd0
Caught exception in looking up space point ptr... 0x401cfbd0, 0x401cfdb0
Caught exception in looking up space point ptr... 0x40df5d60, 0x40df5a40
Caught exception in looking up space point ptr... 0x40df5a40, 0x40df5d60
Caught exception in looking up space point ptr... 0x1aaee430, 0x1aae6050
Caught exception in looking up space point ptr... 0x1aaee300, 0x1aaee430
Caught exception in looking up space point ptr... 0x2ef55650, 0x2ef55b20
Caught exception in looking up space point ptr... 0x2ef55b20, 0x2ef55650
Caught exception in looking up space point ptr... 0x4ef503b0, 0x4ef50040
Caught exception in looking up space point ptr... 0x4ef50040, 0x4ef503b0
Caught exception in looking up space point ptr... 0x5538f1a0, 0x553910f0
Caught exception in looking up space point ptr... 0x553910f0, 0x429548d0
Caught exception in looking up space point ptr... 0x7a920dd0, 0x7a920f60
Caught exception in looking up space point ptr... 0x7a920f60, 0x7a9210f0
Caught exception in looking up space point ptr... 0x7a880d30, 0x7a880950
Caught exception in looking up space point ptr... 0x7a880c00, 0x7a880d30
Caught exception in looking up space point ptr... 0x19fcee70, 0x19fce9f0
Caught exception in looking up space point ptr... 0x19fce9f0, 0x19fcee70
Caught exception in looking up space point ptr... 0x1625e8c0, 0x1625ebe0
Caught exception in looking up space point ptr... 0x1625ebe0, 0x1625a1f0
Caught exception in looking up space point ptr... 0x41381850, 0x41381530
Caught exception in looking up space point ptr... 0x41381530, 0x413813a0
Caught exception in looking up space point ptr... 0x413749d0, 0x41374b60
Caught exception in looking up space point ptr... 0x41374b60, 0x41374e80
++++>>>> total num hits: 116331, num free: 111278
04-Feb-2026 00:35:49 GMT  Closed output file "np02vd_raw_run042433_0285_df-s04-d1_dw_0_20260203T203029_reco_stage0_20260204T003549_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         126.769       157.232       458.344       139.212       71.3709        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.8278e-05    0.000118497   0.000271522   9.4143e-05    4.60843e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                14.7845       31.7559        312.8        15.6906       66.2737        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.59212       0.67041      0.961596      0.642568      0.0814424       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0195171     0.0212512     0.040123      0.019724     0.00461824       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0192871     0.0199207     0.0240142     0.0196857    0.000998259      19     
produce:beamevent:BeamEvent                      9.7458e-05    0.000207319   0.00123208    0.000153072   0.000242397      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.17553       3.74967       25.4036       2.59009       5.10719        19     
produce:ophit:OpHitFinder                         0.0223134     0.0337412     0.0432359     0.0336904    0.00620064       19     
produce:wclsdatavd:WireCellToolkit                 80.8569       91.1784       101.54        90.5997       5.24677        19     
produce:gaushit:GausHitFinder                     0.762681       1.29021       1.73123       1.19495      0.279563        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000196208   0.000306702   0.000543815   0.000297461   7.73621e-05      19     
produce:reco3d:SpacePointSolver                    13.6158       20.4349       27.4462       20.6162       3.35499        19     
produce:hitpdune:DisambigFromSpacePoints          0.118267      0.195591      0.352327      0.183741       0.05873        19     
produce:cluster3d:Cluster3D                        1.56705       2.93591       7.73285       2.63601       1.33514        19     
[art]:TriggerResults:TriggerResultInserter       3.0866e-05    4.15929e-05   9.6566e-05    3.8839e-05    1.3586e-05       19     
end_path:out1:RootOutput                          6.48e-06     9.67526e-06   2.5389e-05     8.853e-06    3.81895e-06      19     
end_path:out1:RootOutput(write)                    3.86604       4.88337       5.46915       4.88051       0.43106        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5410.04 MB
  Peak resident set size usage (VmHWM): 3383.69 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0285_df-s04-d1_dw_0_20260203T203029_reco_stage0_20260204T003549_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0285_df-s04-d1_dw_0_20260203T203029_reco_stage0_20260204T003549_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0285_df-s04-d1_dw_0_20260203T203029_reco_stage0_20260204T003549_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162308.0,
    "core.end_time": 1770165351.0,
    "core.events": [
      32501,
      32507,
      32513,
      32519,
      32525,
      32531,
      32537,
      32543,
      32549,
      32555,
      32561,
      32567,
      32573,
      32579,
      32585,
      32591,
      32597,
      32603,
      32609
    ],
    "core.event_count": 19,
    "core.first_event_number": 32501,
    "core.last_event_number": 32609,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0285_df-s04-d1_dw_0_20260203T203029.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:05 UTC       justIN version: 01.06.00