justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297919.177@dunegpschedd02.fnal.gov

Jobsub ID297919.177@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:25
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod02
Last heartbeat2026-02-04 01:17:22
From worker nodeHostnamewl05.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:36:30
Input filesvd-protodune:np02vd_raw_run042421_0078_df-s03-d0_dw_0_20260203T141315.hdf5
JobscriptExit code0
Real time1h (5868s)
CPU time1h (5164s = 88%)
Max RSS bytes2712559616 (2586 MiB)
Outputting started2026-02-04 01:14:22
Output files
Finished2026-02-04 01:17:22
Saved logsjustin-logs:297919.177-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ht exception in looking up space point ptr... 0x15c70ae0, 0x4fd4ea40
Caught exception in looking up space point ptr... 0x2cabe270, 0x6819aeb0
Caught exception in looking up space point ptr... 0x6819aeb0, 0x612f9af0
Caught exception in looking up space point ptr... 0x13e570b0, 0x298c4970
Caught exception in looking up space point ptr... 0x298c4970, 0x3c34b250
Caught exception in looking up space point ptr... 0x42a66fb0, 0x3289f080
Caught exception in looking up space point ptr... 0x3289f080, 0x5105ec20
Caught exception in looking up space point ptr... 0x4fb55440, 0x4fb437e0
Caught exception in looking up space point ptr... 0x4fb437e0, 0x4f6810f0
Caught exception in looking up space point ptr... 0x4292d3e0, 0x4292d250
Caught exception in looking up space point ptr... 0x4291ae20, 0x4292d3e0
Caught exception in looking up space point ptr... 0x40b1bda0, 0x408a2570
Caught exception in looking up space point ptr... 0x408a2570, 0x476fe640
Caught exception in looking up space point ptr... 0x3ec83620, 0x214bd880
Caught exception in looking up space point ptr... 0x29283f10, 0x3ec83620
Caught exception in looking up space point ptr... 0x42671a40, 0x42671d10
Caught exception in looking up space point ptr... 0x42671d10, 0x2a9634e0
Caught exception in looking up space point ptr... 0x4eac4d60, 0x50fe5e30
Caught exception in looking up space point ptr... 0x50fe5e30, 0x40818040
Caught exception in looking up space point ptr... 0x2a1cf090, 0x2a1cf7d0
Caught exception in looking up space point ptr... 0x2a1cf7d0, 0x2a1cf090
Caught exception in looking up space point ptr... 0x125326a0, 0x12540f70
Caught exception in looking up space point ptr... 0x12540f70, 0x12541100
Caught exception in looking up space point ptr... 0x126638c0, 0x4cb451c0
Caught exception in looking up space point ptr... 0x4cb451c0, 0x4cb45350
Caught exception in looking up space point ptr... 0x5c5c4f10, 0x5c5d74d0
Caught exception in looking up space point ptr... 0x5c5d74d0, 0x5c5e9a90
Caught exception in looking up space point ptr... 0x383319a0, 0x3831f3e0
Caught exception in looking up space point ptr... 0x5110d470, 0x383319a0
Caught exception in looking up space point ptr... 0x32d9f980, 0x30d64d70
Caught exception in looking up space point ptr... 0x508e9910, 0x32d9f980
Caught exception in looking up space point ptr... 0x29767950, 0x4c9e58d0
Caught exception in looking up space point ptr... 0x5b18ce10, 0x29767950
Caught exception in looking up space point ptr... 0x50686720, 0x2e8eb630
Caught exception in looking up space point ptr... 0x2e8eb630, 0x50686590
Caught exception in looking up space point ptr... 0x388b8cb0, 0x388a84a0
Caught exception in looking up space point ptr... 0x532f5e70, 0x388b8cb0
Caught exception in looking up space point ptr... 0x23dbded0, 0x51b81760
Caught exception in looking up space point ptr... 0x51b81760, 0x51b818f0
Caught exception in looking up space point ptr... 0x12536e80, 0x5f8a0b80
Caught exception in looking up space point ptr... 0x125455c0, 0x12536e80
Caught exception in looking up space point ptr... 0x5c665fd0, 0x849e1e0
Caught exception in looking up space point ptr... 0x5c678590, 0x5c665fd0
Caught exception in looking up space point ptr... 0x288c82f0, 0x288c72c0
Caught exception in looking up space point ptr... 0x10d9ffc0, 0x288c82f0
Caught exception in looking up space point ptr... 0x377aa790, 0x5a7d1a00
Caught exception in looking up space point ptr... 0x850ad60, 0x377aa790
Caught exception in looking up space point ptr... 0x2e802170, 0x50de36a0
Caught exception in looking up space point ptr... 0x50de36a0, 0x2e802170
Caught exception in looking up space point ptr... 0x2bf88030, 0x2bf88320
Caught exception in looking up space point ptr... 0x2bf88320, 0x2bf88030
Caught exception in looking up space point ptr... 0x49c14e50, 0x3ffa8c50
Caught exception in looking up space point ptr... 0x3ffa8670, 0x49c14e50
Caught exception in looking up space point ptr... 0x10636ed0, 0x434f5290
Caught exception in looking up space point ptr... 0x434f5290, 0x10636ed0
Caught exception in looking up space point ptr... 0x60e34c70, 0x99d7360
Caught exception in looking up space point ptr... 0x99d7360, 0x60e34c70
Caught exception in looking up space point ptr... 0x25fd1de0, 0x40b2ee70
Caught exception in looking up space point ptr... 0x40b2ee70, 0x25fd1de0
Caught exception in looking up space point ptr... 0x2e884730, 0x2e885380
Caught exception in looking up space point ptr... 0x2e885380, 0x2e884730
Caught exception in looking up space point ptr... 0x37750fb0, 0x4ec84660
Caught exception in looking up space point ptr... 0x49e91ab0, 0x37750fb0
Caught exception in looking up space point ptr... 0x47005c50, 0x47005b20
Caught exception in looking up space point ptr... 0x47005b20, 0x3287a060
++++>>>> total num hits: 320336, num free: 314081
04-Feb-2026 01:14:09 GMT  Closed output file "np02vd_raw_run042421_0078_df-s03-d0_dw_0_20260203T141315_reco_stage0_20260204T011409_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         146.967       212.995       250.596       229.742       34.4504        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.7814e-05    0.000160116   0.00030533    0.000174296   4.98587e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                11.7678       13.179        15.3041       13.1229      0.670619        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.487323      0.512733      0.563493       0.51017      0.0175315       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0162665     0.0176109     0.0304665     0.0168002    0.00321028       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0803323     0.0831132     0.0989376     0.0820334    0.00434008       27     
produce:beamevent:BeamEvent                      0.000107303   0.000229032   0.00157296    0.000170355   0.00026756       27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.76296       4.89418       6.23467       4.89331      0.490715        27     
produce:ophit:OpHitFinder                         0.0389174     0.0512886     0.0606599     0.0515497    0.00496424       27     
produce:wclsdatavd:WireCellToolkit                 86.3083       115.64        125.652       117.48        8.46694        27     
produce:gaushit:GausHitFinder                      1.26875       2.11857       2.97428       2.13749      0.450811        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000387492   0.000568541    0.0015584    0.000512636   0.000226837      27     
produce:reco3d:SpacePointSolver                    6.37689       12.5422       19.7653       11.8771       3.60985        27     
produce:hitpdune:DisambigFromSpacePoints          0.112736      0.337949      0.510241      0.352321      0.123545        27     
produce:cluster3d:Cluster3D                        2.20664       54.2946       82.6702       69.524        30.7091        27     
[art]:TriggerResults:TriggerResultInserter       3.3321e-05    5.77491e-05   0.000135754   4.9852e-05    2.78946e-05      27     
end_path:out1:RootOutput                          5.149e-06    1.42504e-05   8.2393e-05     1.033e-05    1.4496e-05       27     
end_path:out1:RootOutput(write)                    6.7029        9.28857       10.2037       9.80308      0.970112        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4643.22 MB
  Peak resident set size usage (VmHWM): 2712.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0078_df-s03-d0_dw_0_20260203T141315_reco_stage0_20260204T011409_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0078_df-s03-d0_dw_0_20260203T141315_reco_stage0_20260204T011409_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0078_df-s03-d0_dw_0_20260203T141315_reco_stage0_20260204T011409_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161806.0,
    "core.end_time": 1770167651.0,
    "core.events": [
      12704,
      12710,
      12716,
      12722,
      12728,
      12734,
      12740,
      12746,
      12752,
      12758,
      12764,
      12770,
      12776,
      12782,
      12788,
      12794,
      12800,
      12806,
      12812,
      12818,
      12824,
      12830,
      12836,
      12842,
      12848,
      12854,
      12860
    ],
    "core.event_count": 27,
    "core.first_event_number": 12704,
    "core.last_event_number": 12860,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0078_df-s03-d0_dw_0_20260203T141315.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:51 UTC       justIN version: 01.06.00