justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303730.2@dunegpschedd01.fnal.gov

Jobsub ID303730.2@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:20:32
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 03:55:11
From worker nodeHostnamewn1904250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:21:15
Input filesvd-protodune:np02vd_raw_run042433_0485_df-s04-d0_dw_0_20260203T214627.hdf5
JobscriptExit code0
Real time2h (9078s)
CPU time53m (3226s = 35%)
Max RSS bytes3385516032 (3228 MiB)
Outputting started2026-02-04 03:52:35
Output files
Finished2026-02-04 03:55:11
Saved logsjustin-logs:303730.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g up space point ptr... 0x3c4a2050, 0x3c4a0bf0
Caught exception in looking up space point ptr... 0x44f7cd70, 0x3c4a2050
Caught exception in looking up space point ptr... 0x59284270, 0x4b17fde0
Caught exception in looking up space point ptr... 0x4b17fde0, 0x59285f10
Caught exception in looking up space point ptr... 0xb0d29a0, 0x51985d20
Caught exception in looking up space point ptr... 0x51985d20, 0x3c47d2d0
Caught exception in looking up space point ptr... 0xe224620, 0x121eedb0
Caught exception in looking up space point ptr... 0x121eedb0, 0xe224620
Caught exception in looking up space point ptr... 0xe224620, 0xdafe6f0
Caught exception in looking up space point ptr... 0xdafe6f0, 0xe224620
Caught exception in looking up space point ptr... 0x29742860, 0x297425f0
Caught exception in looking up space point ptr... 0x297425f0, 0xd671820
Caught exception in looking up space point ptr... 0x327f4680, 0x3949ad00
Caught exception in looking up space point ptr... 0x3949ad00, 0x3945af30
Caught exception in looking up space point ptr... 0x150cd8e0, 0x632688d0
Caught exception in looking up space point ptr... 0x632688d0, 0x19a6a060
Caught exception in looking up space point ptr... 0x324d4800, 0x3906f050
Caught exception in looking up space point ptr... 0x3906f050, 0x324d4800
Caught exception in looking up space point ptr... 0x51979f50, 0x5197a0e0
Caught exception in looking up space point ptr... 0x5197a0e0, 0xdaf22d0
Caught exception in looking up space point ptr... 0x18a92bc0, 0x11bf60b0
Caught exception in looking up space point ptr... 0x11bf60b0, 0x18a928a0
Caught exception in looking up space point ptr... 0xe263530, 0x6363dcf0
Caught exception in looking up space point ptr... 0x6363dcf0, 0x15018c10
Caught exception in looking up space point ptr... 0x9182b20, 0x3e3445e0
Caught exception in looking up space point ptr... 0x3e3445e0, 0x803b5810
Caught exception in looking up space point ptr... 0x69e19290, 0x3ca41710
Caught exception in looking up space point ptr... 0x3ca41710, 0x69e19290
Caught exception in looking up space point ptr... 0x5f4a4560, 0x5f493260
Caught exception in looking up space point ptr... 0x5f493260, 0x5f4a4560
Caught exception in looking up space point ptr... 0x150cba30, 0x121b8630
Caught exception in looking up space point ptr... 0x121b8630, 0x50c610b0
Caught exception in looking up space point ptr... 0xd664d60, 0x477300d0
Caught exception in looking up space point ptr... 0x477300d0, 0x477326d0
Caught exception in looking up space point ptr... 0x4f963c50, 0x150216c0
Caught exception in looking up space point ptr... 0x150216c0, 0x3748e230
Caught exception in looking up space point ptr... 0x1501e050, 0x4f963c50
Caught exception in looking up space point ptr... 0x10da4e80, 0x10da5010
Caught exception in looking up space point ptr... 0x10fd13d0, 0x10da4e80
Caught exception in looking up space point ptr... 0x45059610, 0x31d69730
Caught exception in looking up space point ptr... 0x31d69730, 0x64ad77e0
Caught exception in looking up space point ptr... 0x37f4cc40, 0x11bf77f0
Caught exception in looking up space point ptr... 0x11bf77f0, 0x37f4cc40
Caught exception in looking up space point ptr... 0x3e35a640, 0x3e35a3a0
Caught exception in looking up space point ptr... 0x3e35a7e0, 0x3e35a640
Caught exception in looking up space point ptr... 0x3984c990, 0x519a4570
Caught exception in looking up space point ptr... 0x519a4570, 0x3984c620
Caught exception in looking up space point ptr... 0x13272900, 0x132743e0
Caught exception in looking up space point ptr... 0x132743e0, 0xfc9c830
Caught exception in looking up space point ptr... 0x4772cc20, 0x47731d80
Caught exception in looking up space point ptr... 0x47735550, 0x4772cc20
Caught exception in looking up space point ptr... 0xdb2d710, 0xdb28e10
Caught exception in looking up space point ptr... 0xdb28e10, 0xdb2a570
Caught exception in looking up space point ptr... 0xfc22730, 0x61f840d0
Caught exception in looking up space point ptr... 0x61f840d0, 0xfc20d00
Caught exception in looking up space point ptr... 0x15e79b70, 0x15e7a090
Caught exception in looking up space point ptr... 0x15e7a090, 0x15e79e00
Caught exception in looking up space point ptr... 0x42febf90, 0x43cc2d90
Caught exception in looking up space point ptr... 0x43cc2d90, 0x355f9e10
Caught exception in looking up space point ptr... 0x5e646620, 0x155e9bd0
Caught exception in looking up space point ptr... 0x155e9bd0, 0x5e646620
Caught exception in looking up space point ptr... 0x489d8d70, 0x3ac44120
Caught exception in looking up space point ptr... 0x3ac44120, 0x489d8d70
Caught exception in looking up space point ptr... 0xecfcc80, 0x1a3a5240
Caught exception in looking up space point ptr... 0x1a3a5240, 0xecfcc80
Caught exception in looking up space point ptr... 0x305e5f00, 0x5592bdb0
Caught exception in looking up space point ptr... 0x5592bdb0, 0x305e5e20
++++>>>> total num hits: 76424, num free: 73526
04-Feb-2026 03:52:26 GMT  Closed output file "np02vd_raw_run042433_0485_df-s04-d0_dw_0_20260203T214627_reco_stage0_20260204T035226_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         357.885       471.956       561.87        480.85        56.2725        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.9328e-05    0.000158772   0.000423156   0.000124034   8.70947e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                167.844       289.578       356.931       297.843       51.5657        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.808412       1.1881        2.00821       1.07518      0.307906        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220696     0.0308656     0.059306      0.0280035     0.0105943       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0220054     0.0250086     0.0736384      0.02226      0.0114639       19     
produce:beamevent:BeamEvent                      0.000114545   0.000275388   0.00123458    0.000211237   0.000241781      19     
produce:pdvddaphne:DAPHNEReaderPDVD                16.5666       23.6522       32.8026       24.3407       4.57625        19     
produce:ophit:OpHitFinder                         0.0329096     0.0451455     0.055367      0.0453552    0.00485708       19     
produce:wclsdatavd:WireCellToolkit                 100.405       126.779       146.73        135.109       16.0158        19     
produce:gaushit:GausHitFinder                     0.903815       1.4521        1.90802       1.59638      0.368103        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00028726    0.000425304   0.000779827   0.00042034    0.000107681      19     
produce:reco3d:SpacePointSolver                    13.2038       20.006        26.405        19.5588       3.6768         19     
produce:hitpdune:DisambigFromSpacePoints          0.104496      0.202864      0.303753      0.231975      0.0606866       19     
produce:cluster3d:Cluster3D                        1.58755       3.04974       4.41646       3.08066      0.789179        19     
[art]:TriggerResults:TriggerResultInserter       2.9645e-05    4.64968e-05   0.000144071   4.0096e-05    2.44416e-05      19     
end_path:out1:RootOutput                          4.278e-06    8.33937e-06   3.8863e-05     5.32e-06     7.84399e-06      19     
end_path:out1:RootOutput(write)                    4.39292       5.89323       7.73218       6.2114        1.18702        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5414.8 MB
  Peak resident set size usage (VmHWM): 3385.52 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0485_df-s04-d0_dw_0_20260203T214627_reco_stage0_20260204T035226_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0485_df-s04-d0_dw_0_20260203T214627_reco_stage0_20260204T035226_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0485_df-s04-d0_dw_0_20260203T214627_reco_stage0_20260204T035226_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168087.0,
    "core.end_time": 1770177148.0,
    "core.events": [
      55300,
      55306,
      55312,
      55318,
      55324,
      55330,
      55336,
      55342,
      55348,
      55354,
      55360,
      55366,
      55372,
      55378,
      55384,
      55390,
      55396,
      55402,
      55408
    ],
    "core.event_count": 19,
    "core.first_event_number": 55300,
    "core.last_event_number": 55408,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0485_df-s04-d0_dw_0_20260203T214627.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:29:15 UTC       justIN version: 01.06.00