justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.18@dunegpschedd02.fnal.gov

Jobsub ID297923.18@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 02:57:59
From worker nodeHostnamewn1208091.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:57:45
Input filesvd-protodune:np02vd_raw_run042433_0097_df-s02-d1_dw_0_20260203T191859.hdf5
JobscriptExit code0
Real time2h (10583s)
CPU time1h (4771s = 45%)
Max RSS bytes3382800384 (3226 MiB)
Outputting started2026-02-04 02:54:09
Output files
Finished2026-02-04 02:57:42
Saved logsjustin-logs:297923.18-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ooking up space point ptr... 0x6231ad70, 0x6231aad0
Caught exception in looking up space point ptr... 0x6231aad0, 0x6231ad70
Caught exception in looking up space point ptr... 0x6231ad70, 0x6231ac20
Caught exception in looking up space point ptr... 0x6231ac20, 0x6231ad70
Caught exception in looking up space point ptr... 0x6231ad70, 0x14fb9830
Caught exception in looking up space point ptr... 0x14fb9830, 0x6231ad70
Caught exception in looking up space point ptr... 0x43ce2de0, 0x43ce2fe0
Caught exception in looking up space point ptr... 0x43ce2fe0, 0x43ce2de0
Caught exception in looking up space point ptr... 0x62439ba0, 0x62439cf0
Caught exception in looking up space point ptr... 0x1681cf30, 0x62439ba0
Caught exception in looking up space point ptr... 0x5b9693c0, 0x5b968f70
Caught exception in looking up space point ptr... 0x5b968f70, 0x5b9693c0
Caught exception in looking up space point ptr... 0x5b9693c0, 0x5b967f10
Caught exception in looking up space point ptr... 0x5b967f10, 0x5b9693c0
Caught exception in looking up space point ptr... 0x59bc7410, 0x59bc7f90
Caught exception in looking up space point ptr... 0x59bc2700, 0x59bc7410
Caught exception in looking up space point ptr... 0x443e8890, 0x3509cc20
Caught exception in looking up space point ptr... 0x3509cc20, 0x3509c1e0
Caught exception in looking up space point ptr... 0xf8b7580, 0xf8b7810
Caught exception in looking up space point ptr... 0xf8b72f0, 0xf8b7580
Caught exception in looking up space point ptr... 0x5c146a60, 0x5c136150
Caught exception in looking up space point ptr... 0x5c136150, 0x286e4690
Caught exception in looking up space point ptr... 0x4d88e4c0, 0x4d887040
Caught exception in looking up space point ptr... 0x4d887040, 0x4d88e4c0
Caught exception in looking up space point ptr... 0x7446660, 0xc231b40
Caught exception in looking up space point ptr... 0xc231b40, 0x7446660
Caught exception in looking up space point ptr... 0x567f8ca0, 0x18af1150
Caught exception in looking up space point ptr... 0x18af1150, 0x567f8ca0
Caught exception in looking up space point ptr... 0x65c842d0, 0x17175f90
Caught exception in looking up space point ptr... 0x17175f90, 0x65c842d0
Caught exception in looking up space point ptr... 0xf3c3700, 0xf3c38e0
Caught exception in looking up space point ptr... 0xf3c38e0, 0xf3c3700
Caught exception in looking up space point ptr... 0x18ce55c0, 0x18ce54c0
Caught exception in looking up space point ptr... 0x18ce54c0, 0x18ce56c0
Caught exception in looking up space point ptr... 0x596fb210, 0x287b5580
Caught exception in looking up space point ptr... 0x287b5580, 0x287b5c60
Caught exception in looking up space point ptr... 0x4d88bc20, 0x4d88b900
Caught exception in looking up space point ptr... 0x4d88b900, 0x4d88b590
Caught exception in looking up space point ptr... 0x32795f80, 0x34af7cc0
Caught exception in looking up space point ptr... 0x34af7cc0, 0x32795f80
Caught exception in looking up space point ptr... 0x18f1eda0, 0x5660b110
Caught exception in looking up space point ptr... 0x5660b110, 0x18f1eda0
Caught exception in looking up space point ptr... 0x18cf0090, 0x18ceff60
Caught exception in looking up space point ptr... 0x18ceff60, 0x18cf0090
Caught exception in looking up space point ptr... 0xf8b9ad0, 0xf8b97b0
Caught exception in looking up space point ptr... 0xf8b97b0, 0xf8b9ad0
Caught exception in looking up space point ptr... 0xf72d4f0, 0xf730bf0
Caught exception in looking up space point ptr... 0xf730bf0, 0x2e51af30
Caught exception in looking up space point ptr... 0x47368e20, 0x47368280
Caught exception in looking up space point ptr... 0x47368280, 0x47368e20
Caught exception in looking up space point ptr... 0x56a183f0, 0x56a17f50
Caught exception in looking up space point ptr... 0x56a17f50, 0x56a183f0
Caught exception in looking up space point ptr... 0x56a183f0, 0x56a18ae0
Caught exception in looking up space point ptr... 0x56a18ae0, 0x56a183f0
Caught exception in looking up space point ptr... 0xfb50ae0, 0xfb504f0
Caught exception in looking up space point ptr... 0xfb504f0, 0xfb50ae0
Caught exception in looking up space point ptr... 0x65d9bec0, 0x3033f580
Caught exception in looking up space point ptr... 0x3033f580, 0x65dc2b60
Caught exception in looking up space point ptr... 0x5970b410, 0x5970c8f0
Caught exception in looking up space point ptr... 0x5970c8f0, 0x5970b410
Caught exception in looking up space point ptr... 0x807c5310, 0x7411830
Caught exception in looking up space point ptr... 0x7411830, 0x741a600
Caught exception in looking up space point ptr... 0x698ef2b0, 0x569d51c0
Caught exception in looking up space point ptr... 0x569d51c0, 0x698ef2b0
Caught exception in looking up space point ptr... 0x14189b20, 0xc22b9e0
Caught exception in looking up space point ptr... 0xc22b9e0, 0x5b9c2130
Caught exception in looking up space point ptr... 0x5b9c2130, 0x60a1be00
++++>>>> total num hits: 230804, num free: 222857
04-Feb-2026 02:53:56 GMT  Closed output file "np02vd_raw_run042433_0097_df-s02-d1_dw_0_20260203T191859_reco_stage0_20260204T025356_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         447.984       549.646       679.26        541.543       52.1756        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.6531e-05    0.000124218   0.000369106   0.000105037   5.9972e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                206.864       284.485       400.978       280.991       46.6282        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.900401       1.4013        2.19369       1.29666      0.359358        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0221135     0.0325659     0.066121      0.0242939     0.0135582       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0219566     0.0376819     0.281046      0.0229148     0.0575674       19     
produce:beamevent:BeamEvent                      8.2966e-05    0.000199503   0.00122151    0.000138782   0.000244319      19     
produce:pdvddaphne:DAPHNEReaderPDVD                16.8339       23.1194       35.2963       23.4669       4.28888        19     
produce:ophit:OpHitFinder                         0.0413145      0.05412      0.0620095     0.0545555    0.00520866       19     
produce:wclsdatavd:WireCellToolkit                 181.226       191.237       205.733        190.9        6.95514        19     
produce:gaushit:GausHitFinder                      1.50737       2.57588       3.92967       2.42434      0.665736        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000405484   0.000609447   0.000941465   0.000580624   0.000134053      19     
produce:reco3d:SpacePointSolver                    18.1964       30.8387       63.7865       30.6874       9.88531        19     
produce:hitpdune:DisambigFromSpacePoints          0.213503      0.378328      0.641348      0.361695      0.0965856       19     
produce:cluster3d:Cluster3D                        3.27051       6.16442       15.0399       5.22004       2.93404        19     
[art]:TriggerResults:TriggerResultInserter       2.8043e-05    3.93727e-05   0.000131398   3.5427e-05    2.20348e-05      19     
end_path:out1:RootOutput                          4.147e-06    7.05316e-06   3.2892e-05     5.23e-06     6.5219e-06       19     
end_path:out1:RootOutput(write)                    8.48027       9.25824       10.7661       9.24619      0.538131        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5355.86 MB
  Peak resident set size usage (VmHWM): 3382.8 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0097_df-s02-d1_dw_0_20260203T191859_reco_stage0_20260204T025356_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0097_df-s02-d1_dw_0_20260203T191859_reco_stage0_20260204T025356_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0097_df-s02-d1_dw_0_20260203T191859_reco_stage0_20260204T025356_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163081.0,
    "core.end_time": 1770173639.0,
    "core.events": [
      11065,
      11071,
      11077,
      11083,
      11089,
      11095,
      11101,
      11107,
      11113,
      11119,
      11125,
      11131,
      11137,
      11143,
      11149,
      11155,
      11161,
      11167,
      11173
    ],
    "core.event_count": 19,
    "core.first_event_number": 11065,
    "core.last_event_number": 11173,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0097_df-s02-d1_dw_0_20260203T191859.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:20:18 UTC       justIN version: 01.06.00