justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.47@dunegpschedd01.fnal.gov

Jobsub ID303654.47@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 03:03:18
From worker nodeHostnamewn1208250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:50:47
Input filesvd-protodune:np02vd_raw_run042433_0328_df-s04-d0_dw_0_20260203T204647.hdf5
JobscriptExit code0
Real time3h (11349s)
CPU time1h (6023s = 53%)
Max RSS bytes3542745088 (3378 MiB)
Outputting started2026-02-04 02:59:58
Output files
Finished2026-02-04 03:03:18
Saved logsjustin-logs:303654.47-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x6d805000, 0x3ddca740
Caught exception in looking up space point ptr... 0x45ec3db0, 0x5318b900
Caught exception in looking up space point ptr... 0x39efc360, 0x45ec3db0
Caught exception in looking up space point ptr... 0x8e8bf90, 0x144e0600
Caught exception in looking up space point ptr... 0x144f2760, 0x8e8bf90
Caught exception in looking up space point ptr... 0x471fa2b0, 0x16c2cb50
Caught exception in looking up space point ptr... 0x16c2cb50, 0x47207330
Caught exception in looking up space point ptr... 0x46925bf0, 0x38f311c0
Caught exception in looking up space point ptr... 0x38f311c0, 0x46925bf0
Caught exception in looking up space point ptr... 0x463a1ff0, 0x4638f850
Caught exception in looking up space point ptr... 0x4638f6c0, 0x463a1ff0
Caught exception in looking up space point ptr... 0x4ec1f2f0, 0x3ac1c2d0
Caught exception in looking up space point ptr... 0x3ac1c2d0, 0x3ac1c140
Caught exception in looking up space point ptr... 0x8016f880, 0x1729ce80
Caught exception in looking up space point ptr... 0x1729ce80, 0x1729ccf0
Caught exception in looking up space point ptr... 0x542918b0, 0x15ce5d30
Caught exception in looking up space point ptr... 0x15ce5d30, 0x15ce5ba0
Caught exception in looking up space point ptr... 0x17260260, 0x172600d0
Caught exception in looking up space point ptr... 0x1725fdb0, 0x17260260
Caught exception in looking up space point ptr... 0x1ae7cb30, 0x4167f1f0
Caught exception in looking up space point ptr... 0x4167f1f0, 0xb03f930
Caught exception in looking up space point ptr... 0x12574b00, 0x39b91c40
Caught exception in looking up space point ptr... 0x39b91c40, 0x12574b00
Caught exception in looking up space point ptr... 0x7c6a3750, 0x40628380
Caught exception in looking up space point ptr... 0x437ac4e0, 0x7c6a3750
Caught exception in looking up space point ptr... 0x3fbce640, 0x3861d960
Caught exception in looking up space point ptr... 0x3fbcde90, 0x3fbce640
Caught exception in looking up space point ptr... 0x650888e0, 0x304be7e0
Caught exception in looking up space point ptr... 0x65077a90, 0x650888e0
Caught exception in looking up space point ptr... 0xfcdb990, 0x14276680
Caught exception in looking up space point ptr... 0xfcdbcb0, 0xfcdb990
Caught exception in looking up space point ptr... 0x45fc3a60, 0x1728be20
Caught exception in looking up space point ptr... 0x647616e0, 0x45fc3a60
Caught exception in looking up space point ptr... 0x646bc250, 0x646c8be0
Caught exception in looking up space point ptr... 0x646aa5f0, 0x646bc250
Caught exception in looking up space point ptr... 0x1002de70, 0x1001bea0
Caught exception in looking up space point ptr... 0x1001bea0, 0x1000a0b0
Caught exception in looking up space point ptr... 0x6b24de60, 0x6b23c520
Caught exception in looking up space point ptr... 0x6b260420, 0x6b24de60
Caught exception in looking up space point ptr... 0x644934b0, 0x16e518a0
Caught exception in looking up space point ptr... 0x16e518a0, 0x45da9f20
Caught exception in looking up space point ptr... 0x2c765d10, 0x2c753f20
Caught exception in looking up space point ptr... 0x2c753f20, 0x2c741fa0
Caught exception in looking up space point ptr... 0x55a7f680, 0x30439470
Caught exception in looking up space point ptr... 0x30439470, 0x55a7f680
Caught exception in looking up space point ptr... 0x3ae558d0, 0x10839520
Caught exception in looking up space point ptr... 0x649cb580, 0x3ae558d0
Caught exception in looking up space point ptr... 0x4729aee0, 0x4729ad50
Caught exception in looking up space point ptr... 0x4729ad50, 0xec952a0
Caught exception in looking up space point ptr... 0x6f97e190, 0x64f03420
Caught exception in looking up space point ptr... 0x64f03420, 0x6f97e190
Caught exception in looking up space point ptr... 0x3a92b9f0, 0x3a92bc90
Caught exception in looking up space point ptr... 0x3a92bc90, 0x3a92b9f0
Caught exception in looking up space point ptr... 0x73a0d0f0, 0x6b8c2b70
Caught exception in looking up space point ptr... 0x6b8c2b70, 0x73a0d0f0
Caught exception in looking up space point ptr... 0x55ca6ac0, 0x55ca6ca0
Caught exception in looking up space point ptr... 0x55ca6ca0, 0x55ca6ac0
Caught exception in looking up space point ptr... 0x5b870020, 0x41deed00
Caught exception in looking up space point ptr... 0x41deed00, 0x5b870020
Caught exception in looking up space point ptr... 0x17a89a80, 0x17a898a0
Caught exception in looking up space point ptr... 0x17a898a0, 0x17a89a80
Caught exception in looking up space point ptr... 0x3b1a4780, 0x3c00a390
Caught exception in looking up space point ptr... 0x3c00a390, 0x3b1a4780
Caught exception in looking up space point ptr... 0x388806e0, 0x2e775230
Caught exception in looking up space point ptr... 0x2e775230, 0x2e7750e0
Caught exception in looking up space point ptr... 0x2e7750e0, 0x38880550
Caught exception in looking up space point ptr... 0x38880550, 0x388806e0
++++>>>> total num hits: 540801, num free: 531361
04-Feb-2026 02:59:45 GMT  Closed output file "np02vd_raw_run042433_0328_df-s04-d0_dw_0_20260203T204647_reco_stage0_20260204T025945_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         481.662       590.957       676.424       612.676       61.5925        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        9.606e-05    0.000131003   0.000320523   0.000104717   6.59899e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                186.334       259.549       378.073       265.862       39.8484        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.787384       1.28097       1.89268       1.17262      0.322684        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0219635     0.0285725     0.0601545     0.0228907    0.00976138       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.021841      0.0228124     0.0240416     0.022713     0.000504702      19     
produce:beamevent:BeamEvent                       8.005e-05    0.000200472   0.00137609    0.000133822   0.000278385      19     
produce:pdvddaphne:DAPHNEReaderPDVD                17.7264       22.7826       33.5785       22.7003       3.66793        19     
produce:ophit:OpHitFinder                         0.0342239     0.0541224     0.0686952     0.0553671    0.00689203       19     
produce:wclsdatavd:WireCellToolkit                 173.078       194.586       216.414       195.623       10.338         19     
produce:gaushit:GausHitFinder                      1.53467       3.61358       5.33097       3.68956       1.01359        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00043573    0.000631051   0.000987988   0.000604387   0.000122651      19     
produce:reco3d:SpacePointSolver                    15.5221       27.0259       42.6566       26.3214       7.81461        19     
produce:hitpdune:DisambigFromSpacePoints          0.248493      0.477993      0.767153      0.448159      0.147642        19     
produce:cluster3d:Cluster3D                        2.92632       70.5586       116.555       94.5964       45.1953        19     
[art]:TriggerResults:TriggerResultInserter       2.3153e-05    4.37137e-05   0.000146456    3.186e-05    2.83693e-05      19     
end_path:out1:RootOutput                          3.697e-06    7.06484e-06   2.9625e-05     5.11e-06     5.89011e-06      19     
end_path:out1:RootOutput(write)                    8.75603       10.9247       13.0951       11.5267       1.41004        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5641.05 MB
  Peak resident set size usage (VmHWM): 3542.75 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0328_df-s04-d0_dw_0_20260203T204647_reco_stage0_20260204T025945_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0328_df-s04-d0_dw_0_20260203T204647_reco_stage0_20260204T025945_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0328_df-s04-d0_dw_0_20260203T204647_reco_stage0_20260204T025945_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162658.0,
    "core.end_time": 1770173988.0,
    "core.events": [
      37402,
      37408,
      37414,
      37420,
      37426,
      37432,
      37438,
      37444,
      37450,
      37456,
      37462,
      37468,
      37474,
      37480,
      37486,
      37492,
      37498,
      37504,
      37510
    ],
    "core.event_count": 19,
    "core.first_event_number": 37402,
    "core.last_event_number": 37510,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0328_df-s04-d0_dw_0_20260203T204647.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:47 UTC       justIN version: 01.06.00