justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.4@dunegpschedd01.fnal.gov

Jobsub ID303654.4@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 02:16:00
From worker nodeHostnamet2wn068.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:49:04
Input filesvd-protodune:np02vd_raw_run042433_0472_df-s04-d0_dw_0_20260203T214132.hdf5
JobscriptExit code0
Real time2h (8591s)
CPU time1h (4199s = 48%)
Max RSS bytes3447930880 (3288 MiB)
Outputting started2026-02-04 02:12:18
Output files
Finished2026-02-04 02:16:00
Saved logsjustin-logs:303654.4-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tr... 0x3b2a8b00, 0x3b2a8c90
Caught exception in looking up space point ptr... 0xff1b850, 0x78f71680
Caught exception in looking up space point ptr... 0x78f74a00, 0xff1b850
Caught exception in looking up space point ptr... 0x448d0690, 0x3a39bb30
Caught exception in looking up space point ptr... 0x7a1e1980, 0x448d0690
Caught exception in looking up space point ptr... 0x3a45bcf0, 0x5416c6e0
Caught exception in looking up space point ptr... 0x3a437490, 0x3a45bcf0
Caught exception in looking up space point ptr... 0x5a608a10, 0x5a608780
Caught exception in looking up space point ptr... 0x5a605920, 0x5a608a10
Caught exception in looking up space point ptr... 0x3bb56680, 0xd892f70
Caught exception in looking up space point ptr... 0xd892f70, 0x52930af0
Caught exception in looking up space point ptr... 0x5b6a0bc0, 0x5b6a0970
Caught exception in looking up space point ptr... 0x5b6a0970, 0x5b6ad990
Caught exception in looking up space point ptr... 0x5a7b19a0, 0x5a7b1710
Caught exception in looking up space point ptr... 0x5a7b1710, 0x5a7b4de0
Caught exception in looking up space point ptr... 0x30a452a0, 0x30a45050
Caught exception in looking up space point ptr... 0x30a45050, 0x30a48680
Caught exception in looking up space point ptr... 0x522136e0, 0xfe02520
Caught exception in looking up space point ptr... 0xfe02520, 0x52215260
Caught exception in looking up space point ptr... 0x511a1b20, 0x30a475d0
Caught exception in looking up space point ptr... 0x30a475d0, 0x511a46e0
Caught exception in looking up space point ptr... 0x6d51c3f0, 0x6d2ae830
Caught exception in looking up space point ptr... 0x6d2ae830, 0x4481b920
Caught exception in looking up space point ptr... 0x31774960, 0x31774af0
Caught exception in looking up space point ptr... 0x31588d00, 0x31774960
Caught exception in looking up space point ptr... 0x1a96f910, 0x3b85cc70
Caught exception in looking up space point ptr... 0x3b85cc70, 0x3b85c7c0
Caught exception in looking up space point ptr... 0x3b58fd10, 0x3b58b310
Caught exception in looking up space point ptr... 0x3b58b310, 0x3b59f8b0
Caught exception in looking up space point ptr... 0x1aaa35c0, 0x1182b6f0
Caught exception in looking up space point ptr... 0x1182b6f0, 0x1182b560
Caught exception in looking up space point ptr... 0x856da270, 0x856da730
Caught exception in looking up space point ptr... 0x856db7d0, 0x856da270
Caught exception in looking up space point ptr... 0x6fb96d50, 0x6fba8cd0
Caught exception in looking up space point ptr... 0x6fb887a0, 0x6fb96d50
Caught exception in looking up space point ptr... 0x37c40a40, 0x3fb0bfa0
Caught exception in looking up space point ptr... 0x37c40bd0, 0x37c40a40
Caught exception in looking up space point ptr... 0x165dbc20, 0x52f639d0
Caught exception in looking up space point ptr... 0x165dbdb0, 0x165dbc20
Caught exception in looking up space point ptr... 0x37c35eb0, 0x5a87f0a0
Caught exception in looking up space point ptr... 0x37c36040, 0x37c35eb0
Caught exception in looking up space point ptr... 0x81f9dfb0, 0x81f8f870
Caught exception in looking up space point ptr... 0x81f8f870, 0x81f7e0c0
Caught exception in looking up space point ptr... 0x12ce6150, 0x19bf7240
Caught exception in looking up space point ptr... 0x19bf7240, 0x69b4ed80
Caught exception in looking up space point ptr... 0x81f58a50, 0x81f674b0
Caught exception in looking up space point ptr... 0x81f674b0, 0x81f76870
Caught exception in looking up space point ptr... 0x82501be0, 0x3b2ce580
Caught exception in looking up space point ptr... 0x3b2ce580, 0x3b2ccfd0
Caught exception in looking up space point ptr... 0x688faec0, 0x6890d480
Caught exception in looking up space point ptr... 0x6890d480, 0x6891fa90
Caught exception in looking up space point ptr... 0x4df8a4f0, 0x1ad80d70
Caught exception in looking up space point ptr... 0x1ad80d70, 0x4df8a4f0
Caught exception in looking up space point ptr... 0x383375e0, 0x38337770
Caught exception in looking up space point ptr... 0x38337770, 0x38337400
Caught exception in looking up space point ptr... 0x2a053e10, 0x6370f740
Caught exception in looking up space point ptr... 0x6370f740, 0x2a053e10
Caught exception in looking up space point ptr... 0x820cbf60, 0x820ca1a0
Caught exception in looking up space point ptr... 0x820ca1a0, 0x636c61b0
Caught exception in looking up space point ptr... 0x3bdf5170, 0x53c43d90
Caught exception in looking up space point ptr... 0x53c43d90, 0x3bdf5170
Caught exception in looking up space point ptr... 0x4335bc50, 0x4460d290
Caught exception in looking up space point ptr... 0x4460d290, 0x4335bc50
Caught exception in looking up space point ptr... 0x51e6c720, 0x3c1382d0
Caught exception in looking up space point ptr... 0x3c1382d0, 0x51e6c850
Caught exception in looking up space point ptr... 0x5bbf8050, 0x60d9c820
Caught exception in looking up space point ptr... 0x60d9c820, 0x5bbf8050
++++>>>> total num hits: 542895, num free: 534237
04-Feb-2026 02:12:05 GMT  Closed output file "np02vd_raw_run042433_0472_df-s04-d0_dw_0_20260203T214132_reco_stage0_20260204T021205_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         319.87        445.095       540.212       453.509       56.0551        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.8113e-05    0.000153293   0.000252933   0.000140032   4.71774e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                142.518       213.747       274.596       212.441       38.0446        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.793838       1.05137       1.40975       1.03522      0.162607        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.019618      0.0252563     0.0470222     0.0232335    0.00707366       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0194743     0.0251315     0.0953323     0.019766      0.0175358       19     
produce:beamevent:BeamEvent                      0.00013904    0.000256903   0.00158875    0.000169096   0.000315784      19     
produce:pdvddaphne:DAPHNEReaderPDVD                12.944        18.7493       23.7813       19.1997       3.08977        19     
produce:ophit:OpHitFinder                         0.0349627     0.0465146     0.0612848     0.0466947    0.00618824       19     
produce:wclsdatavd:WireCellToolkit                 124.111       136.254       147.709       135.246       7.23321        19     
produce:gaushit:GausHitFinder                      1.01605       2.38604       3.70304       2.30874      0.771001        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000272339   0.000450568   0.00061571    0.000470068   7.88085e-05      19     
produce:reco3d:SpacePointSolver                    10.3739       17.1554       24.2801       17.3597       3.57296        19     
produce:hitpdune:DisambigFromSpacePoints          0.157196      0.358346       0.51432       0.37438      0.105097        19     
produce:cluster3d:Cluster3D                        2.24488       47.8048       81.6007       63.414        30.8017        19     
[art]:TriggerResults:TriggerResultInserter       3.1569e-05    5.26112e-05   0.000151352   4.7048e-05    2.50584e-05      19     
end_path:out1:RootOutput                          7.143e-06    1.56083e-05   6.9911e-05    1.2313e-05    1.3569e-05       19     
end_path:out1:RootOutput(write)                    5.82132       7.43694       9.6644        7.49199      0.828935        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5563.57 MB
  Peak resident set size usage (VmHWM): 3447.93 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0472_df-s04-d0_dw_0_20260203T214132_reco_stage0_20260204T021205_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0472_df-s04-d0_dw_0_20260203T214132_reco_stage0_20260204T021205_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0472_df-s04-d0_dw_0_20260203T214132_reco_stage0_20260204T021205_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162587.0,
    "core.end_time": 1770171127.0,
    "core.events": [
      53818,
      53824,
      53830,
      53836,
      53842,
      53848,
      53854,
      53860,
      53866,
      53872,
      53878,
      53884,
      53890,
      53896,
      53902,
      53908,
      53914,
      53920,
      53926
    ],
    "core.event_count": 19,
    "core.first_event_number": 53818,
    "core.last_event_number": 53926,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0472_df-s04-d0_dw_0_20260203T214132.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:27:08 UTC       justIN version: 01.06.00