justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.23@dunegpschedd02.fnal.gov

Jobsub ID297921.23@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:51:42
From worker nodeHostnamet2wn175.physics.ox.ac.uk
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:45:01
Input filesvd-protodune:np02vd_raw_run042433_0315_df-s04-d0_dw_0_20260203T204151.hdf5
JobscriptExit code0
Real time2h (7407s)
CPU time53m (3211s = 43%)
Max RSS bytes3509678080 (3347 MiB)
Outputting started2026-02-04 01:48:30
Output files
Finished2026-02-04 01:51:42
Saved logsjustin-logs:297921.23-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x4a6c76b0, 0x4a6b5280
Caught exception in looking up space point ptr... 0x74f405b0, 0x157a62f0
Caught exception in looking up space point ptr... 0x157a62f0, 0x74f405b0
Caught exception in looking up space point ptr... 0x4712ed10, 0x6c695210
Caught exception in looking up space point ptr... 0x5e86b3a0, 0x4712ed10
Caught exception in looking up space point ptr... 0x550c4ec0, 0x2eb0e8b0
Caught exception in looking up space point ptr... 0x2eb0e8b0, 0x550b1fd0
Caught exception in looking up space point ptr... 0x9c56170, 0x54e841f0
Caught exception in looking up space point ptr... 0x9c56300, 0x9c56170
Caught exception in looking up space point ptr... 0x82e30650, 0x82e30960
Caught exception in looking up space point ptr... 0x82e39d70, 0x82e30650
Caught exception in looking up space point ptr... 0x9c7bb00, 0x9c8dda0
Caught exception in looking up space point ptr... 0x9c8dda0, 0x9c8df30
Caught exception in looking up space point ptr... 0x131ad570, 0x6c66a0f0
Caught exception in looking up space point ptr... 0x19a3d700, 0x131ad570
Caught exception in looking up space point ptr... 0x4af4f630, 0x8341af00
Caught exception in looking up space point ptr... 0x4af4f4a0, 0x4af4f630
Caught exception in looking up space point ptr... 0x1a87f460, 0x1a84c750
Caught exception in looking up space point ptr... 0x1a84c750, 0x1a84c670
Caught exception in looking up space point ptr... 0x1aada6b0, 0x1aada460
Caught exception in looking up space point ptr... 0x1aada460, 0x1aae3610
Caught exception in looking up space point ptr... 0x33f19ed0, 0x33f1a370
Caught exception in looking up space point ptr... 0x39792550, 0x33f19ed0
Caught exception in looking up space point ptr... 0x33b7e660, 0x33b7dfe0
Caught exception in looking up space point ptr... 0x33b7dfe0, 0x7486f530
Caught exception in looking up space point ptr... 0x4f3d0fd0, 0x4f3e30e0
Caught exception in looking up space point ptr... 0x4f3bf050, 0x4f3d0fd0
Caught exception in looking up space point ptr... 0x1b0f3f90, 0x1b0ee100
Caught exception in looking up space point ptr... 0x1b0ee100, 0x1b0f3d00
Caught exception in looking up space point ptr... 0x2b96cc80, 0x390ccf80
Caught exception in looking up space point ptr... 0x390ccf80, 0x72cd1670
Caught exception in looking up space point ptr... 0x4d7fcb80, 0x3da72000
Caught exception in looking up space point ptr... 0x3da66ba0, 0x4d7fcb80
Caught exception in looking up space point ptr... 0x3871eb80, 0x6913cac0
Caught exception in looking up space point ptr... 0x3871ef70, 0x3871eb80
Caught exception in looking up space point ptr... 0xb9ac270, 0x45f46200
Caught exception in looking up space point ptr... 0x45f46200, 0x4a617c60
Caught exception in looking up space point ptr... 0x16a0dd00, 0x398df3f0
Caught exception in looking up space point ptr... 0x398df3f0, 0x398df580
Caught exception in looking up space point ptr... 0x9cf7c50, 0x6c6a7ff0
Caught exception in looking up space point ptr... 0x9cf7de0, 0x9cf7c50
Caught exception in looking up space point ptr... 0x17648d80, 0x17648bf0
Caught exception in looking up space point ptr... 0x17648bf0, 0x1763a4b0
Caught exception in looking up space point ptr... 0x9cfd690, 0x9cfd500
Caught exception in looking up space point ptr... 0x9cfd500, 0x14ba9510
Caught exception in looking up space point ptr... 0x3ec9ef70, 0x3ec8d180
Caught exception in looking up space point ptr... 0x3ec8d180, 0x3ec7aee0
Caught exception in looking up space point ptr... 0x6fa53610, 0x14ba4310
Caught exception in looking up space point ptr... 0x6fa537a0, 0x6fa53610
Caught exception in looking up space point ptr... 0x14df4570, 0x14e06810
Caught exception in looking up space point ptr... 0x14de1fb0, 0x14df4570
Caught exception in looking up space point ptr... 0x3e735b20, 0x2bc1f740
Caught exception in looking up space point ptr... 0x2bc1f740, 0x410e7da0
Caught exception in looking up space point ptr... 0x3a28f760, 0x8b2ef90
Caught exception in looking up space point ptr... 0x8b2ef90, 0x3a4959f0
Caught exception in looking up space point ptr... 0x157a6e20, 0x4feae1d0
Caught exception in looking up space point ptr... 0x4feae1d0, 0x157a6e20
Caught exception in looking up space point ptr... 0x157a6e20, 0x4feade60
Caught exception in looking up space point ptr... 0x4feade60, 0x157a6e20
Caught exception in looking up space point ptr... 0x753aa440, 0x753aa730
Caught exception in looking up space point ptr... 0x753aa730, 0x694b4980
Caught exception in looking up space point ptr... 0x2af738c0, 0x12b782a0
Caught exception in looking up space point ptr... 0x12b782a0, 0x2af738c0
Caught exception in looking up space point ptr... 0x38bb2250, 0x69844bb0
Caught exception in looking up space point ptr... 0x69844bb0, 0x38bb2250
Caught exception in looking up space point ptr... 0x39f01ba0, 0x839438c0
Caught exception in looking up space point ptr... 0x839438c0, 0x39f01ba0
++++>>>> total num hits: 509251, num free: 501589
04-Feb-2026 01:48:20 GMT  Closed output file "np02vd_raw_run042433_0315_df-s04-d0_dw_0_20260203T204151_reco_stage0_20260204T014820_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         292.383       385.732       457.242       382.623       46.9412        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.5495e-05    0.000112005   0.00016465    9.9842e-05    2.41121e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                135.479       207.438       291.991       191.415       43.8438        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.71979       1.0922        3.16198      0.990802      0.506188        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0195049     0.0264849     0.0648212     0.0231301     0.0104083       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0194827     0.0197927     0.0212368     0.0197062    0.00036001       19     
produce:beamevent:BeamEvent                      0.000126191   0.000209427   0.00114162    0.000160043   0.000220219      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.2038       16.9606       23.1192       16.8976       3.37906        19     
produce:ophit:OpHitFinder                         0.028885      0.0369745     0.0444816     0.0362976    0.00387829       19     
produce:wclsdatavd:WireCellToolkit                 87.2172       95.3781       102.833        95.56        3.8349         19     
produce:gaushit:GausHitFinder                     0.829899       1.94681       2.83883       1.94498      0.559649        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000208095   0.000350195   0.000572699   0.000353817   8.68791e-05      19     
produce:reco3d:SpacePointSolver                    13.2481       18.7626       24.2056       19.0657       3.35101        19     
produce:hitpdune:DisambigFromSpacePoints          0.113662      0.281959      0.390516      0.297994      0.0789406       19     
produce:cluster3d:Cluster3D                        1.06001       37.7856       63.6489       56.7593       27.1349        19     
[art]:TriggerResults:TriggerResultInserter       3.4862e-05    4.79348e-05   0.000134895   4.0431e-05     2.204e-05       19     
end_path:out1:RootOutput                          7.382e-06    1.6553e-05    0.000121414    9.735e-06    2.50848e-05      19     
end_path:out1:RootOutput(write)                    3.72771       5.94666       6.89304       6.2414       0.849744        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5605.38 MB
  Peak resident set size usage (VmHWM): 3509.68 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0315_df-s04-d0_dw_0_20260203T204151_reco_stage0_20260204T014820_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0315_df-s04-d0_dw_0_20260203T204151_reco_stage0_20260204T014820_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0315_df-s04-d0_dw_0_20260203T204151_reco_stage0_20260204T014820_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162310.0,
    "core.end_time": 1770169701.0,
    "core.events": [
      35920,
      35926,
      35932,
      35938,
      35944,
      35950,
      35956,
      35962,
      35968,
      35974,
      35980,
      35986,
      35992,
      35998,
      36004,
      36010,
      36016,
      36022,
      36028
    ],
    "core.event_count": 19,
    "core.first_event_number": 35920,
    "core.last_event_number": 36028,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0315_df-s04-d0_dw_0_20260203T204151.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:13 UTC       justIN version: 01.06.00