justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303646.178@dunegpschedd01.fnal.gov

Jobsub ID303646.178@dunegpschedd01.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:44:24
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2026-02-04 01:22:28
From worker nodeHostnamecomp08-00
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:32
Input filesvd-protodune:np02vd_raw_run042422_0065_df-s03-d0_dw_0_20260203T154954.hdf5
JobscriptExit code0
Real time2h (8831s)
CPU time42m (2544s = 28%)
Max RSS bytes2591117312 (2471 MiB)
Outputting started2026-02-04 01:13:58
Output files
Finished2026-02-04 01:22:28
Saved logsjustin-logs:303646.178-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0, 0x8bad750
Caught exception in looking up space point ptr... 0x427d5380, 0x427d51f0
Caught exception in looking up space point ptr... 0x8fcd810, 0x427d5380
Caught exception in looking up space point ptr... 0x516d9380, 0x2f3d4510
Caught exception in looking up space point ptr... 0x2f3d4510, 0x2f3d4280
Caught exception in looking up space point ptr... 0x2b40f5e0, 0x2b40f900
Caught exception in looking up space point ptr... 0x2b40f900, 0x2b40fa90
Caught exception in looking up space point ptr... 0xe2c0ac0, 0x35a4a980
Caught exception in looking up space point ptr... 0xe2c0bc0, 0xe2c0ac0
Caught exception in looking up space point ptr... 0x39a5ad70, 0x39a5a8c0
Caught exception in looking up space point ptr... 0x39a5a8c0, 0x39a5aa50
Caught exception in looking up space point ptr... 0x4746dfe0, 0x461eed90
Caught exception in looking up space point ptr... 0x56421c90, 0x4746dfe0
Caught exception in looking up space point ptr... 0x4eb92b60, 0x4eb91c50
Caught exception in looking up space point ptr... 0x4eb91c50, 0x4eb92b60
Caught exception in looking up space point ptr... 0x846f2b0, 0x2f378c80
Caught exception in looking up space point ptr... 0x2f378c80, 0x846ee30
Caught exception in looking up space point ptr... 0x2268c620, 0x2aa87f30
Caught exception in looking up space point ptr... 0x2fb12430, 0x2268c620
Caught exception in looking up space point ptr... 0x2fb4d710, 0x32799bf0
Caught exception in looking up space point ptr... 0x32799bf0, 0x2fb4d710
Caught exception in looking up space point ptr... 0x2b70a6a0, 0x2b70a5a0
Caught exception in looking up space point ptr... 0x2b70a5a0, 0x2b70a8a0
Caught exception in looking up space point ptr... 0x266c3410, 0x36efb210
Caught exception in looking up space point ptr... 0x4f575f60, 0x266c3410
Caught exception in looking up space point ptr... 0x55101980, 0x551017f0
Caught exception in looking up space point ptr... 0x551017f0, 0x55101980
Caught exception in looking up space point ptr... 0x29603c70, 0x58a5e930
Caught exception in looking up space point ptr... 0x58a5e930, 0x29603c70
Caught exception in looking up space point ptr... 0x5f677b90, 0x5f6777d0
Caught exception in looking up space point ptr... 0x5f6777d0, 0x5f6783e0
Caught exception in looking up space point ptr... 0x2f3d1f50, 0x3ff55d80
Caught exception in looking up space point ptr... 0x3ff55d80, 0x35ea69b0
Caught exception in looking up space point ptr... 0x35ea69b0, 0x3ff55bf0
Caught exception in looking up space point ptr... 0x5a0da140, 0x4f5a94c0
Caught exception in looking up space point ptr... 0x4f5a94c0, 0x5a0da140
Caught exception in looking up space point ptr... 0x1379c340, 0x1379c020
Caught exception in looking up space point ptr... 0x1379c020, 0x1379c340
Caught exception in looking up space point ptr... 0x4e534dd0, 0x4e534f60
Caught exception in looking up space point ptr... 0x4e534f60, 0x4e52f220
Caught exception in looking up space point ptr... 0x80afd80, 0x80afe80
Caught exception in looking up space point ptr... 0x80afe80, 0x80afd80
Caught exception in looking up space point ptr... 0x14c164b0, 0x4e9de050
Caught exception in looking up space point ptr... 0x4e9de050, 0x14c164b0
Caught exception in looking up space point ptr... 0x11601f50, 0x116075e0
Caught exception in looking up space point ptr... 0x116075e0, 0x11601f50
Caught exception in looking up space point ptr... 0x9176c10, 0x91768f0
Caught exception in looking up space point ptr... 0x91768f0, 0x9176c10
Caught exception in looking up space point ptr... 0x53623860, 0x536339f0
Caught exception in looking up space point ptr... 0x536339f0, 0x363d5650
Caught exception in looking up space point ptr... 0x2611dd00, 0xcad9da0
Caught exception in looking up space point ptr... 0xcad9da0, 0x551581f0
Caught exception in looking up space point ptr... 0x551581f0, 0xcad9f30
Caught exception in looking up space point ptr... 0xcad9f30, 0x2611dd00
Caught exception in looking up space point ptr... 0x121663a0, 0x12152010
Caught exception in looking up space point ptr... 0x12152010, 0x12151910
Caught exception in looking up space point ptr... 0xcada570, 0x57ac80b0
Caught exception in looking up space point ptr... 0x57ac80b0, 0xcada700
Caught exception in looking up space point ptr... 0x15420210, 0x15420080
Caught exception in looking up space point ptr... 0x507de750, 0x15420210
Caught exception in looking up space point ptr... 0x4b3dbe20, 0x4ae63970
Caught exception in looking up space point ptr... 0x4ae63970, 0x4b3dbe20
Caught exception in looking up space point ptr... 0x2611bdf0, 0x56cec640
Caught exception in looking up space point ptr... 0x56cec640, 0x2611bdf0
Caught exception in looking up space point ptr... 0x236db330, 0xacc2f30
Caught exception in looking up space point ptr... 0x5b956fa0, 0x236db330
++++>>>> total num hits: 120793, num free: 117185
04-Feb-2026 01:12:02 GMT  Closed output file "np02vd_raw_run042422_0065_df-s03-d0_dw_0_20260203T154954_reco_stage0_20260204T011202_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         198.171       320.139       673.564       292.765       108.902        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.616e-05    6.45224e-05   0.000128326   5.6916e-05    1.85026e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                101.442       199.397       558.174       179.438       99.6986        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.772104       1.17615       2.36534       1.07546      0.396824        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0230911     0.0343344     0.0495119     0.0330437    0.00878763       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.114842      0.152312       0.32062      0.140495      0.0456827       27     
produce:beamevent:BeamEvent                      5.6042e-05    0.00010168    0.000778338   7.1571e-05    0.000133252      27     
produce:pdvddaphne:DAPHNEReaderPDVD                10.7013       26.4655       59.3195       22.911        12.2991        27     
produce:ophit:OpHitFinder                         0.0266252     0.0325387     0.0362813     0.0324525    0.00230785       27     
produce:wclsdatavd:WireCellToolkit                 64.9592       70.0805       89.604        68.2041       5.02879        27     
produce:gaushit:GausHitFinder                     0.563371      0.930515       1.39237      0.945062      0.213794        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000197248   0.000353443   0.000910098   0.000321797   0.000129317      27     
produce:reco3d:SpacePointSolver                    4.45488       10.303        16.1282       10.0137       2.84924        27     
produce:hitpdune:DisambigFromSpacePoints          0.0506882     0.102443      0.187035      0.0937995     0.031615        27     
produce:cluster3d:Cluster3D                       0.609252       1.74901       5.51466       1.56123       1.01334        27     
[art]:TriggerResults:TriggerResultInserter       1.9306e-05    2.61428e-05   7.4342e-05    2.3976e-05    9.96186e-06      27     
end_path:out1:RootOutput                          8.256e-06    0.000101085   0.00244181    1.0719e-05    0.000459063      27     
end_path:out1:RootOutput(write)                    3.01829       9.69731       134.08        3.42925       24.6218        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4492.49 MB
  Peak resident set size usage (VmHWM): 2591.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0065_df-s03-d0_dw_0_20260203T154954_reco_stage0_20260204T011202_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0065_df-s03-d0_dw_0_20260203T154954_reco_stage0_20260204T011202_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0065_df-s03-d0_dw_0_20260203T154954_reco_stage0_20260204T011202_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158800.0,
    "core.end_time": 1770167616.0,
    "core.events": [
      10538,
      10544,
      10550,
      10556,
      10562,
      10568,
      10574,
      10580,
      10586,
      10592,
      10598,
      10604,
      10610,
      10616,
      10622,
      10628,
      10634,
      10640,
      10646,
      10652,
      10658,
      10664,
      10670,
      10676,
      10682,
      10688,
      10694
    ],
    "core.event_count": 27,
    "core.first_event_number": 10538,
    "core.last_event_number": 10694,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0065_df-s03-d0_dw_0_20260203T154954.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:49:05 UTC       justIN version: 01.06.00