justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.55@dunegpschedd01.fnal.gov

Jobsub ID303653.55@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 00:36:22
From worker nodeHostnamebf1770120913.novalocal
cpuinfoIntel Xeon Processor (Skylake, IBRS)
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:41:43
Input filesvd-protodune:np02vd_raw_run042421_0119_df-s03-d1_dw_0_20260203T143524.hdf5
JobscriptExit code0
Real time52m (3120s)
CPU time46m (2800s = 89%)
Max RSS bytes2610008064 (2489 MiB)
Outputting started2026-02-04 00:33:45
Output files
Finished2026-02-04 00:36:22
Saved logsjustin-logs:303653.55-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

927b0
Caught exception in looking up space point ptr... 0x11b927b0, 0x3e03bf00
Caught exception in looking up space point ptr... 0x11b92510, 0x11b92680
Caught exception in looking up space point ptr... 0x11b92680, 0x11b92510
Caught exception in looking up space point ptr... 0x639c7570, 0xbc32650
Caught exception in looking up space point ptr... 0xbc32650, 0x378b32e0
Caught exception in looking up space point ptr... 0x2d46f800, 0x2d470ee0
Caught exception in looking up space point ptr... 0x2d471380, 0x2d46f800
Caught exception in looking up space point ptr... 0x3c8ad2f0, 0x3c8aae80
Caught exception in looking up space point ptr... 0x3c8aae80, 0x3c8aaf20
Caught exception in looking up space point ptr... 0x258f8680, 0x258ebe30
Caught exception in looking up space point ptr... 0x258f7140, 0x258f8680
Caught exception in looking up space point ptr... 0x554423f0, 0x16f23c40
Caught exception in looking up space point ptr... 0x16f23c40, 0x554423f0
Caught exception in looking up space point ptr... 0x642d78f0, 0x642d77f0
Caught exception in looking up space point ptr... 0x642d77f0, 0x642d4f90
Caught exception in looking up space point ptr... 0x23c7c750, 0x23c7c5c0
Caught exception in looking up space point ptr... 0x23c7c5c0, 0x23c7c750
Caught exception in looking up space point ptr... 0xb852c70, 0xb852e10
Caught exception in looking up space point ptr... 0xb852e10, 0xb852fa0
Caught exception in looking up space point ptr... 0x642c1790, 0x3503eea0
Caught exception in looking up space point ptr... 0x12094550, 0x642c1790
Caught exception in looking up space point ptr... 0x12097b40, 0x12097d90
Caught exception in looking up space point ptr... 0x12098480, 0x12097b40
Caught exception in looking up space point ptr... 0x117b6350, 0xfdc1700
Caught exception in looking up space point ptr... 0xfdc1700, 0x117b6350
Caught exception in looking up space point ptr... 0x2db00190, 0x3789ffc0
Caught exception in looking up space point ptr... 0x3789ffc0, 0x2daffe70
Caught exception in looking up space point ptr... 0x12a5ee50, 0x378b4660
Caught exception in looking up space point ptr... 0x378a9f10, 0x12a5ee50
Caught exception in looking up space point ptr... 0x468c06d0, 0x15785630
Caught exception in looking up space point ptr... 0x15785630, 0x2c32c390
Caught exception in looking up space point ptr... 0x2a9b8de0, 0xb891230
Caught exception in looking up space point ptr... 0xb891230, 0x2a9b8de0
Caught exception in looking up space point ptr... 0x1388a2f0, 0x1388a480
Caught exception in looking up space point ptr... 0x1388a480, 0x1388a2f0
Caught exception in looking up space point ptr... 0x2a9667d0, 0x2a962840
Caught exception in looking up space point ptr... 0x2a962840, 0x2a9661e0
Caught exception in looking up space point ptr... 0x527b08a0, 0x469d7210
Caught exception in looking up space point ptr... 0x469d7210, 0x527a61b0
Caught exception in looking up space point ptr... 0x527ac140, 0xbc5e3f0
Caught exception in looking up space point ptr... 0xbc5e3f0, 0x527b08a0
Caught exception in looking up space point ptr... 0x1578de20, 0x1578e070
Caught exception in looking up space point ptr... 0x1578e2c0, 0x1578de20
Caught exception in looking up space point ptr... 0x11bb1d10, 0x11bb0680
Caught exception in looking up space point ptr... 0x11bb0680, 0x11bb1000
Caught exception in looking up space point ptr... 0x48192e90, 0x3863fbd0
Caught exception in looking up space point ptr... 0x3863fbd0, 0x48192e90
Caught exception in looking up space point ptr... 0x15e9e3a0, 0x37ecd650
Caught exception in looking up space point ptr... 0x37ecd650, 0x37ecd7e0
Caught exception in looking up space point ptr... 0x5544b6f0, 0x16f3e8c0
Caught exception in looking up space point ptr... 0x16f3e8c0, 0x5544b6f0
Caught exception in looking up space point ptr... 0x23b51f50, 0x23b51070
Caught exception in looking up space point ptr... 0x23b51070, 0x23b51f50
Caught exception in looking up space point ptr... 0x50519810, 0x5051fc60
Caught exception in looking up space point ptr... 0x5051fc60, 0x5051f7a0
Caught exception in looking up space point ptr... 0x2b4b2250, 0x226b7e20
Caught exception in looking up space point ptr... 0x226b7e20, 0x2b4b2250
Caught exception in looking up space point ptr... 0x11598d10, 0x2d4788c0
Caught exception in looking up space point ptr... 0x2d4788c0, 0x16c368d0
Caught exception in looking up space point ptr... 0x46e51b70, 0x295fb310
Caught exception in looking up space point ptr... 0x295fb310, 0x46e51b70
Caught exception in looking up space point ptr... 0x2d4a8f40, 0x2d4a8db0
Caught exception in looking up space point ptr... 0x2d4a8db0, 0x2d4a8f40
Caught exception in looking up space point ptr... 0x534a2100, 0x14e18480
Caught exception in looking up space point ptr... 0x14e18480, 0x14e18380
++++>>>> total num hits: 71977, num free: 69081
04-Feb-2026 00:33:34 GMT  Closed output file "np02vd_raw_run042421_0119_df-s03-d1_dw_0_20260203T143524_reco_stage0_20260204T003334_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         104.902       112.608       128.178       112.41        5.31133        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000131962   0.000169514   0.000257357   0.000153748   3.47514e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.2621       14.127        18.1807       14.215        1.58975        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.504211      0.525396      0.564071      0.525818      0.0122001       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0169151     0.018506      0.0298764     0.0173879    0.00331928       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0836723     0.0857315     0.103201      0.0849058    0.00356576       27     
produce:beamevent:BeamEvent                      0.000171385   0.000289189   0.00167359    0.000225985   0.000275098      27     
produce:pdvddaphne:DAPHNEReaderPDVD                4.22977       4.51164       4.92229       4.49119      0.169109        27     
produce:ophit:OpHitFinder                         0.0340105     0.0382458     0.0445698     0.0383134    0.00248887       27     
produce:wclsdatavd:WireCellToolkit                 71.9932       75.9398       88.6166       75.4378       3.30057        27     
produce:gaushit:GausHitFinder                     0.715787       1.10721       1.69049       1.1005       0.242118        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000267365   0.000446486   0.000772443   0.000451188   0.000100237      27     
produce:reco3d:SpacePointSolver                    5.90965       10.271        14.7309       10.1903       2.57741        27     
produce:hitpdune:DisambigFromSpacePoints          0.065142       0.11752      0.170241      0.116445      0.0299468       27     
produce:cluster3d:Cluster3D                       0.940743       1.98435       4.44638       1.84622      0.883334        27     
[art]:TriggerResults:TriggerResultInserter       2.7145e-05    3.55747e-05   7.8177e-05    3.2444e-05    9.58513e-06      27     
end_path:out1:RootOutput                          4.811e-06    1.06529e-05   1.9117e-05     9.785e-06    4.46882e-06      27     
end_path:out1:RootOutput(write)                    3.45152       3.84443       4.24345       3.81174      0.216009        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4510.57 MB
  Peak resident set size usage (VmHWM): 2610.01 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0119_df-s03-d1_dw_0_20260203T143524_reco_stage0_20260204T003334_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0119_df-s03-d1_dw_0_20260203T143524_reco_stage0_20260204T003334_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0119_df-s03-d1_dw_0_20260203T143524_reco_stage0_20260204T003334_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162112.0,
    "core.end_time": 1770165217.0,
    "core.events": [
      19347,
      19353,
      19359,
      19365,
      19371,
      19377,
      19383,
      19389,
      19395,
      19401,
      19407,
      19413,
      19419,
      19425,
      19431,
      19437,
      19443,
      19449,
      19455,
      19461,
      19467,
      19473,
      19479,
      19485,
      19491,
      19497,
      19503
    ],
    "core.event_count": 27,
    "core.first_event_number": 19347,
    "core.last_event_number": 19503,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0119_df-s03-d1_dw_0_20260203T143524.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:05 UTC       justIN version: 01.06.00