justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.111@dunegpschedd01.fnal.gov

Jobsub ID303652.111@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 03:02:34
From worker nodeHostnamewn1908240.tier2.hep.manchester.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:34
Input filesvd-protodune:np02vd_raw_run042433_0355_df-s02-d1_dw_0_20260203T205701.hdf5
JobscriptExit code0
Real time3h (12576s)
CPU time3h (11996s = 95%)
Max RSS bytes8046858240 (7674 MiB)
Outputting started2026-02-04 02:58:21
Output files
Finished2026-02-04 03:02:34
Saved logsjustin-logs:303652.111-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

r... 0xb867a890, 0xb867aa50
Caught exception in looking up space point ptr... 0xb867aa50, 0xb8649850
Caught exception in looking up space point ptr... 0xc80bdba0, 0xc80bdd30
Caught exception in looking up space point ptr... 0xc80bdd30, 0xc80bd8b0
Caught exception in looking up space point ptr... 0x355d6360, 0x60de4df0
Caught exception in looking up space point ptr... 0x60de4df0, 0x60de70e0
Caught exception in looking up space point ptr... 0x60de70e0, 0x60de13e0
Caught exception in looking up space point ptr... 0x3bd2adc0, 0x3bd2a980
Caught exception in looking up space point ptr... 0x3bd2a980, 0x3bd42480
Caught exception in looking up space point ptr... 0xad9d6b40, 0xad9d6600
Caught exception in looking up space point ptr... 0xad9d6600, 0xad9d5f70
Caught exception in looking up space point ptr... 0x57637700, 0x57628ca0
Caught exception in looking up space point ptr... 0x576580e0, 0x57637700
Caught exception in looking up space point ptr... 0x7981e5c0, 0x4666fd40
Caught exception in looking up space point ptr... 0x4666fea0, 0x7981e5c0
Caught exception in looking up space point ptr... 0x821f11f0, 0x11ca27a0
Caught exception in looking up space point ptr... 0x11ca27a0, 0x6fe409c0
Caught exception in looking up space point ptr... 0x93f2f800, 0x36960ef0
Caught exception in looking up space point ptr... 0x36960ef0, 0x93f2f800
Caught exception in looking up space point ptr... 0x6048b910, 0x6048b6c0
Caught exception in looking up space point ptr... 0x6048b6c0, 0x60495650
Caught exception in looking up space point ptr... 0x13d06840, 0x13d16560
Caught exception in looking up space point ptr... 0x13d16560, 0x13d166f0
Caught exception in looking up space point ptr... 0x2c6c5340, 0x2c6b0bc0
Caught exception in looking up space point ptr... 0x2c6c51b0, 0x2c6c5340
Caught exception in looking up space point ptr... 0x6dd94f20, 0x988427e0
Caught exception in looking up space point ptr... 0x988427e0, 0x6dd94f20
Caught exception in looking up space point ptr... 0x3a6f5b60, 0x6fd431c0
Caught exception in looking up space point ptr... 0x6fd431c0, 0x6fd30f20
Caught exception in looking up space point ptr... 0x75f1c330, 0x75f1c770
Caught exception in looking up space point ptr... 0x75f1c770, 0x75f1c0b0
Caught exception in looking up space point ptr... 0x62030fa0, 0x620b7780
Caught exception in looking up space point ptr... 0x9c3c6fd0, 0x62030fa0
Caught exception in looking up space point ptr... 0x4b8f1880, 0x5ac3d560
Caught exception in looking up space point ptr... 0x5ac3d560, 0x5ac3d6b0
Caught exception in looking up space point ptr... 0x11e7b9f0, 0xd916390
Caught exception in looking up space point ptr... 0xd9252a0, 0x11e7b9f0
Caught exception in looking up space point ptr... 0x3d8ec140, 0x7052f820
Caught exception in looking up space point ptr... 0x7052f820, 0x7052f9b0
Caught exception in looking up space point ptr... 0x72051590, 0x3d8db530
Caught exception in looking up space point ptr... 0x720518b0, 0x72051590
Caught exception in looking up space point ptr... 0x30657d20, 0x577e2a50
Caught exception in looking up space point ptr... 0x577e2a50, 0x7783bef0
Caught exception in looking up space point ptr... 0xd07d0510, 0x5a8e23e0
Caught exception in looking up space point ptr... 0x6e084b00, 0xd07d0510
Caught exception in looking up space point ptr... 0x99816a0, 0x9c45bde0
Caught exception in looking up space point ptr... 0x9c45bde0, 0xc7355d70
Caught exception in looking up space point ptr... 0x15d8b4c0, 0x15d40640
Caught exception in looking up space point ptr... 0x15d40640, 0x3b1225d0
Caught exception in looking up space point ptr... 0xf3d0eb20, 0xb1813a40
Caught exception in looking up space point ptr... 0xb1810430, 0xf3d0eb20
Caught exception in looking up space point ptr... 0x15dd4cd0, 0xd99ac70
Caught exception in looking up space point ptr... 0xc734b980, 0x15dd4cd0
Caught exception in looking up space point ptr... 0x5f341610, 0x5f331a80
Caught exception in looking up space point ptr... 0x5f331a80, 0x5f321720
Caught exception in looking up space point ptr... 0x10ec2530, 0x9c3cede0
Caught exception in looking up space point ptr... 0x12419650, 0x10ec2530
Caught exception in looking up space point ptr... 0x96127ad0, 0x36a558c0
Caught exception in looking up space point ptr... 0x36a558c0, 0x36a556f0
Caught exception in looking up space point ptr... 0x3fc470c0, 0x3fc46f70
Caught exception in looking up space point ptr... 0x3fc46f70, 0x3fc46ea0
Caught exception in looking up space point ptr... 0x3b1ccba0, 0x3b15dcd0
Caught exception in looking up space point ptr... 0x3b15dcd0, 0xaeca9e0
Caught exception in looking up space point ptr... 0x3fc77bf0, 0x4cd1fbe0
Caught exception in looking up space point ptr... 0x3ee3c3b0, 0x3fc77bf0
Caught exception in looking up space point ptr... 0x5c475a30, 0x31ff3ee0
Caught exception in looking up space point ptr... 0x31ff3ee0, 0x5c475cd0
++++>>>> total num hits: 566704, num free: 557776
04-Feb-2026 02:57:55 GMT  Closed output file "np02vd_raw_run042433_0355_df-s02-d1_dw_0_20260203T205701_reco_stage0_20260204T025755_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         341.745       653.891       1362.65       471.271       369.143        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000120085   0.00024974    0.000794011   0.000236992   0.000144166      19     
produce:tpcrawdecoder:PDVDTPCReader                 17.82        19.6765       21.5753       19.4468      0.856924        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.654259      0.672478      0.705718      0.665635      0.0156512       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0220387     0.0223717     0.0227288     0.022355     0.000145793      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0221315     0.0224482     0.0236536     0.022376     0.000321614      19     
produce:beamevent:BeamEvent                      0.000174807   0.000309358     0.00132     0.00024756    0.000244329      19     
produce:pdvddaphne:DAPHNEReaderPDVD                7.7267        8.92458       10.1302       9.06748      0.677551        19     
produce:ophit:OpHitFinder                         0.0543294     0.0763095     0.0944998     0.0782419    0.00896243       19     
produce:wclsdatavd:WireCellToolkit                 256.937       275.083        298.3        273.923       11.3448        19     
produce:gaushit:GausHitFinder                      3.1206        5.47962       12.9219       4.7009        2.3797         19     
produce:nhitsfilter:NumberOfHitsFilter           0.000690423   0.00105957     0.0016133    0.00100585    0.000288395      19     
produce:reco3d:SpacePointSolver                    19.2298       43.0702       86.0893       37.873        19.9047        19     
produce:hitpdune:DisambigFromSpacePoints          0.508019       1.44848       4.17103      0.729792       1.33275        19     
produce:cluster3d:Cluster3D                        5.65542       281.922       927.934       120.799       343.52         19     
[art]:TriggerResults:TriggerResultInserter       5.7194e-05    8.59253e-05   0.000164274   8.1553e-05    2.38633e-05      19     
end_path:out1:RootOutput                         1.2224e-05    2.15826e-05   5.8567e-05    1.9551e-05    1.01051e-05      19     
end_path:out1:RootOutput(write)                    11.1355       17.3769       33.2425       13.5857       8.00696        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10464.3 MB
  Peak resident set size usage (VmHWM): 8046.86 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0355_df-s02-d1_dw_0_20260203T205701_reco_stage0_20260204T025755_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0355_df-s02-d1_dw_0_20260203T205701_reco_stage0_20260204T025755_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0355_df-s02-d1_dw_0_20260203T205701_reco_stage0_20260204T025755_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161328.0,
    "core.end_time": 1770173882.0,
    "core.events": [
      40477,
      40483,
      40489,
      40495,
      40501,
      40507,
      40513,
      40519,
      40525,
      40531,
      40537,
      40543,
      40549,
      40555,
      40561,
      40567,
      40573,
      40579,
      40585
    ],
    "core.event_count": 19,
    "core.first_event_number": 40477,
    "core.last_event_number": 40585,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0355_df-s02-d1_dw_0_20260203T205701.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:27:11 UTC       justIN version: 01.06.00