justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.112@dunegpschedd02.fnal.gov

Jobsub ID297923.112@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 01:32:09
From worker nodeHostnamecn515.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:03:32
Input filesvd-protodune:np02vd_raw_run042433_0560_df-s03-d1_dw_0_20260203T221459.hdf5
JobscriptExit code0
Real time1h (5159s)
CPU time1h (4925s = 95%)
Max RSS bytes3369820160 (3213 MiB)
Outputting started2026-02-04 01:29:33
Output files
Finished2026-02-04 01:32:09
Saved logsjustin-logs:297923.112-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x398d2860, 0x398d71a0
Caught exception in looking up space point ptr... 0x4c434bd0, 0x6b5f3800
Caught exception in looking up space point ptr... 0x6b5f3800, 0x50cb17d0
Caught exception in looking up space point ptr... 0x53f71a80, 0x53f718a0
Caught exception in looking up space point ptr... 0x53f718a0, 0x53f714e0
Caught exception in looking up space point ptr... 0x53f714e0, 0x6c812130
Caught exception in looking up space point ptr... 0x6c812130, 0x6c8122c0
Caught exception in looking up space point ptr... 0x6c8122c0, 0x6c812450
Caught exception in looking up space point ptr... 0x6c812450, 0x53f71a80
Caught exception in looking up space point ptr... 0x5ff05450, 0x5ff051b0
Caught exception in looking up space point ptr... 0x5ff051b0, 0x5ff05450
Caught exception in looking up space point ptr... 0x11325760, 0x5ed4cec0
Caught exception in looking up space point ptr... 0x5ed4cec0, 0x5ed3ddf0
Caught exception in looking up space point ptr... 0x3b3989b0, 0x67875cb0
Caught exception in looking up space point ptr... 0x4964fed0, 0x3b3989b0
Caught exception in looking up space point ptr... 0x52ac7ee0, 0xd405970
Caught exception in looking up space point ptr... 0xd405970, 0x52ac7ee0
Caught exception in looking up space point ptr... 0x17d26cc0, 0x17d26e50
Caught exception in looking up space point ptr... 0x17d26e50, 0x17d26fe0
Caught exception in looking up space point ptr... 0x53f823b0, 0x53f80a10
Caught exception in looking up space point ptr... 0x53f80a10, 0x53f823b0
Caught exception in looking up space point ptr... 0x53f823b0, 0x53f83050
Caught exception in looking up space point ptr... 0x53f83050, 0x53f823b0
Caught exception in looking up space point ptr... 0x4cd5b2a0, 0x4cd5d270
Caught exception in looking up space point ptr... 0x4cd5d270, 0x4cd5b2a0
Caught exception in looking up space point ptr... 0x4cd5b2a0, 0x4cd5d370
Caught exception in looking up space point ptr... 0x4cd5d370, 0x4cd5b2a0
Caught exception in looking up space point ptr... 0x36ea2d20, 0x3dc106c0
Caught exception in looking up space point ptr... 0x3dc106c0, 0x4e7a3e40
Caught exception in looking up space point ptr... 0x4cd54820, 0x4cd536d0
Caught exception in looking up space point ptr... 0x4cd536d0, 0x4cd54820
Caught exception in looking up space point ptr... 0x2a627210, 0x2a6273a0
Caught exception in looking up space point ptr... 0x2a6273a0, 0x3b081d50
Caught exception in looking up space point ptr... 0x3b080290, 0x2a629800
Caught exception in looking up space point ptr... 0x2a629800, 0x2a627210
Caught exception in looking up space point ptr... 0x60af22c0, 0x60b0eb30
Caught exception in looking up space point ptr... 0x60b0eb30, 0x60af22c0
Caught exception in looking up space point ptr... 0x6df49b60, 0x6df483a0
Caught exception in looking up space point ptr... 0x6df483a0, 0x6df49b60
Caught exception in looking up space point ptr... 0x6e77d690, 0x6e77d820
Caught exception in looking up space point ptr... 0x6e77d820, 0x6e77d690
Caught exception in looking up space point ptr... 0x84dab390, 0x84dab140
Caught exception in looking up space point ptr... 0x84dab140, 0x84dab390
Caught exception in looking up space point ptr... 0x51858170, 0x518584a0
Caught exception in looking up space point ptr... 0x518584a0, 0x51858170
Caught exception in looking up space point ptr... 0x101c3780, 0xfbb7c90
Caught exception in looking up space point ptr... 0xfbb7c90, 0x17cff7f0
Caught exception in looking up space point ptr... 0xec876f0, 0xec88820
Caught exception in looking up space point ptr... 0xec88820, 0xec84160
Caught exception in looking up space point ptr... 0xd2de910, 0xd6c4130
Caught exception in looking up space point ptr... 0xd6c4130, 0x8139bbc0
Caught exception in looking up space point ptr... 0x518eccc0, 0x519622e0
Caught exception in looking up space point ptr... 0x519622e0, 0x518eccc0
Caught exception in looking up space point ptr... 0x6f9022d0, 0x6f9023a0
Caught exception in looking up space point ptr... 0x81353430, 0x6f9022d0
Caught exception in looking up space point ptr... 0x17d060d0, 0x17d05d60
Caught exception in looking up space point ptr... 0x17d05d60, 0x17d060d0
Caught exception in looking up space point ptr... 0x17d07e00, 0x17d07f90
Caught exception in looking up space point ptr... 0x17d07f90, 0x17d07b20
Caught exception in looking up space point ptr... 0x17d07b20, 0x17d082b0
Caught exception in looking up space point ptr... 0x17d082b0, 0x17d07e00
Caught exception in looking up space point ptr... 0x63827fc0, 0x60b8fd00
Caught exception in looking up space point ptr... 0x3c5c4c00, 0x63827fc0
Caught exception in looking up space point ptr... 0x16a6c0e0, 0x16a6bdc0
Caught exception in looking up space point ptr... 0x16a6bdc0, 0x16a6c0e0
Caught exception in looking up space point ptr... 0x1158bf30, 0x1158c0a0
Caught exception in looking up space point ptr... 0x1158c0a0, 0x1158bf30
++++>>>> total num hits: 69701, num free: 66608
04-Feb-2026 01:29:21 GMT  Closed output file "np02vd_raw_run042433_0560_df-s03-d1_dw_0_20260203T221459_reco_stage0_20260204T012921_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         209.266       265.431       331.688       266.218       34.5193        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.6534e-05    0.000133334   0.000269869   0.000128649   4.31677e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                16.2301       16.675        17.3388       16.6926      0.284138        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.490888      0.508824      0.545158      0.505224      0.0150471       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165517     0.0173508     0.0237943     0.016816     0.00166967       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0166251     0.0167806     0.0171458     0.0167498    0.000138646      19     
produce:beamevent:BeamEvent                      0.000149017   0.000264037   0.00147849    0.000196909   0.000288378      19     
produce:pdvddaphne:DAPHNEReaderPDVD                6.03509       7.27907       9.1935        7.29077      0.864522        19     
produce:ophit:OpHitFinder                         0.0392908     0.0656296     0.0857404     0.0713684     0.0147626       19     
produce:wclsdatavd:WireCellToolkit                 158.76        203.008       261.644       202.909       31.5188        19     
produce:gaushit:GausHitFinder                      1.37639       2.38739       3.25574        2.322       0.441539        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000471055   0.000738508   0.00152034    0.000674712   0.000222488      19     
produce:reco3d:SpacePointSolver                    12.2437       23.0103       29.3346       23.2521       4.18176        19     
produce:hitpdune:DisambigFromSpacePoints          0.171349      0.426667      0.732205      0.421857      0.133675        19     
produce:cluster3d:Cluster3D                        1.25905       3.68724       6.21501       3.53409       1.21051        19     
[art]:TriggerResults:TriggerResultInserter       3.1541e-05    4.62319e-05   0.000125389   4.1715e-05    1.99986e-05      19     
end_path:out1:RootOutput                          7.281e-06    1.18366e-05   3.3636e-05    1.0745e-05    5.52911e-06      19     
end_path:out1:RootOutput(write)                    5.86924       8.28257       10.5541       9.04313       1.80328        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5311.3 MB
  Peak resident set size usage (VmHWM): 3369.82 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0560_df-s03-d1_dw_0_20260203T221459_reco_stage0_20260204T012921_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0560_df-s03-d1_dw_0_20260203T221459_reco_stage0_20260204T012921_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0560_df-s03-d1_dw_0_20260203T221459_reco_stage0_20260204T012921_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163423.0,
    "core.end_time": 1770168564.0,
    "core.events": [
      63849,
      63855,
      63861,
      63867,
      63873,
      63879,
      63885,
      63891,
      63897,
      63903,
      63909,
      63915,
      63921,
      63927,
      63933,
      63939,
      63945,
      63951,
      63957
    ],
    "core.event_count": 19,
    "core.first_event_number": 63849,
    "core.last_event_number": 63957,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0560_df-s03-d1_dw_0_20260203T221459.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:37:00 UTC       justIN version: 01.06.00