justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.120@dunegpschedd01.fnal.gov

Jobsub ID303654.120@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce04
Last heartbeat2026-02-04 01:43:44
From worker nodeHostnamedune001-1520744.0-lcg2567.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:09:43
Input filesvd-protodune:np02vd_raw_run042433_0512_df-s02-d1_dw_0_20260203T215643.hdf5
JobscriptExit code0
Real time1h (5506s)
CPU time1h (5177s = 94%)
Max RSS bytes3570475008 (3405 MiB)
Outputting started2026-02-04 01:41:32
Output files
Finished2026-02-04 01:43:44
Saved logsjustin-logs:303654.120-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x5e681710, 0x1ee06230
Caught exception in looking up space point ptr... 0x691f5840, 0x2b512730
Caught exception in looking up space point ptr... 0x53f5d390, 0x691f5840
Caught exception in looking up space point ptr... 0x93b50ae0, 0x93b50630
Caught exception in looking up space point ptr... 0x93b50630, 0x93b50ae0
Caught exception in looking up space point ptr... 0x944e1710, 0x9b62ff90
Caught exception in looking up space point ptr... 0x9b62ff90, 0x7eb4c3d0
Caught exception in looking up space point ptr... 0x44104940, 0x44103ca0
Caught exception in looking up space point ptr... 0x44103ca0, 0x44104ce0
Caught exception in looking up space point ptr... 0x5dfc4f70, 0x4ad22f60
Caught exception in looking up space point ptr... 0x4ad22f60, 0x71a9a2a0
Caught exception in looking up space point ptr... 0x64f6b780, 0x789329c0
Caught exception in looking up space point ptr... 0x789329c0, 0x79d69040
Caught exception in looking up space point ptr... 0x6426fe50, 0x82b40030
Caught exception in looking up space point ptr... 0x82b40030, 0x6426fe50
Caught exception in looking up space point ptr... 0x9aeb87a0, 0x4423a780
Caught exception in looking up space point ptr... 0x4423a780, 0x4423a520
Caught exception in looking up space point ptr... 0x66ecef80, 0x66ece860
Caught exception in looking up space point ptr... 0x66ece860, 0x66ecef80
Caught exception in looking up space point ptr... 0x22389070, 0x525d9440
Caught exception in looking up space point ptr... 0x525d9440, 0x22389070
Caught exception in looking up space point ptr... 0x446b2ac0, 0x5f2f1d00
Caught exception in looking up space point ptr... 0x5f2f1d00, 0x446b2ac0
Caught exception in looking up space point ptr... 0x52f84030, 0x31e27c00
Caught exception in looking up space point ptr... 0x31e27c00, 0x8bfa72d0
Caught exception in looking up space point ptr... 0x5f2299e0, 0x93e3fea0
Caught exception in looking up space point ptr... 0x93e3fea0, 0x5f2299e0
Caught exception in looking up space point ptr... 0x7e7bf820, 0x7e7bf5c0
Caught exception in looking up space point ptr... 0x7e7bf5c0, 0x7e7bf820
Caught exception in looking up space point ptr... 0x8d17b5c0, 0x5ffea780
Caught exception in looking up space point ptr... 0x5ffea780, 0x5ffea5f0
Caught exception in looking up space point ptr... 0x22bc0e40, 0x22bc1110
Caught exception in looking up space point ptr... 0x22bc1110, 0x22bbdd60
Caught exception in looking up space point ptr... 0x583b1380, 0x2224cdb0
Caught exception in looking up space point ptr... 0x2224cdb0, 0x583b1280
Caught exception in looking up space point ptr... 0x7e2b23e0, 0x58daca10
Caught exception in looking up space point ptr... 0x58daca10, 0x7e2b23e0
Caught exception in looking up space point ptr... 0x4b31f260, 0x4b31f3b0
Caught exception in looking up space point ptr... 0x4b31f3b0, 0x4b31f690
Caught exception in looking up space point ptr... 0x52f83750, 0x9ece5070
Caught exception in looking up space point ptr... 0x9ece5070, 0x52f83750
Caught exception in looking up space point ptr... 0x52f83750, 0x52f83870
Caught exception in looking up space point ptr... 0x52f83870, 0x52f83750
Caught exception in looking up space point ptr... 0x52f83750, 0x30852810
Caught exception in looking up space point ptr... 0x30852810, 0x52f83750
Caught exception in looking up space point ptr... 0x68fc1c50, 0x5147cfb0
Caught exception in looking up space point ptr... 0x5147cfb0, 0x458cb620
Caught exception in looking up space point ptr... 0x2fcb4da0, 0x2fcb6040
Caught exception in looking up space point ptr... 0x2fcb6040, 0x640773a0
Caught exception in looking up space point ptr... 0x2fcb4f30, 0x2fcb5050
Caught exception in looking up space point ptr... 0x2fcb5050, 0x2fcb4da0
Caught exception in looking up space point ptr... 0x5ac442d0, 0x58d2e1f0
Caught exception in looking up space point ptr... 0x58d2e1f0, 0x58d2c050
Caught exception in looking up space point ptr... 0x68fbe7f0, 0x9c981550
Caught exception in looking up space point ptr... 0x9c981550, 0x68fbe7f0
Caught exception in looking up space point ptr... 0x82592830, 0x21cddec0
Caught exception in looking up space point ptr... 0x21cddec0, 0x82592830
Caught exception in looking up space point ptr... 0x516c6fc0, 0x42014720
Caught exception in looking up space point ptr... 0x42014720, 0x516c6fc0
Caught exception in looking up space point ptr... 0x24c779d0, 0x24c76c50
Caught exception in looking up space point ptr... 0x24c76c50, 0x24c779d0
Caught exception in looking up space point ptr... 0x24c779d0, 0x24c76ac0
Caught exception in looking up space point ptr... 0x24c76ac0, 0x24c779d0
Caught exception in looking up space point ptr... 0x24c779d0, 0x82e19890
Caught exception in looking up space point ptr... 0x82e19890, 0x24c779d0
Caught exception in looking up space point ptr... 0x2b41fc00, 0x31650820
Caught exception in looking up space point ptr... 0x31650820, 0x2b41fc00
++++>>>> total num hits: 73484, num free: 70592
04-Feb-2026 01:41:18 UTC  Closed output file "np02vd_raw_run042433_0512_df-s02-d1_dw_0_20260203T215643_reco_stage0_20260204T014118_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         211.459       283.578       352.848       294.803       51.7587        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000124822   0.000199763   0.000528688   0.000149353   0.000103573      19     
produce:tpcrawdecoder:PDVDTPCReader                18.5475       19.2906       19.8299       19.2456       0.35101        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.687739      0.707715      0.739889      0.706384      0.0139818       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0231871     0.0235161     0.024143      0.0234817    0.000224059      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0230789     0.0236146     0.0266872     0.0234435    0.000756941      19     
produce:beamevent:BeamEvent                      0.000136273   0.000405865   0.00247641    0.000261775   0.000498758      19     
produce:pdvddaphne:DAPHNEReaderPDVD                4.02937       4.55022       4.92778       4.48914      0.238484        19     
produce:ophit:OpHitFinder                         0.0444927     0.0547275     0.0696154     0.0542454    0.00620552       19     
produce:wclsdatavd:WireCellToolkit                 138.718       172.485       186.114       173.021       11.7925        19     
produce:gaushit:GausHitFinder                      1.43034       2.87517       4.54671       2.51526       1.02982        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000386846   0.000592408   0.00105947    0.000562569   0.000144552      19     
produce:reco3d:SpacePointSolver                    12.4358       24.6507       35.2032       26.2418       6.09534        19     
produce:hitpdune:DisambigFromSpacePoints           0.19067      0.420323      0.671544      0.361805       0.13827        19     
produce:cluster3d:Cluster3D                        2.05378       49.1457       105.444       78.7929       43.8505        19     
[art]:TriggerResults:TriggerResultInserter        2.55e-05     5.15316e-05   0.000179593   4.4071e-05    3.35466e-05      19     
end_path:out1:RootOutput                          4.66e-06     1.22713e-05   4.4831e-05     1.046e-05    9.10597e-06      19     
end_path:out1:RootOutput(write)                    6.20456       9.27897       11.371        8.64123       1.40594        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5701.45 MB
  Peak resident set size usage (VmHWM): 3570.48 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0512_df-s02-d1_dw_0_20260203T215643_reco_stage0_20260204T014118_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0512_df-s02-d1_dw_0_20260203T215643_reco_stage0_20260204T014118_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0512_df-s02-d1_dw_0_20260203T215643_reco_stage0_20260204T014118_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163799.0,
    "core.end_time": 1770169280.0,
    "core.events": [
      58375,
      58381,
      58387,
      58393,
      58399,
      58405,
      58411,
      58417,
      58423,
      58429,
      58435,
      58441,
      58447,
      58453,
      58459,
      58465,
      58471,
      58477,
      58483
    ],
    "core.event_count": 19,
    "core.first_event_number": 58375,
    "core.last_event_number": 58483,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0512_df-s02-d1_dw_0_20260203T215643.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:32:42 UTC       justIN version: 01.06.00