justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298019.1@dunegpschedd02.fnal.gov

Jobsub ID298019.1@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:52:31
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 02:55:32
From worker nodeHostnamewn2209251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:52:56
Input filesvd-protodune:np02vd_raw_run042422_0083_df-s03-d0_dw_0_20260203T155940.hdf5
JobscriptExit code0
Real time59m (3591s)
CPU time50m (3048s = 84%)
Max RSS bytes2592686080 (2472 MiB)
Outputting started2026-02-04 02:52:48
Output files
Finished2026-02-04 02:55:32
Saved logsjustin-logs:298019.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

2fe5f0
Caught exception in looking up space point ptr... 0xf180c10, 0x25ce7d50
Caught exception in looking up space point ptr... 0x25ce7d50, 0xf180c10
Caught exception in looking up space point ptr... 0x2f2e0960, 0x50c02120
Caught exception in looking up space point ptr... 0x50c02120, 0x50c01e00
Caught exception in looking up space point ptr... 0x2ea5a400, 0x2ea5a590
Caught exception in looking up space point ptr... 0x2ea5a590, 0x2ea5bf40
Caught exception in looking up space point ptr... 0x438ba7e0, 0x4f7660c0
Caught exception in looking up space point ptr... 0x4f7660c0, 0x438ba7e0
Caught exception in looking up space point ptr... 0x3af10b70, 0x3af109e0
Caught exception in looking up space point ptr... 0x3af109e0, 0x3af10b70
Caught exception in looking up space point ptr... 0xf060540, 0x272aedc0
Caught exception in looking up space point ptr... 0x272aedc0, 0x4db7ac80
Caught exception in looking up space point ptr... 0x3cf80fb0, 0x3cf81120
Caught exception in looking up space point ptr... 0x3cf81120, 0x3cf80fb0
Caught exception in looking up space point ptr... 0x3cf80fb0, 0x2d74e530
Caught exception in looking up space point ptr... 0x2d74e530, 0x3cf80fb0
Caught exception in looking up space point ptr... 0x38ae0f80, 0x38ae1160
Caught exception in looking up space point ptr... 0x38ae1160, 0x38ae1340
Caught exception in looking up space point ptr... 0x1591fb30, 0x13c9bff0
Caught exception in looking up space point ptr... 0x13c9bff0, 0x1591fb30
Caught exception in looking up space point ptr... 0x3e06b160, 0x530398a0
Caught exception in looking up space point ptr... 0x530398a0, 0x52237f70
Caught exception in looking up space point ptr... 0x52237f70, 0x443bc4e0
Caught exception in looking up space point ptr... 0x2114e5e0, 0x2114e4b0
Caught exception in looking up space point ptr... 0x2114e4b0, 0x21157490
Caught exception in looking up space point ptr... 0x43826e70, 0x4a761bc0
Caught exception in looking up space point ptr... 0x4a761bc0, 0x52419020
Caught exception in looking up space point ptr... 0x31cb9b10, 0x5229d060
Caught exception in looking up space point ptr... 0x5229d060, 0x939e970
Caught exception in looking up space point ptr... 0x116ecfa0, 0x3e9976b0
Caught exception in looking up space point ptr... 0x3e9976b0, 0x2fca7510
Caught exception in looking up space point ptr... 0x331b70d0, 0x242b6680
Caught exception in looking up space point ptr... 0x242b6680, 0x331b70d0
Caught exception in looking up space point ptr... 0x2da03dd0, 0x2da03c60
Caught exception in looking up space point ptr... 0x2da03c60, 0x2da03dd0
Caught exception in looking up space point ptr... 0x14e76710, 0x50b72e70
Caught exception in looking up space point ptr... 0x2f3379d0, 0x14e76710
Caught exception in looking up space point ptr... 0x98f4f20, 0x3a99d0b0
Caught exception in looking up space point ptr... 0x3a99d0b0, 0x98f4f20
Caught exception in looking up space point ptr... 0x28aae120, 0xe2145d0
Caught exception in looking up space point ptr... 0xe2145d0, 0x28aae120
Caught exception in looking up space point ptr... 0x52355670, 0x2bf3ac30
Caught exception in looking up space point ptr... 0x2bf3ac30, 0x52355670
Caught exception in looking up space point ptr... 0x524aed50, 0x30ab7c00
Caught exception in looking up space point ptr... 0x30ab7c00, 0x524aed50
Caught exception in looking up space point ptr... 0x1135a020, 0x979c850
Caught exception in looking up space point ptr... 0x979c850, 0x11358e50
Caught exception in looking up space point ptr... 0x21156650, 0x21156520
Caught exception in looking up space point ptr... 0x21156520, 0x21156650
Caught exception in looking up space point ptr... 0x4b11b0f0, 0x2f37a130
Caught exception in looking up space point ptr... 0x2f37a130, 0x4b11b0f0
Caught exception in looking up space point ptr... 0x2788b6f0, 0x2788b450
Caught exception in looking up space point ptr... 0x2788b450, 0x2788b530
Caught exception in looking up space point ptr... 0x2ff02310, 0x52ab1060
Caught exception in looking up space point ptr... 0x52ab1060, 0x2ff02310
Caught exception in looking up space point ptr... 0xb3a7490, 0xa284c60
Caught exception in looking up space point ptr... 0x1591fed0, 0xb3a7490
Caught exception in looking up space point ptr... 0x49c7fed0, 0x49c81a10
Caught exception in looking up space point ptr... 0x49c81a10, 0x16cb1030
Caught exception in looking up space point ptr... 0x568f7fc0, 0x2bfccdf0
Caught exception in looking up space point ptr... 0x2bfccdf0, 0x273dc580
Caught exception in looking up space point ptr... 0x592729d0, 0x57a71660
Caught exception in looking up space point ptr... 0x57a71660, 0x592729d0
Caught exception in looking up space point ptr... 0x592729d0, 0x59272b60
Caught exception in looking up space point ptr... 0x59272b60, 0x592729d0
++++>>>> total num hits: 53000, num free: 51099
04-Feb-2026 02:52:38 GMT  Closed output file "np02vd_raw_run042422_0083_df-s03-d0_dw_0_20260203T155940_reco_stage0_20260204T025238_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         105.777       128.642       157.291       128.127       9.85735        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.4851e-05    0.000131871   0.000255961   0.000111851   5.30137e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                15.1465       18.306        39.6557       17.5888       4.30116        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.655548      0.716474       1.63076      0.676293      0.180868        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0217543     0.0224282     0.0246962     0.0222785    0.000565909      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder            0.10857      0.114851      0.134022      0.110293     0.00895511       27     
produce:beamevent:BeamEvent                      9.7804e-05    0.000230103   0.00145157    0.000180039   0.000244081      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.5632        4.28379       5.30007       4.28288      0.384566        27     
produce:ophit:OpHitFinder                         0.0358622     0.0450508     0.0572211     0.044451     0.00423106       27     
produce:wclsdatavd:WireCellToolkit                 66.0047       81.5147       90.009        82.6218       5.75069        27     
produce:gaushit:GausHitFinder                      0.6093        1.16511       1.78436       1.14908       0.27509        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00020718    0.000385269   0.000675781   0.000382921   9.85581e-05      27     
produce:reco3d:SpacePointSolver                    10.1345       14.7113       24.9799       15.333        3.1153         27     
produce:hitpdune:DisambigFromSpacePoints          0.0749829     0.137561      0.247944      0.130454      0.0355569       27     
produce:cluster3d:Cluster3D                        1.60285       2.87015       6.25225       2.68621       0.93161        27     
[art]:TriggerResults:TriggerResultInserter       2.8574e-05    4.15111e-05   0.000127821   3.6308e-05    1.83898e-05      27     
end_path:out1:RootOutput                          5.511e-06    9.96993e-06    3.698e-05     9.378e-06    5.59247e-06      27     
end_path:out1:RootOutput(write)                    3.70474       4.71424       5.9162        4.89072      0.660276        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4537.39 MB
  Peak resident set size usage (VmHWM): 2592.69 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0083_df-s03-d0_dw_0_20260203T155940_reco_stage0_20260204T025238_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0083_df-s03-d0_dw_0_20260203T155940_reco_stage0_20260204T025238_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0083_df-s03-d0_dw_0_20260203T155940_reco_stage0_20260204T025238_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770169990.0,
    "core.end_time": 1770173560.0,
    "core.events": [
      13460,
      13466,
      13472,
      13478,
      13484,
      13490,
      13496,
      13502,
      13508,
      13514,
      13520,
      13526,
      13532,
      13538,
      13544,
      13550,
      13556,
      13562,
      13568,
      13574,
      13580,
      13586,
      13592,
      13598,
      13604,
      13610,
      13616
    ],
    "core.event_count": 27,
    "core.first_event_number": 13460,
    "core.last_event_number": 13616,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0083_df-s03-d0_dw_0_20260203T155940.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:19:15 UTC       justIN version: 01.06.00