justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.143@dunegpschedd01.fnal.gov

Jobsub ID303653.143@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 02:57:16
From worker nodeHostnamewn1208250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:46:07
Input filesvd-protodune:np02vd_raw_run042421_0098_df-s02-d1_dw_0_20260203T142400.hdf5
JobscriptExit code0
Real time3h (11266s)
CPU time1h (5500s = 48%)
Max RSS bytes2685530112 (2561 MiB)
Outputting started2026-02-04 02:53:55
Output files
Finished2026-02-04 02:57:16
Saved logsjustin-logs:303653.143-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t exception in looking up space point ptr... 0x3fc45ef0, 0x3fc45d70
Caught exception in looking up space point ptr... 0x46718ff0, 0x467079d0
Caught exception in looking up space point ptr... 0x3a8aaf60, 0x46718ff0
Caught exception in looking up space point ptr... 0x2a64e5e0, 0x2a641160
Caught exception in looking up space point ptr... 0x535688f0, 0x2a64e5e0
Caught exception in looking up space point ptr... 0x2a645f80, 0x2a653400
Caught exception in looking up space point ptr... 0x2a638b00, 0x2a645f80
Caught exception in looking up space point ptr... 0x2ae125a0, 0x50feb940
Caught exception in looking up space point ptr... 0x50feb940, 0x53d617e0
Caught exception in looking up space point ptr... 0x46448e40, 0x2a4674a0
Caught exception in looking up space point ptr... 0x2a4674a0, 0x2a454ee0
Caught exception in looking up space point ptr... 0x863a2a0, 0x415aa910
Caught exception in looking up space point ptr... 0x415aa910, 0x60688df0
Caught exception in looking up space point ptr... 0x5359e900, 0x5358c7f0
Caught exception in looking up space point ptr... 0x535ab8d0, 0x5359e900
Caught exception in looking up space point ptr... 0x607fee80, 0x607f16e0
Caught exception in looking up space point ptr... 0x607f16e0, 0x607fecf0
Caught exception in looking up space point ptr... 0x607fd0d0, 0x607ef930
Caught exception in looking up space point ptr... 0x607ef930, 0x607fcf40
Caught exception in looking up space point ptr... 0x2a61d0d0, 0x2a60f7a0
Caught exception in looking up space point ptr... 0x2a60f7a0, 0x5ebce170
Caught exception in looking up space point ptr... 0x5ee98b90, 0x5ee8b3f0
Caught exception in looking up space point ptr... 0x5ee8b3f0, 0x60809a60
Caught exception in looking up space point ptr... 0xacb55c0, 0x60db6600
Caught exception in looking up space point ptr... 0x60db6600, 0xacb5430
Caught exception in looking up space point ptr... 0x53004b20, 0x32e08330
Caught exception in looking up space point ptr... 0x32e08330, 0x53004990
Caught exception in looking up space point ptr... 0x28e25420, 0x425b2870
Caught exception in looking up space point ptr... 0x4fc8d300, 0x28e25420
Caught exception in looking up space point ptr... 0x5ec84b90, 0x4fa309b0
Caught exception in looking up space point ptr... 0x4fa309b0, 0x5ec84b90
Caught exception in looking up space point ptr... 0x469ec160, 0x469fa210
Caught exception in looking up space point ptr... 0x469fa210, 0x469ec160
Caught exception in looking up space point ptr... 0x5cbbc590, 0x5cbbc290
Caught exception in looking up space point ptr... 0x5cbbc290, 0x5cbd3300
Caught exception in looking up space point ptr... 0x2531c040, 0x2531c360
Caught exception in looking up space point ptr... 0x2531c360, 0x2531c040
Caught exception in looking up space point ptr... 0x4ff147f0, 0x5c917e50
Caught exception in looking up space point ptr... 0x5c917e50, 0x5c912ae0
Caught exception in looking up space point ptr... 0x5c912ae0, 0x2c92ea20
Caught exception in looking up space point ptr... 0x2c92ea20, 0x415fa050
Caught exception in looking up space point ptr... 0x3b761360, 0x3b764500
Caught exception in looking up space point ptr... 0x3b764500, 0x3b766b10
Caught exception in looking up space point ptr... 0x3fc46f50, 0x3fc46cf0
Caught exception in looking up space point ptr... 0x3fc46cf0, 0x3fc46f50
Caught exception in looking up space point ptr... 0x5a4a68b0, 0x5407bf20
Caught exception in looking up space point ptr... 0x5407bb90, 0x5a4a68b0
Caught exception in looking up space point ptr... 0x46a03f20, 0x2a368600
Caught exception in looking up space point ptr... 0x2a368600, 0x2a3684d0
Caught exception in looking up space point ptr... 0x46a03b80, 0x2a367c80
Caught exception in looking up space point ptr... 0x2a367c80, 0x2a367a20
Caught exception in looking up space point ptr... 0x3b6b4940, 0x3b6b4ad0
Caught exception in looking up space point ptr... 0x3b6b4ad0, 0x3b6b4940
Caught exception in looking up space point ptr... 0x53d3b420, 0x53d3cb70
Caught exception in looking up space point ptr... 0x53d3cb70, 0x53d3b420
Caught exception in looking up space point ptr... 0x253ca690, 0x2117c6c0
Caught exception in looking up space point ptr... 0x2117c6c0, 0x45ecc2b0
Caught exception in looking up space point ptr... 0x2a357040, 0x2a356d20
Caught exception in looking up space point ptr... 0x2a356d20, 0x2a356eb0
Caught exception in looking up space point ptr... 0x531141a0, 0x53114330
Caught exception in looking up space point ptr... 0x53114330, 0x531141a0
Caught exception in looking up space point ptr... 0x2aa99a80, 0x50a12ac0
Caught exception in looking up space point ptr... 0x50a12ac0, 0x2aa99a80
Caught exception in looking up space point ptr... 0x2a362e30, 0x2a36df80
Caught exception in looking up space point ptr... 0x2a36df80, 0x2a362e30
++++>>>> total num hits: 381255, num free: 373943
04-Feb-2026 02:53:43 GMT  Closed output file "np02vd_raw_run042421_0098_df-s02-d1_dw_0_20260203T142400_reco_stage0_20260204T025343_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         306.088       412.837       540.967       411.29        49.3647        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.8456e-05    0.000119606   0.000292791   0.000108655   3.89176e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                143.643       192.172       281.486       184.757       36.6546        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.705825       1.03084       2.03436      0.873264      0.346252        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0221899     0.0273676     0.0445423     0.0230845    0.00695683       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108888      0.118631      0.161219      0.111679      0.0143443       27     
produce:beamevent:BeamEvent                      9.2414e-05    0.000167067   0.00118797    0.000124705   0.000201569      27     
produce:pdvddaphne:DAPHNEReaderPDVD                17.8523       23.8702       37.5012       23.3345       4.12654        27     
produce:ophit:OpHitFinder                         0.040051      0.0536208     0.0673601     0.0542607    0.00608993       27     
produce:wclsdatavd:WireCellToolkit                 103.156       119.594       129.66        119.349       6.84015        27     
produce:gaushit:GausHitFinder                     0.746805       2.10748       2.97986       2.07123      0.520367        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000252135   0.000459973    0.0007813    0.000442663   0.000113574      27     
produce:reco3d:SpacePointSolver                    6.14084       13.0144       28.7071       12.8498       4.63173        27     
produce:hitpdune:DisambigFromSpacePoints          0.0993657     0.327708      0.667641      0.335148      0.144203        27     
produce:cluster3d:Cluster3D                        1.19758       52.9346       83.0797       72.1695       32.4705        27     
[art]:TriggerResults:TriggerResultInserter       2.8073e-05    4.07017e-05   0.000110387   3.4525e-05    1.66939e-05      27     
end_path:out1:RootOutput                          4.278e-06     7.831e-06    3.3623e-05     5.751e-06    6.98265e-06      27     
end_path:out1:RootOutput(write)                    5.32775       7.54724       9.11501       7.60439       1.08205        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4704.96 MB
  Peak resident set size usage (VmHWM): 2685.53 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0098_df-s02-d1_dw_0_20260203T142400_reco_stage0_20260204T025343_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0098_df-s02-d1_dw_0_20260203T142400_reco_stage0_20260204T025343_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0098_df-s02-d1_dw_0_20260203T142400_reco_stage0_20260204T025343_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162378.0,
    "core.end_time": 1770173625.0,
    "core.events": [
      15937,
      15943,
      15949,
      15955,
      15961,
      15967,
      15973,
      15979,
      15985,
      15991,
      15997,
      16003,
      16009,
      16015,
      16021,
      16027,
      16033,
      16039,
      16045,
      16051,
      16057,
      16063,
      16069,
      16075,
      16081,
      16087,
      16093
    ],
    "core.event_count": 27,
    "core.first_event_number": 15937,
    "core.last_event_number": 16093,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0098_df-s02-d1_dw_0_20260203T142400.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:18:38 UTC       justIN version: 01.06.00