justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297968.2@dunegpschedd02.fnal.gov

Jobsub ID297968.2@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 00:36:28
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 02:58:53
From worker nodeHostnamewn2205251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:37:34
Input filesvd-protodune:np02vd_raw_run042421_0130_df-s02-d0_dw_0_20260203T144117.hdf5
JobscriptExit code0
Real time2h (8309s)
CPU time49m (2984s = 35%)
Max RSS bytes2592595968 (2472 MiB)
Outputting started2026-02-04 02:56:05
Output files
Finished2026-02-04 02:58:53
Saved logsjustin-logs:297968.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0xdbef360
Caught exception in looking up space point ptr... 0xdbef040, 0xdbeef20
Caught exception in looking up space point ptr... 0x43a91bf0, 0x151bd060
Caught exception in looking up space point ptr... 0x151bd060, 0x13cc9d30
Caught exception in looking up space point ptr... 0x15153750, 0x3bfaf180
Caught exception in looking up space point ptr... 0x3bfaf180, 0x15153750
Caught exception in looking up space point ptr... 0x2c0f6c20, 0x10aefad0
Caught exception in looking up space point ptr... 0x3a3f8d30, 0x44eb44c0
Caught exception in looking up space point ptr... 0x44eb44c0, 0x44eb4be0
Caught exception in looking up space point ptr... 0x44eb4be0, 0x10aedc10
Caught exception in looking up space point ptr... 0x10aedc10, 0x456d6810
Caught exception in looking up space point ptr... 0x456d6810, 0x2c0f6c20
Caught exception in looking up space point ptr... 0x4fb67a60, 0x4fb68370
Caught exception in looking up space point ptr... 0x4fb68370, 0x4fb67a60
Caught exception in looking up space point ptr... 0x6449ad90, 0x11099ac0
Caught exception in looking up space point ptr... 0x11099ac0, 0x13853870
Caught exception in looking up space point ptr... 0x5cfbc070, 0x5cfbc390
Caught exception in looking up space point ptr... 0x5cfbc390, 0x5cfbc200
Caught exception in looking up space point ptr... 0x5673b0b0, 0x3548ab10
Caught exception in looking up space point ptr... 0x3548ab10, 0x5673b0b0
Caught exception in looking up space point ptr... 0x2f543aa0, 0x92632d0
Caught exception in looking up space point ptr... 0x92632d0, 0x2f543aa0
Caught exception in looking up space point ptr... 0x151ff650, 0x151ff550
Caught exception in looking up space point ptr... 0x25313fe0, 0x151ff650
Caught exception in looking up space point ptr... 0x3e1edd00, 0x3e1edb00
Caught exception in looking up space point ptr... 0x3e1edb00, 0x3e1edc00
Caught exception in looking up space point ptr... 0x416e58e0, 0x267903f0
Caught exception in looking up space point ptr... 0x267903f0, 0x416e58e0
Caught exception in looking up space point ptr... 0x2ce51ae0, 0xaea9bb0
Caught exception in looking up space point ptr... 0xaea9bb0, 0x155e6420
Caught exception in looking up space point ptr... 0x456d2cf0, 0x456d2e80
Caught exception in looking up space point ptr... 0x456d2e80, 0x456d31a0
Caught exception in looking up space point ptr... 0x2c860b70, 0x2c860a20
Caught exception in looking up space point ptr... 0x2c860a20, 0x2c860b70
Caught exception in looking up space point ptr... 0x2c860b70, 0x2c8608d0
Caught exception in looking up space point ptr... 0x2c8608d0, 0x2c860b70
Caught exception in looking up space point ptr... 0x10af52f0, 0x5ef42bf0
Caught exception in looking up space point ptr... 0x5ef42bf0, 0x10af52f0
Caught exception in looking up space point ptr... 0x4d73f480, 0x4d73f580
Caught exception in looking up space point ptr... 0x4d73f580, 0x15def130
Caught exception in looking up space point ptr... 0x2f5439b0, 0x313dda20
Caught exception in looking up space point ptr... 0x313dda20, 0x15624f30
Caught exception in looking up space point ptr... 0x3340b550, 0x3340b0a0
Caught exception in looking up space point ptr... 0x3340b0a0, 0x3340b550
Caught exception in looking up space point ptr... 0xdbf65b0, 0xea44b90
Caught exception in looking up space point ptr... 0xdbf3a00, 0xdbf65b0
Caught exception in looking up space point ptr... 0x3915e5b0, 0x3915e740
Caught exception in looking up space point ptr... 0x3915e740, 0x3915e5b0
Caught exception in looking up space point ptr... 0x3915e5b0, 0xf9dd930
Caught exception in looking up space point ptr... 0xf9dd930, 0x3915e5b0
Caught exception in looking up space point ptr... 0x416f6cd0, 0x30d17bd0
Caught exception in looking up space point ptr... 0x30d17bd0, 0x5acff670
Caught exception in looking up space point ptr... 0x4147c480, 0x3325b0b0
Caught exception in looking up space point ptr... 0x3325b0b0, 0x4147c480
Caught exception in looking up space point ptr... 0x118f1b20, 0x118f19c0
Caught exception in looking up space point ptr... 0x118f19c0, 0x118f1b20
Caught exception in looking up space point ptr... 0x118f1b20, 0x29d71640
Caught exception in looking up space point ptr... 0x29d71640, 0x118f1b20
Caught exception in looking up space point ptr... 0x8ae4980, 0x8f00180
Caught exception in looking up space point ptr... 0x8f00180, 0x97f5c80
Caught exception in looking up space point ptr... 0x2c4d5390, 0x2c4d58b0
Caught exception in looking up space point ptr... 0x2c4d58b0, 0x150bbe10
Caught exception in looking up space point ptr... 0x12cdb2c0, 0x103fab00
Caught exception in looking up space point ptr... 0x103fab00, 0x12cdb2c0
Caught exception in looking up space point ptr... 0x2c767490, 0x3d94ddd0
Caught exception in looking up space point ptr... 0x3d94ddd0, 0x41260930
++++>>>> total num hits: 56675, num free: 54498
04-Feb-2026 02:55:54 GMT  Closed output file "np02vd_raw_run042421_0130_df-s02-d0_dw_0_20260203T144117_reco_stage0_20260204T025554_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         244.116       303.923       363.252       297.525       32.662         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.1264e-05    0.000139759   0.000266492   0.000107633   5.67672e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                129.896       178.192       236.016       176.363       28.3032        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.808446       1.10273       1.66528       1.06096       0.18475        27     
produce:timingrawdecoder:PDHDTimingRawDecoder       0.022       0.0386612      0.10577      0.0359799     0.017544        27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.108559      0.127304       0.19485      0.112973      0.0237795       27     
produce:beamevent:BeamEvent                      8.6553e-05    0.000195254   0.00122013    0.000166053   0.000203717      27     
produce:pdvddaphne:DAPHNEReaderPDVD                13.7559       21.3278       30.2023       20.5498       3.73283        27     
produce:ophit:OpHitFinder                         0.0330303     0.0448934     0.0569422     0.0457154    0.00585003       27     
produce:wclsdatavd:WireCellToolkit                 71.5667       82.0941       88.9788       82.8305       5.14845        27     
produce:gaushit:GausHitFinder                     0.584335       1.22372       1.99294       1.24884      0.358686        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000161915   0.000310527   0.000531762   0.000311817   8.99473e-05      27     
produce:reco3d:SpacePointSolver                    4.91776       11.9134       22.3321       11.5176       4.16776        27     
produce:hitpdune:DisambigFromSpacePoints          0.0614302     0.128576      0.226312      0.127227      0.0463365       27     
produce:cluster3d:Cluster3D                       0.881118       2.83582       6.63344       2.50101       1.29986        27     
[art]:TriggerResults:TriggerResultInserter       2.0008e-05    3.6586e-05     0.0001063    3.6379e-05    1.61264e-05      27     
end_path:out1:RootOutput                          3.516e-06    1.13851e-05   5.9753e-05     8.886e-06    1.12346e-05      27     
end_path:out1:RootOutput(write)                    3.79464       4.85723       6.01569       4.78552      0.609213        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4546.58 MB
  Peak resident set size usage (VmHWM): 2592.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0130_df-s02-d0_dw_0_20260203T144117_reco_stage0_20260204T025554_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0130_df-s02-d0_dw_0_20260203T144117_reco_stage0_20260204T025554_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0130_df-s02-d0_dw_0_20260203T144117_reco_stage0_20260204T025554_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770165466.0,
    "core.end_time": 1770173756.0,
    "core.events": [
      21126,
      21132,
      21138,
      21144,
      21150,
      21156,
      21162,
      21168,
      21174,
      21180,
      21186,
      21192,
      21198,
      21204,
      21210,
      21216,
      21222,
      21228,
      21234,
      21240,
      21246,
      21252,
      21258,
      21264,
      21270,
      21276,
      21282
    ],
    "core.event_count": 27,
    "core.first_event_number": 21126,
    "core.last_event_number": 21282,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0130_df-s02-d0_dw_0_20260203T144117.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:43:57 UTC       justIN version: 01.06.00