justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303654.88@dunegpschedd01.fnal.gov

Jobsub ID303654.88@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:26
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2026-02-04 03:02:10
From worker nodeHostnamewn2209251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:05:53
Input filesvd-protodune:np02vd_raw_run042433_0118_df-s04-d0_dw_0_20260203T192659.hdf5
JobscriptExit code0
Real time2h (10375s)
CPU time1h (5755s = 55%)
Max RSS bytes12154433536 (11591 MiB)
Outputting started2026-02-04 02:58:54
Output files
Finished2026-02-04 03:02:10
Saved logsjustin-logs:303654.88-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

eption in looking up space point ptr... 0x32f2b6a0, 0x32f2b330
Caught exception in looking up space point ptr... 0x32f2b330, 0x32f2b6a0
Caught exception in looking up space point ptr... 0x38c443b0, 0xf89cf140
Caught exception in looking up space point ptr... 0xf89cf140, 0x38c443b0
Caught exception in looking up space point ptr... 0x159f1d7f0, 0x396888d0
Caught exception in looking up space point ptr... 0x396888d0, 0x159f1d7f0
Caught exception in looking up space point ptr... 0x9400ea60, 0x9400ec40
Caught exception in looking up space point ptr... 0x9400ec40, 0x9400ea60
Caught exception in looking up space point ptr... 0xbf752a40, 0xbf7528f0
Caught exception in looking up space point ptr... 0xbf7528f0, 0xbf752a40
Caught exception in looking up space point ptr... 0xbf752a40, 0xf54eb700
Caught exception in looking up space point ptr... 0xf54eb700, 0xbf752a40
Caught exception in looking up space point ptr... 0xde9dbb10, 0x940ac980
Caught exception in looking up space point ptr... 0x940ac980, 0xde9dbb10
Caught exception in looking up space point ptr... 0x1199bdec0, 0x1199be300
Caught exception in looking up space point ptr... 0x1199be300, 0x1199be520
Caught exception in looking up space point ptr... 0x38dd11a0, 0xdb58c4b0
Caught exception in looking up space point ptr... 0xdb58c4b0, 0x38dd13c0
Caught exception in looking up space point ptr... 0xd28c4560, 0xd28c4280
Caught exception in looking up space point ptr... 0xd28c4280, 0xd28c4560
Caught exception in looking up space point ptr... 0xca061d80, 0xcb9a7fb0
Caught exception in looking up space point ptr... 0xcb9a7fb0, 0xca061d80
Caught exception in looking up space point ptr... 0xc42332f0, 0x14b960ad0
Caught exception in looking up space point ptr... 0x14b960ad0, 0x124cd8140
Caught exception in looking up space point ptr... 0x124cd8140, 0x112d0ec10
Caught exception in looking up space point ptr... 0x112d0ec10, 0xc42332f0
Caught exception in looking up space point ptr... 0xf46034d0, 0xd1f494a0
Caught exception in looking up space point ptr... 0xd1f494a0, 0x13ccbcd00
Caught exception in looking up space point ptr... 0xe555a9c0, 0xe555a310
Caught exception in looking up space point ptr... 0xe555a310, 0xe555c190
Caught exception in looking up space point ptr... 0x39e70510, 0x178b6e6a0
Caught exception in looking up space point ptr... 0x178b6e6a0, 0x94c212c0
Caught exception in looking up space point ptr... 0xd329b470, 0xd329b6d0
Caught exception in looking up space point ptr... 0xd329b6d0, 0xd329b470
Caught exception in looking up space point ptr... 0xe448cc70, 0xe448cae0
Caught exception in looking up space point ptr... 0xe448cae0, 0xe448cc70
Caught exception in looking up space point ptr... 0xc79e2ee0, 0x12e4dc0d0
Caught exception in looking up space point ptr... 0x12e4dc0d0, 0xc79e2ee0
Caught exception in looking up space point ptr... 0xcb9a7dd0, 0xc8c39940
Caught exception in looking up space point ptr... 0xc8c39940, 0xcb9a7dd0
Caught exception in looking up space point ptr... 0xf6afc1f0, 0xf6afc3d0
Caught exception in looking up space point ptr... 0xf6afc3d0, 0xf6afc560
Caught exception in looking up space point ptr... 0x178a98a80, 0x11acc2c40
Caught exception in looking up space point ptr... 0x11acc2c40, 0x178a98a80
Caught exception in looking up space point ptr... 0x355a0f80, 0x355a0c10
Caught exception in looking up space point ptr... 0x355a0c10, 0x355a0f80
Caught exception in looking up space point ptr... 0xfba43000, 0xfba42ed0
Caught exception in looking up space point ptr... 0xfba42ed0, 0xfba42c20
Caught exception in looking up space point ptr... 0x940402a0, 0xde158730
Caught exception in looking up space point ptr... 0xde158730, 0x947d8c00
Caught exception in looking up space point ptr... 0x1291c8240, 0x1291c83d0
Caught exception in looking up space point ptr... 0x1291c83d0, 0x38f71120
Caught exception in looking up space point ptr... 0x4cca2260, 0x156e5dc00
Caught exception in looking up space point ptr... 0x156e5dc00, 0x156d45440
Caught exception in looking up space point ptr... 0x1422eee30, 0x1422eeb30
Caught exception in looking up space point ptr... 0x1422eeb30, 0x1422eee30
Caught exception in looking up space point ptr... 0x1289af9b0, 0xc033410
Caught exception in looking up space point ptr... 0xc033410, 0x1289af9b0
Caught exception in looking up space point ptr... 0xbe8052d0, 0xbe802ba0
Caught exception in looking up space point ptr... 0xbe802ba0, 0xbe8052d0
Caught exception in looking up space point ptr... 0xfdcc3eb0, 0xfdcc44f0
Caught exception in looking up space point ptr... 0xfdcc44f0, 0xfdcc3eb0
Caught exception in looking up space point ptr... 0x14f476620, 0x14f476a10
Caught exception in looking up space point ptr... 0x14f476a10, 0x14f476770
Caught exception in looking up space point ptr... 0xcbc0c860, 0xcbc0b8a0
Caught exception in looking up space point ptr... 0xcbc0b8a0, 0xcbc0c860
++++>>>> total num hits: 88171, num free: 84139
04-Feb-2026 02:58:38 GMT  Closed output file "np02vd_raw_run042433_0118_df-s04-d0_dw_0_20260203T192659_reco_stage0_20260204T025838_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         392.046       540.554       1729.41       436.662       305.564        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000105018   0.00016513    0.00040364    0.000137199   7.33246e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                189.467       226.262       306.169       221.453       29.8966        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.741794       1.37641       5.75755       1.10616       1.06079        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.022478      0.0296124     0.0532116     0.0281281    0.00764023       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0222378     0.0226044     0.0231889     0.0226946    0.000280279      19     
produce:beamevent:BeamEvent                      0.000141476   0.000325067   0.00184966    0.000236265   0.000363985      19     
produce:pdvddaphne:DAPHNEReaderPDVD                17.6772       23.9112       27.8076       24.244        2.61406        19     
produce:ophit:OpHitFinder                         0.0377882     0.0459809     0.0515934     0.0460608      0.00311        19     
produce:wclsdatavd:WireCellToolkit                 130.943       148.487       164.305       149.208       8.78375        19     
produce:gaushit:GausHitFinder                      1.30906       2.9816        13.9599       1.8629        3.02393        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000298342   0.000571579   0.00157401    0.000474252   0.000301806      19     
produce:reco3d:SpacePointSolver                    9.84484       30.4066       162.633       19.6045       33.6363        19     
produce:hitpdune:DisambigFromSpacePoints          0.117046       1.35258       10.1386      0.266608       2.8007         19     
produce:cluster3d:Cluster3D                        1.5745        96.0844       1169.39       3.69926       269.03         19     
[art]:TriggerResults:TriggerResultInserter       3.0137e-05    5.96487e-05   0.000134774   4.5967e-05    2.96977e-05      19     
end_path:out1:RootOutput                          3.927e-06    1.33071e-05   4.4564e-05     8.095e-06    1.1734e-05       19     
end_path:out1:RootOutput(write)                    5.2437        9.45105       30.4928       7.39385       5.96431        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14216.6 MB
  Peak resident set size usage (VmHWM): 12154.4 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0118_df-s04-d0_dw_0_20260203T192659_reco_stage0_20260204T025838_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0118_df-s04-d0_dw_0_20260203T192659_reco_stage0_20260204T025838_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0118_df-s04-d0_dw_0_20260203T192659_reco_stage0_20260204T025838_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163565.0,
    "core.end_time": 1770173921.0,
    "core.events": [
      13462,
      13468,
      13474,
      13480,
      13486,
      13492,
      13498,
      13504,
      13510,
      13516,
      13522,
      13528,
      13534,
      13540,
      13546,
      13552,
      13558,
      13564,
      13570
    ],
    "core.event_count": 19,
    "core.first_event_number": 13462,
    "core.last_event_number": 13570,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0118_df-s04-d0_dw_0_20260203T192659.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:27 UTC       justIN version: 01.06.00