justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.168@dunegpschedd02.fnal.gov

Jobsub ID297921.168@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-02-04 02:34:02
From worker nodeHostnamecn006.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:33
Input filesvd-protodune:np02vd_raw_run042433_0155_df-s03-d1_dw_0_20260203T194101.hdf5
JobscriptExit code0
Real time2h (9167s)
CPU time1h (5128s = 55%)
Max RSS bytes3578384384 (3412 MiB)
Outputting started2026-02-04 02:29:57
Output files
Finished2026-02-04 02:34:02
Saved logsjustin-logs:297921.168-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tr... 0xadaf9a0, 0x380df8b0
Caught exception in looking up space point ptr... 0x6eef98a0, 0x5caf4950
Caught exception in looking up space point ptr... 0x5caf5c40, 0x6eef98a0
Caught exception in looking up space point ptr... 0x310ed2b0, 0x564a7610
Caught exception in looking up space point ptr... 0x564a7610, 0x310ed2b0
Caught exception in looking up space point ptr... 0x5b4d3e50, 0x5b4d4930
Caught exception in looking up space point ptr... 0x5b4d4930, 0x5b4d3e50
Caught exception in looking up space point ptr... 0x3b3a2070, 0x3b3a1ee0
Caught exception in looking up space point ptr... 0x3b3a1ee0, 0x3b3a1d50
Caught exception in looking up space point ptr... 0x5b00eae0, 0x71883950
Caught exception in looking up space point ptr... 0x71883950, 0x5b00eae0
Caught exception in looking up space point ptr... 0x6be98a20, 0x6beaafe0
Caught exception in looking up space point ptr... 0x6be862d0, 0x6be98a20
Caught exception in looking up space point ptr... 0x8968b230, 0xdb01c30
Caught exception in looking up space point ptr... 0x8968b3c0, 0x8968b230
Caught exception in looking up space point ptr... 0x11aad0e0, 0x11aacaa0
Caught exception in looking up space point ptr... 0x11aacaa0, 0x11aac2d0
Caught exception in looking up space point ptr... 0x12a9f190, 0x12a9ece0
Caught exception in looking up space point ptr... 0x12a9f4b0, 0x12a9f190
Caught exception in looking up space point ptr... 0x197d1610, 0x197d1480
Caught exception in looking up space point ptr... 0x197d1480, 0x78698750
Caught exception in looking up space point ptr... 0x4ea616e0, 0x850b60f0
Caught exception in looking up space point ptr... 0x4ea61870, 0x4ea616e0
Caught exception in looking up space point ptr... 0x4837f100, 0x197a4650
Caught exception in looking up space point ptr... 0x197a4650, 0x197a47e0
Caught exception in looking up space point ptr... 0x651114d0, 0x65122c80
Caught exception in looking up space point ptr... 0x65122af0, 0x651114d0
Caught exception in looking up space point ptr... 0x58acd590, 0x58b0a420
Caught exception in looking up space point ptr... 0x58b0a420, 0x58b0a570
Caught exception in looking up space point ptr... 0x55d784e0, 0x2db71f90
Caught exception in looking up space point ptr... 0x2db71d40, 0x55d784e0
Caught exception in looking up space point ptr... 0x17db7110, 0x32193a10
Caught exception in looking up space point ptr... 0x3a623d90, 0x17db7110
Caught exception in looking up space point ptr... 0x82f3c8f0, 0x82f2a7e0
Caught exception in looking up space point ptr... 0x82f4be40, 0x82f3c8f0
Caught exception in looking up space point ptr... 0x76436ae0, 0x150ba030
Caught exception in looking up space point ptr... 0x150ba030, 0x5f630280
Caught exception in looking up space point ptr... 0x79f42770, 0x79f54ba0
Caught exception in looking up space point ptr... 0x79f54ba0, 0x79f64d70
Caught exception in looking up space point ptr... 0x344ecde0, 0x344fed60
Caught exception in looking up space point ptr... 0x344feef0, 0x344ecde0
Caught exception in looking up space point ptr... 0x37ea30c0, 0x37ea31c0
Caught exception in looking up space point ptr... 0x37ea31c0, 0x37ea32c0
Caught exception in looking up space point ptr... 0x3d739230, 0x40c04240
Caught exception in looking up space point ptr... 0x40c04240, 0x3d739230
Caught exception in looking up space point ptr... 0x615a2fa0, 0x615a3200
Caught exception in looking up space point ptr... 0x615a30d0, 0x615a2fa0
Caught exception in looking up space point ptr... 0x125f4a50, 0x125f47b0
Caught exception in looking up space point ptr... 0x125f47b0, 0x125f4ba0
Caught exception in looking up space point ptr... 0x50b056c0, 0x50bb7690
Caught exception in looking up space point ptr... 0x50bb7690, 0x50b056c0
Caught exception in looking up space point ptr... 0x3f937740, 0x2eae5d80
Caught exception in looking up space point ptr... 0x2eae5d80, 0x3f937740
Caught exception in looking up space point ptr... 0x8f8b040, 0xaeb4f10
Caught exception in looking up space point ptr... 0xaeb4f10, 0x4c0a4fe0
Caught exception in looking up space point ptr... 0x63c4bb00, 0x5b571240
Caught exception in looking up space point ptr... 0x5b571240, 0x63c4bb00
Caught exception in looking up space point ptr... 0x5a781de0, 0x1b1aa3a0
Caught exception in looking up space point ptr... 0x1b1aa3a0, 0x5a781de0
Caught exception in looking up space point ptr... 0x717f1410, 0x43bb8120
Caught exception in looking up space point ptr... 0x43bb8120, 0x717f1410
Caught exception in looking up space point ptr... 0x505ed1a0, 0x505edc40
Caught exception in looking up space point ptr... 0x505edc40, 0x505ed1a0
Caught exception in looking up space point ptr... 0x6f00e0b0, 0x4c0dfa40
Caught exception in looking up space point ptr... 0x4c0dfa40, 0x6f00e0b0
Caught exception in looking up space point ptr... 0xacba330, 0x3cf5fb00
Caught exception in looking up space point ptr... 0x3cf5fb00, 0x4902ed30
++++>>>> total num hits: 555003, num free: 546138
04-Feb-2026 02:29:09 GMT  Closed output file "np02vd_raw_run042433_0155_df-s03-d1_dw_0_20260203T194101_reco_stage0_20260204T022909_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         382.536       477.381       572.422       468.006       53.6407        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.0053e-05    0.000135792   0.000332956   0.000116216   6.22965e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                143.689       202.724       269.54        199.664       32.9631        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.717017      0.950145       1.4028       0.877787      0.209997        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165704     0.0283892     0.101074      0.0215772     0.018705        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.016549      0.0185185     0.0329057     0.0167757    0.00491786       19     
produce:beamevent:BeamEvent                      0.000124625   0.000225042   0.00135425    0.000159644   0.000267472      19     
produce:pdvddaphne:DAPHNEReaderPDVD                16.8561       20.5065       26.9349       20.6013       2.83175        19     
produce:ophit:OpHitFinder                         0.0346203     0.058964      0.0730349     0.0641562     0.0132598       19     
produce:wclsdatavd:WireCellToolkit                 134.339       164.267       200.802       164.341       19.1664        19     
produce:gaushit:GausHitFinder                      1.39929       2.73185       4.09934       2.58952      0.730469        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000548458   0.000724875   0.00104204    0.00068879    0.000127488      19     
produce:reco3d:SpacePointSolver                    8.21898       15.3979       26.4619       14.2719       4.33759        19     
produce:hitpdune:DisambigFromSpacePoints          0.224579      0.438489      0.775379      0.378302      0.166957        19     
produce:cluster3d:Cluster3D                        2.07414       62.6306       101.688       68.368        28.7064        19     
[art]:TriggerResults:TriggerResultInserter       2.9813e-05    4.48993e-05   8.9516e-05    4.1292e-05    1.45042e-05      19     
end_path:out1:RootOutput                          5.32e-06     2.54083e-05   0.000307299    8.378e-06    6.66146e-05      19     
end_path:out1:RootOutput(write)                    4.91675       7.57713       9.96687       7.1971        1.63097        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5495.96 MB
  Peak resident set size usage (VmHWM): 3578.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0155_df-s03-d1_dw_0_20260203T194101_reco_stage0_20260204T022909_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0155_df-s03-d1_dw_0_20260203T194101_reco_stage0_20260204T022909_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0155_df-s03-d1_dw_0_20260203T194101_reco_stage0_20260204T022909_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163003.0,
    "core.end_time": 1770172154.0,
    "core.events": [
      17679,
      17685,
      17691,
      17697,
      17703,
      17709,
      17715,
      17721,
      17727,
      17733,
      17739,
      17745,
      17751,
      17757,
      17763,
      17769,
      17775,
      17781,
      17787
    ],
    "core.event_count": 19,
    "core.first_event_number": 17679,
    "core.last_event_number": 17787,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0155_df-s03-d1_dw_0_20260203T194101.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:08:53 UTC       justIN version: 01.06.00