justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.36@dunegpschedd02.fnal.gov

Jobsub ID297923.36@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-02-04 02:17:23
From worker nodeHostnamecn540.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:29
Input filesvd-protodune:np02vd_raw_run042433_0145_df-s04-d0_dw_0_20260203T193714.hdf5
JobscriptExit code0
Real time2h (8121s)
CPU time1h (4548s = 56%)
Max RSS bytes3386507264 (3229 MiB)
Outputting started2026-02-04 02:14:53
Output files
Finished2026-02-04 02:17:23
Saved logsjustin-logs:297923.36-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

king up space point ptr... 0x361a5710, 0xa623640
Caught exception in looking up space point ptr... 0xa623640, 0x366b00b0
Caught exception in looking up space point ptr... 0x366b00b0, 0xa623640
Caught exception in looking up space point ptr... 0x58885180, 0x189c8c60
Caught exception in looking up space point ptr... 0x189c8c60, 0x58885180
Caught exception in looking up space point ptr... 0x58885180, 0x189c8a00
Caught exception in looking up space point ptr... 0x189c8a00, 0x58885180
Caught exception in looking up space point ptr... 0x58885180, 0x58885050
Caught exception in looking up space point ptr... 0x58885050, 0x58885180
Caught exception in looking up space point ptr... 0xea043d0, 0xea04240
Caught exception in looking up space point ptr... 0xea04240, 0xea043d0
Caught exception in looking up space point ptr... 0x65a59ad0, 0x7137a730
Caught exception in looking up space point ptr... 0x7137a730, 0x65a59ad0
Caught exception in looking up space point ptr... 0x497b0560, 0xd7fbe30
Caught exception in looking up space point ptr... 0xd7fbe30, 0x497b0560
Caught exception in looking up space point ptr... 0x497b0560, 0xd7fbb10
Caught exception in looking up space point ptr... 0xd7fbb10, 0x497b0560
Caught exception in looking up space point ptr... 0x497b0560, 0x497b03d0
Caught exception in looking up space point ptr... 0x497b03d0, 0x497b0560
Caught exception in looking up space point ptr... 0x1308a710, 0x1308a4b0
Caught exception in looking up space point ptr... 0x1305f080, 0x57d49c80
Caught exception in looking up space point ptr... 0x57d49c80, 0x57dcac30
Caught exception in looking up space point ptr... 0x57dcac30, 0x1308a710
Caught exception in looking up space point ptr... 0x74c7cb90, 0x74c7c870
Caught exception in looking up space point ptr... 0x74c7c870, 0x74c7cb90
Caught exception in looking up space point ptr... 0x74c7cb90, 0x74c7c6e0
Caught exception in looking up space point ptr... 0x74c7c6e0, 0x74c7cb90
Caught exception in looking up space point ptr... 0x5958e570, 0x5956fe90
Caught exception in looking up space point ptr... 0x5956fe90, 0x7bf3ab40
Caught exception in looking up space point ptr... 0x585839c0, 0x9020500
Caught exception in looking up space point ptr... 0x9020500, 0x585839c0
Caught exception in looking up space point ptr... 0x6f75c210, 0x6d03ecd0
Caught exception in looking up space point ptr... 0x6d03ecd0, 0x821c3590
Caught exception in looking up space point ptr... 0x72f16300, 0x72f161b0
Caught exception in looking up space point ptr... 0x72f161b0, 0x72f16300
Caught exception in looking up space point ptr... 0x69cd7580, 0x15e59fb0
Caught exception in looking up space point ptr... 0x15e59fb0, 0x67293460
Caught exception in looking up space point ptr... 0x174cad80, 0x59e334e0
Caught exception in looking up space point ptr... 0x405e0d80, 0x174cad80
Caught exception in looking up space point ptr... 0x65b59550, 0x61110090
Caught exception in looking up space point ptr... 0x61110090, 0x65b59870
Caught exception in looking up space point ptr... 0x921f2e0, 0x921f050
Caught exception in looking up space point ptr... 0x921f050, 0x48857000
Caught exception in looking up space point ptr... 0x69c10600, 0x4111f560
Caught exception in looking up space point ptr... 0x4111f560, 0x69c7caa0
Caught exception in looking up space point ptr... 0x5ef0c580, 0x5ef0c7a0
Caught exception in looking up space point ptr... 0x672972e0, 0x5ef0c580
Caught exception in looking up space point ptr... 0xea027b0, 0xea02620
Caught exception in looking up space point ptr... 0xea02620, 0xea027b0
Caught exception in looking up space point ptr... 0x59e2d060, 0x409838b0
Caught exception in looking up space point ptr... 0x40983b40, 0x59e2d060
Caught exception in looking up space point ptr... 0x385419b0, 0x13067220
Caught exception in looking up space point ptr... 0x13067220, 0x48510130
Caught exception in looking up space point ptr... 0x15e61000, 0x14025f00
Caught exception in looking up space point ptr... 0x14025f00, 0x67283d40
Caught exception in looking up space point ptr... 0x901db00, 0x901dc00
Caught exception in looking up space point ptr... 0x16606af0, 0x901db00
Caught exception in looking up space point ptr... 0xf1a9580, 0x5dd46eb0
Caught exception in looking up space point ptr... 0x5dd46eb0, 0xf1a9260
Caught exception in looking up space point ptr... 0xf1a9260, 0x5dd46eb0
Caught exception in looking up space point ptr... 0x5dd46eb0, 0xf1a93f0
Caught exception in looking up space point ptr... 0x57df7250, 0x5ae5c680
Caught exception in looking up space point ptr... 0x5ae5c680, 0x5de40c00
Caught exception in looking up space point ptr... 0x366d3230, 0x5bc20790
Caught exception in looking up space point ptr... 0x5bc20790, 0x62d401e0
Caught exception in looking up space point ptr... 0x12094cf0, 0x71c97630
Caught exception in looking up space point ptr... 0x71c97630, 0x12094cf0
++++>>>> total num hits: 569496, num free: 554842
04-Feb-2026 02:14:42 GMT  Closed output file "np02vd_raw_run042433_0145_df-s04-d0_dw_0_20260203T193714_reco_stage0_20260204T021442_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         307.253        422.5        577.85        421.598       72.1586        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.5727e-05    0.000129629   0.000278849   0.000114634   5.2321e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                114.087       179.873       260.633       170.71        32.8117        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.5333        0.9433         2.623       0.878338      0.434365        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0166848     0.0204275     0.0721125     0.0170262     0.0123107       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.016565      0.016866      0.0177659     0.0168801    0.00026135       19     
produce:beamevent:BeamEvent                      0.000126034   0.000231159   0.00110357    0.000172197   0.000210311      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.9316       18.1487       24.9622       18.145        3.09654        19     
produce:ophit:OpHitFinder                         0.0397019     0.0530955     0.0795404     0.0469043     0.0126793       19     
produce:wclsdatavd:WireCellToolkit                 148.628       180.137       317.06        155.249       44.4753        19     
produce:gaushit:GausHitFinder                      1.59803       2.25145       3.73289       2.15197      0.514464        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000484936   0.000742624   0.00200317    0.000609114   0.000363388      19     
produce:reco3d:SpacePointSolver                    13.5059       19.9451       28.8508       19.0051       4.07664        19     
produce:hitpdune:DisambigFromSpacePoints          0.225389      0.365261      0.600218      0.345446      0.107379        19     
produce:cluster3d:Cluster3D                        1.80821       13.4183       80.1274       3.48904       20.9695        19     
[art]:TriggerResults:TriggerResultInserter       2.6678e-05    4.13982e-05   0.000117056   3.7288e-05    1.88651e-05      19     
end_path:out1:RootOutput                          3.943e-06    2.18227e-05   0.000247543    7.42e-06     5.35485e-05      19     
end_path:out1:RootOutput(write)                    5.48511       7.27673       11.5999       6.32598       1.68558        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5559.41 MB
  Peak resident set size usage (VmHWM): 3386.51 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0145_df-s04-d0_dw_0_20260203T193714_reco_stage0_20260204T021442_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0145_df-s04-d0_dw_0_20260203T193714_reco_stage0_20260204T021442_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0145_df-s04-d0_dw_0_20260203T193714_reco_stage0_20260204T021442_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163179.0,
    "core.end_time": 1770171284.0,
    "core.events": [
      16540,
      16546,
      16552,
      16558,
      16564,
      16570,
      16576,
      16582,
      16588,
      16594,
      16600,
      16606,
      16612,
      16618,
      16624,
      16630,
      16636,
      16642,
      16648
    ],
    "core.event_count": 19,
    "core.first_event_number": 16540,
    "core.last_event_number": 16648,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0145_df-s04-d0_dw_0_20260203T193714.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:23 UTC       justIN version: 01.06.00