justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303650.35@dunegpschedd01.fnal.gov

Jobsub ID303650.35@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:28
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 01:09:38
From worker nodeHostnamewn-lot-020.farm.nikhef.nl
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:13:40
Input filesvd-protodune:np02vd_raw_run042433_0456_df-s04-d0_dw_0_20260203T213526.hdf5
JobscriptExit code0
Real time1h (6828s)
CPU time51m (3072s = 44%)
Max RSS bytes3411140608 (3253 MiB)
Outputting started2026-02-04 01:07:29
Output files
Finished2026-02-04 01:09:38
Saved logsjustin-logs:303650.35-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x596b6070, 0x172fecf0
Caught exception in looking up space point ptr... 0x172fecf0, 0x596b6070
Caught exception in looking up space point ptr... 0x5134b8a0, 0x51348a40
Caught exception in looking up space point ptr... 0x51348a40, 0x51348970
Caught exception in looking up space point ptr... 0x17668280, 0x1766ba20
Caught exception in looking up space point ptr... 0x17667a50, 0x17668280
Caught exception in looking up space point ptr... 0x1d11f380, 0x1d11f130
Caught exception in looking up space point ptr... 0x1d11f130, 0x1d11f030
Caught exception in looking up space point ptr... 0x50627020, 0x50627170
Caught exception in looking up space point ptr... 0x50627170, 0x50627020
Caught exception in looking up space point ptr... 0x2226c7f0, 0x222514c0
Caught exception in looking up space point ptr... 0x222514c0, 0x2225dc80
Caught exception in looking up space point ptr... 0x5d506900, 0x5d4f9bf0
Caught exception in looking up space point ptr... 0x5d4f9bf0, 0x5d506900
Caught exception in looking up space point ptr... 0x88ef2d20, 0x88ef2bd0
Caught exception in looking up space point ptr... 0x88ef2bd0, 0x88ef2d20
Caught exception in looking up space point ptr... 0x43aa2550, 0x270ff780
Caught exception in looking up space point ptr... 0x270ff780, 0x43aa2550
Caught exception in looking up space point ptr... 0x51348520, 0x51346be0
Caught exception in looking up space point ptr... 0x51346be0, 0x51348520
Caught exception in looking up space point ptr... 0x8e2c85a0, 0x8e2c3c40
Caught exception in looking up space point ptr... 0x8e2c3c40, 0x8e2c3ab0
Caught exception in looking up space point ptr... 0x944c98b0, 0x944c9720
Caught exception in looking up space point ptr... 0x944c9720, 0x944c98b0
Caught exception in looking up space point ptr... 0x209303b0, 0x20935970
Caught exception in looking up space point ptr... 0x2092ff80, 0x209303b0
Caught exception in looking up space point ptr... 0x22416c20, 0x4812db00
Caught exception in looking up space point ptr... 0x4812db00, 0x481b3900
Caught exception in looking up space point ptr... 0x6dfbcf50, 0x56d9a1b0
Caught exception in looking up space point ptr... 0x56d9a1b0, 0x6dfbcf50
Caught exception in looking up space point ptr... 0x8e2a9f00, 0x8e2a9560
Caught exception in looking up space point ptr... 0x8e2a9560, 0x8e2ad080
Caught exception in looking up space point ptr... 0x8e2ad080, 0x8e2aae00
Caught exception in looking up space point ptr... 0x4c7641e0, 0x170f4b10
Caught exception in looking up space point ptr... 0x170f4b10, 0x4c7634d0
Caught exception in looking up space point ptr... 0x1eda0cd0, 0x1edc15a0
Caught exception in looking up space point ptr... 0x1edc15a0, 0x1edb8060
Caught exception in looking up space point ptr... 0x681e2ab0, 0x681e29b0
Caught exception in looking up space point ptr... 0x681e29b0, 0x681e2ab0
Caught exception in looking up space point ptr... 0x8e2ac300, 0x8e2ac640
Caught exception in looking up space point ptr... 0x8e2ac640, 0x8e2ac300
Caught exception in looking up space point ptr... 0x8e2ac300, 0x8e2ac4a0
Caught exception in looking up space point ptr... 0x8e2ac4a0, 0x8e2ac300
Caught exception in looking up space point ptr... 0x5aceb0a0, 0x5aceb340
Caught exception in looking up space point ptr... 0x5aceb340, 0x5aceb0a0
Caught exception in looking up space point ptr... 0x22270680, 0x22254660
Caught exception in looking up space point ptr... 0x22254660, 0x22270680
Caught exception in looking up space point ptr... 0x2711a000, 0x47f5b750
Caught exception in looking up space point ptr... 0x47f5b750, 0x2711a000
Caught exception in looking up space point ptr... 0x2711a000, 0x270e8360
Caught exception in looking up space point ptr... 0x270e8360, 0x2711a000
Caught exception in looking up space point ptr... 0x7500a530, 0x6eef0fa0
Caught exception in looking up space point ptr... 0x6eef0fa0, 0x3b048d60
Caught exception in looking up space point ptr... 0x594dc6e0, 0x594dbc30
Caught exception in looking up space point ptr... 0x594dbc30, 0x594dc6e0
Caught exception in looking up space point ptr... 0x7a6bc710, 0x7a6bc580
Caught exception in looking up space point ptr... 0x7a6bc580, 0x7a6bc710
Caught exception in looking up space point ptr... 0x594d9520, 0x594d64c0
Caught exception in looking up space point ptr... 0x594d64c0, 0x594d69b0
Caught exception in looking up space point ptr... 0x185788b0, 0x185785b0
Caught exception in looking up space point ptr... 0x185785b0, 0x185788b0
Caught exception in looking up space point ptr... 0x22248300, 0x222485e0
Caught exception in looking up space point ptr... 0x222485e0, 0x22248300
Caught exception in looking up space point ptr... 0x418a9ee0, 0x418a8e00
Caught exception in looking up space point ptr... 0x418a8e00, 0x418a9ee0
Caught exception in looking up space point ptr... 0x594d9f30, 0x594d9ce0
Caught exception in looking up space point ptr... 0x594d9ce0, 0x594da180
++++>>>> total num hits: 127352, num free: 122571
04-Feb-2026 02:07:14 CET  Closed output file "np02vd_raw_run042433_0456_df-s04-d0_dw_0_20260203T213526_reco_stage0_20260204T010714_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         289.387       354.505       468.799       346.585       46.9281        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        7.466e-05    9.88432e-05   0.000226094   8.9628e-05    3.19188e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                127.623       188.596       279.832       179.917       44.805         19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.636246       1.17208       8.08971      0.781418       1.63398        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0174703     0.0350578     0.264293      0.019697      0.0545233       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0172878     0.022259      0.0421485     0.017518     0.00935999       19     
produce:beamevent:BeamEvent                      8.9718e-05    0.000153771   0.000978944   0.000105408   0.000194981      19     
produce:pdvddaphne:DAPHNEReaderPDVD                12.4889       17.0054       24.672        15.8313       3.37819        19     
produce:ophit:OpHitFinder                         0.0276881     0.0533619     0.0687647     0.0563024    0.00950569       19     
produce:wclsdatavd:WireCellToolkit                 97.9356       110.828       124.04        111.877       6.83657        19     
produce:gaushit:GausHitFinder                     0.746571       1.26748       1.59471       1.26107      0.216481        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000336491   0.000598005    0.0011881    0.000454862   0.000301631      19     
produce:reco3d:SpacePointSolver                    13.2051       23.4405       31.4549       23.2726       4.79235        19     
produce:hitpdune:DisambigFromSpacePoints           0.11845      0.203195      0.277739      0.194647      0.0393347       19     
produce:cluster3d:Cluster3D                        0.91175       2.39122       3.27582       2.54893      0.641194        19     
[art]:TriggerResults:TriggerResultInserter       1.7683e-05    2.55555e-05   9.6771e-05    1.9727e-05    1.71343e-05      19     
end_path:out1:RootOutput                          3.276e-06    5.41063e-06   2.4997e-05     3.677e-06    4.92791e-06      19     
end_path:out1:RootOutput(write)                    7.57019       9.45248       11.5319       9.63536       1.00569        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5345.41 MB
  Peak resident set size usage (VmHWM): 3411.14 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0456_df-s04-d0_dw_0_20260203T213526_reco_stage0_20260204T010714_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0456_df-s04-d0_dw_0_20260203T213526_reco_stage0_20260204T010714_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0456_df-s04-d0_dw_0_20260203T213526_reco_stage0_20260204T010714_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160429.0,
    "core.end_time": 1770167237.0,
    "core.events": [
      51994,
      52000,
      52006,
      52012,
      52018,
      52024,
      52030,
      52036,
      52042,
      52048,
      52054,
      52060,
      52066,
      52072,
      52078,
      52084,
      52090,
      52096,
      52102
    ],
    "core.event_count": 19,
    "core.first_event_number": 51994,
    "core.last_event_number": 52102,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0456_df-s04-d0_dw_0_20260203T213526.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:23 UTC       justIN version: 01.06.00