justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297920.26@dunegpschedd02.fnal.gov

Jobsub ID297920.26@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:54:27
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2026-02-04 00:48:48
From worker nodeHostnamewn-snel-031.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:37:20
Input filesvd-protodune:np02vd_raw_run042433_0467_df-s04-d0_dw_0_20260203T213937.hdf5
JobscriptExit code0
Real time1h (4135s)
CPU time1h (3930s = 95%)
Max RSS bytes3463409664 (3302 MiB)
Outputting started2026-02-04 00:46:16
Output files
Finished2026-02-04 00:48:48
Saved logsjustin-logs:297920.26-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

. 0x5864c3b0, 0x5864b290
Caught exception in looking up space point ptr... 0x4e1faed0, 0x42b13840
Caught exception in looking up space point ptr... 0x42b13840, 0x5a84ce00
Caught exception in looking up space point ptr... 0x5a84ce00, 0x52c22370
Caught exception in looking up space point ptr... 0x52c22370, 0x77c60600
Caught exception in looking up space point ptr... 0x721ae130, 0x4e1faed0
Caught exception in looking up space point ptr... 0x429795c0, 0x47745880
Caught exception in looking up space point ptr... 0x47745880, 0x429795c0
Caught exception in looking up space point ptr... 0x4bf85720, 0x5d1be910
Caught exception in looking up space point ptr... 0x5d1be910, 0x4bf85720
Caught exception in looking up space point ptr... 0x4bf85720, 0x4bf85940
Caught exception in looking up space point ptr... 0x4bf85940, 0x4bf85720
Caught exception in looking up space point ptr... 0x4bf85720, 0x5d1beab0
Caught exception in looking up space point ptr... 0x5d1beab0, 0x4bf85720
Caught exception in looking up space point ptr... 0x251b3aa0, 0x22f39470
Caught exception in looking up space point ptr... 0x22f39470, 0x251b3aa0
Caught exception in looking up space point ptr... 0x7445a3b0, 0x4cb9d2c0
Caught exception in looking up space point ptr... 0x4cb9d2c0, 0x12e203a0
Caught exception in looking up space point ptr... 0x4bf85130, 0x70d9b430
Caught exception in looking up space point ptr... 0x70d9b430, 0x4bf85c50
Caught exception in looking up space point ptr... 0x4bf85c50, 0x70d9b190
Caught exception in looking up space point ptr... 0x70d9b2e0, 0x70d9b430
Caught exception in looking up space point ptr... 0x70d9b430, 0x4bf84a10
Caught exception in looking up space point ptr... 0x4bf84a10, 0x70d9b430
Caught exception in looking up space point ptr... 0x70d9b430, 0x4bf85130
Caught exception in looking up space point ptr... 0x8a25e480, 0x8a25e2c0
Caught exception in looking up space point ptr... 0x8a25e2c0, 0x8a25e480
Caught exception in looking up space point ptr... 0x727e4550, 0x771f8250
Caught exception in looking up space point ptr... 0x771f8250, 0x68392560
Caught exception in looking up space point ptr... 0x4bf844e0, 0x90886ab0
Caught exception in looking up space point ptr... 0x90886ab0, 0x4bf844e0
Caught exception in looking up space point ptr... 0x4bf844e0, 0x4bf843f0
Caught exception in looking up space point ptr... 0x4bf843f0, 0x4bf844e0
Caught exception in looking up space point ptr... 0x4bf844e0, 0x90884cc0
Caught exception in looking up space point ptr... 0x90884cc0, 0x4bf844e0
Caught exception in looking up space point ptr... 0x1fb8e8f0, 0x76ca7ee0
Caught exception in looking up space point ptr... 0x76ca7ee0, 0x76ca7d40
Caught exception in looking up space point ptr... 0x54819a80, 0x54819020
Caught exception in looking up space point ptr... 0x54819020, 0x43299630
Caught exception in looking up space point ptr... 0x770482e0, 0x29e84b00
Caught exception in looking up space point ptr... 0x29e84b00, 0x5a8f3fb0
Caught exception in looking up space point ptr... 0x92cacc20, 0x7660cd70
Caught exception in looking up space point ptr... 0x7660cd70, 0x82a5e390
Caught exception in looking up space point ptr... 0x29cd7ef0, 0x916fbab0
Caught exception in looking up space point ptr... 0x916fbab0, 0x92cacc20
Caught exception in looking up space point ptr... 0x26ce1280, 0x91519d10
Caught exception in looking up space point ptr... 0x91519d10, 0x601581d0
Caught exception in looking up space point ptr... 0x18e2f520, 0x3fe4a6c0
Caught exception in looking up space point ptr... 0x3fe4a6c0, 0x52d12530
Caught exception in looking up space point ptr... 0x4036d0e0, 0x1cf73cd0
Caught exception in looking up space point ptr... 0x1cf73cd0, 0x4036d0e0
Caught exception in looking up space point ptr... 0x3e41e080, 0x61922080
Caught exception in looking up space point ptr... 0x61922080, 0x58490940
Caught exception in looking up space point ptr... 0x7d6d05e0, 0x7d6d0730
Caught exception in looking up space point ptr... 0x7d6d0730, 0x7d6d09d0
Caught exception in looking up space point ptr... 0x77c428f0, 0x247432f0
Caught exception in looking up space point ptr... 0x247432f0, 0x77c428f0
Caught exception in looking up space point ptr... 0x5bb99750, 0x5415bf90
Caught exception in looking up space point ptr... 0x5415bf90, 0x5bb99750
Caught exception in looking up space point ptr... 0x7b9c77b0, 0x71118b60
Caught exception in looking up space point ptr... 0x71118b60, 0x61303b90
Caught exception in looking up space point ptr... 0x6fe78150, 0x523d8b00
Caught exception in looking up space point ptr... 0x523d8b00, 0x6fe78150
Caught exception in looking up space point ptr... 0x6fe78150, 0x48d0ee40
Caught exception in looking up space point ptr... 0x48d0ee40, 0x6fe78150
Caught exception in looking up space point ptr... 0x56d6da30, 0x56d6ded0
Caught exception in looking up space point ptr... 0x56d6ded0, 0x56d6dc80
++++>>>> total num hits: 50199, num free: 48322
04-Feb-2026 01:46:00 CET  Closed output file "np02vd_raw_run042433_0467_df-s04-d0_dw_0_20260203T213937_reco_stage0_20260204T004600_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         178.979       211.614       242.205       215.376       17.2733        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.6212e-05    0.000110859   0.000303821   9.9627e-05    4.5965e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                14.3557       14.8025       15.122        14.8515      0.245362        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.523251      0.544195      0.751819      0.527086      0.0499112       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0173379     0.0177678     0.0180447     0.0177771    0.000181754      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0172728     0.0177687     0.0181765     0.0177915    0.000190562      19     
produce:beamevent:BeamEvent                      7.7936e-05    0.000184815   0.00118186    0.000141757   0.000236237      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.47375       3.88925       4.1568        3.94075      0.182365        19     
produce:ophit:OpHitFinder                         0.034027      0.0434895     0.0563115     0.0432689    0.00605133       19     
produce:wclsdatavd:WireCellToolkit                 127.393       147.356       166.525       148.946       10.4027        19     
produce:gaushit:GausHitFinder                      1.26284       1.96759       2.74243       2.03405      0.404759        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000430038   0.000690753   0.00117711    0.000678295   0.000189155      19     
produce:reco3d:SpacePointSolver                    15.0254       29.4208       37.4161       30.7796       5.76988        19     
produce:hitpdune:DisambigFromSpacePoints          0.122884      0.322157      0.499557      0.326099      0.0953201       19     
produce:cluster3d:Cluster3D                        1.95423       5.61668       22.5662       4.8556        4.17516        19     
[art]:TriggerResults:TriggerResultInserter       2.3815e-05    3.97521e-05   0.000122741   3.4034e-05    2.1463e-05       19     
end_path:out1:RootOutput                          3.837e-06    6.29395e-06    3.142e-05     4.769e-06    6.01078e-06      19     
end_path:out1:RootOutput(write)                     6.077        7.55239       8.49194       7.57511      0.694271        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5372.29 MB
  Peak resident set size usage (VmHWM): 3463.41 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0467_df-s04-d0_dw_0_20260203T213937_reco_stage0_20260204T004600_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0467_df-s04-d0_dw_0_20260203T213937_reco_stage0_20260204T004600_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0467_df-s04-d0_dw_0_20260203T213937_reco_stage0_20260204T004600_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161854.0,
    "core.end_time": 1770165962.0,
    "core.events": [
      53248,
      53254,
      53260,
      53266,
      53272,
      53278,
      53284,
      53290,
      53296,
      53302,
      53308,
      53314,
      53320,
      53326,
      53332,
      53338,
      53344,
      53350,
      53356
    ],
    "core.event_count": 19,
    "core.first_event_number": 53248,
    "core.last_event_number": 53356,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0467_df-s04-d0_dw_0_20260203T213937.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:24 UTC       justIN version: 01.06.00