justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.76@dunegpschedd02.fnal.gov

Jobsub ID297923.76@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 02:07:46
From worker nodeHostnamewn-snel-032.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:59:39
Input filesvd-protodune:np02vd_raw_run042433_0186_df-s04-d0_dw_0_20260203T195250.hdf5
JobscriptExit code0
Real time2h (7520s)
CPU time1h (4025s = 53%)
Max RSS bytes3582230528 (3416 MiB)
Outputting started2026-02-04 02:05:03
Output files
Finished2026-02-04 02:07:46
Saved logsjustin-logs:297923.76-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x216daab0, 0x216da920
Caught exception in looking up space point ptr... 0x4165e9b0, 0x4164de50
Caught exception in looking up space point ptr... 0x4164de50, 0x4165ec00
Caught exception in looking up space point ptr... 0x85141c70, 0x85141ae0
Caught exception in looking up space point ptr... 0x851414a0, 0x85141c70
Caught exception in looking up space point ptr... 0x206debe0, 0x7e8cf5e0
Caught exception in looking up space point ptr... 0x206de8c0, 0x206debe0
Caught exception in looking up space point ptr... 0x49597780, 0x714bc5a0
Caught exception in looking up space point ptr... 0x714bc5a0, 0x49597780
Caught exception in looking up space point ptr... 0x29623040, 0x29622ba0
Caught exception in looking up space point ptr... 0x29621dc0, 0x29623040
Caught exception in looking up space point ptr... 0x573529b0, 0x57480310
Caught exception in looking up space point ptr... 0x57480310, 0x573529b0
Caught exception in looking up space point ptr... 0x40df0120, 0x40dff6c0
Caught exception in looking up space point ptr... 0x40ddfdc0, 0x40df0120
Caught exception in looking up space point ptr... 0x6a4e2d10, 0x41337510
Caught exception in looking up space point ptr... 0x41337510, 0x4bcb1580
Caught exception in looking up space point ptr... 0x91459df0, 0x6a663e00
Caught exception in looking up space point ptr... 0x91459f80, 0x91459df0
Caught exception in looking up space point ptr... 0x69a55740, 0x5225e390
Caught exception in looking up space point ptr... 0x69a55950, 0x69a55740
Caught exception in looking up space point ptr... 0x8e488630, 0x8e4884a0
Caught exception in looking up space point ptr... 0x8e4884a0, 0x4144da10
Caught exception in looking up space point ptr... 0x40d5fdf0, 0x1a567af0
Caught exception in looking up space point ptr... 0x40d5ff80, 0x40d5fdf0
Caught exception in looking up space point ptr... 0x290e4070, 0x290c4510
Caught exception in looking up space point ptr... 0x290e4200, 0x290e4070
Caught exception in looking up space point ptr... 0x44b38e20, 0x44b4a120
Caught exception in looking up space point ptr... 0x44b2be50, 0x44b38e20
Caught exception in looking up space point ptr... 0x7670ee40, 0x72920f70
Caught exception in looking up space point ptr... 0x72920f70, 0x7dcaaab0
Caught exception in looking up space point ptr... 0x4aaaa2e0, 0x4aa98cc0
Caught exception in looking up space point ptr... 0x4aa98cc0, 0x4aa86a20
Caught exception in looking up space point ptr... 0x475f50a0, 0x695cf9e0
Caught exception in looking up space point ptr... 0x695cf9e0, 0x475f50a0
Caught exception in looking up space point ptr... 0x475f50a0, 0x28a9b2f0
Caught exception in looking up space point ptr... 0x28a9b2f0, 0x475f50a0
Caught exception in looking up space point ptr... 0x475f50a0, 0x5253f300
Caught exception in looking up space point ptr... 0x5253f300, 0x475f50a0
Caught exception in looking up space point ptr... 0x216b20b0, 0x7fd778d0
Caught exception in looking up space point ptr... 0x216c2be0, 0x216b20b0
Caught exception in looking up space point ptr... 0x898eac90, 0x46fb8880
Caught exception in looking up space point ptr... 0x46fb8880, 0x2636cfe0
Caught exception in looking up space point ptr... 0x68d736f0, 0x715ed4f0
Caught exception in looking up space point ptr... 0x715ed4f0, 0x6a3af700
Caught exception in looking up space point ptr... 0x75c0af40, 0x75c0add0
Caught exception in looking up space point ptr... 0x75c0add0, 0x470c6410
Caught exception in looking up space point ptr... 0x7764ed20, 0x5247b770
Caught exception in looking up space point ptr... 0x5247b770, 0x7764ed20
Caught exception in looking up space point ptr... 0x79c25180, 0x79c254a0
Caught exception in looking up space point ptr... 0x79c254a0, 0x79c25180
Caught exception in looking up space point ptr... 0x293a6220, 0x8988ffc0
Caught exception in looking up space point ptr... 0x3bed0e40, 0x293a6220
Caught exception in looking up space point ptr... 0x69db5030, 0x69db4cc0
Caught exception in looking up space point ptr... 0x69db4cc0, 0x69db5030
Caught exception in looking up space point ptr... 0x48c05c90, 0x56ab0f50
Caught exception in looking up space point ptr... 0x56ab0f50, 0x48c05b90
Caught exception in looking up space point ptr... 0x294a7e30, 0x5253abe0
Caught exception in looking up space point ptr... 0x5253abe0, 0x294a7e30
Caught exception in looking up space point ptr... 0x255eb490, 0x255eb620
Caught exception in looking up space point ptr... 0x255eb620, 0x255eb490
Caught exception in looking up space point ptr... 0x49067360, 0x49067400
Caught exception in looking up space point ptr... 0x49067400, 0x4ee574c0
Caught exception in looking up space point ptr... 0x4d2ab7c0, 0x697ff410
Caught exception in looking up space point ptr... 0x697ff410, 0x81afc940
Caught exception in looking up space point ptr... 0x18f2e5d0, 0x475eeac0
Caught exception in looking up space point ptr... 0x475eeac0, 0x18f2e5d0
++++>>>> total num hits: 514171, num free: 506046
04-Feb-2026 03:04:51 CET  Closed output file "np02vd_raw_run042433_0186_df-s04-d0_dw_0_20260203T195250_reco_stage0_20260204T020451_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         292.624       390.903       444.672       402.641       44.2227        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.2664e-05    0.000120852   0.000330238   0.000105968   5.17693e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                117.88        177.82        264.55        177.245        34.88         19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.597894      0.881846       1.2403       0.863423      0.176704        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0175825     0.0199575     0.0272483     0.0180711    0.00325048       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0172131     0.0177798     0.0187305     0.0177269    0.000341036      19     
produce:beamevent:BeamEvent                      7.5682e-05    0.00020418    0.00164377    0.000115226   0.000340629      19     
produce:pdvddaphne:DAPHNEReaderPDVD                11.6851       15.6387       19.6592       15.7604       2.5613         19     
produce:ophit:OpHitFinder                         0.0265548     0.0439443     0.0637909     0.0418931    0.00785163       19     
produce:wclsdatavd:WireCellToolkit                 115.765       124.956       138.882       125.685       5.68031        19     
produce:gaushit:GausHitFinder                     0.999219       2.03644       2.73457       2.24396      0.512795        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000479047   0.000932754   0.00148645    0.000895617   0.00024827       19     
produce:reco3d:SpacePointSolver                    9.81749       16.3081       21.2711       16.2333       3.05873        19     
produce:hitpdune:DisambigFromSpacePoints          0.135431      0.289509      0.448378      0.291173      0.0785747       19     
produce:cluster3d:Cluster3D                        2.13706       45.1774       70.8922       61.9168       28.5103        19     
[art]:TriggerResults:TriggerResultInserter       2.5077e-05    3.5575e-05    0.000127278   2.8814e-05    2.20649e-05      19     
end_path:out1:RootOutput                          4.018e-06    7.24726e-06   3.0557e-05     4.889e-06    6.67325e-06      19     
end_path:out1:RootOutput(write)                    6.28667       7.67036       10.5796       7.13639       1.23176        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5579.26 MB
  Peak resident set size usage (VmHWM): 3582.23 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0186_df-s04-d0_dw_0_20260203T195250_reco_stage0_20260204T020451_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0186_df-s04-d0_dw_0_20260203T195250_reco_stage0_20260204T020451_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0186_df-s04-d0_dw_0_20260203T195250_reco_stage0_20260204T020451_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163190.0,
    "core.end_time": 1770170694.0,
    "core.events": [
      21214,
      21220,
      21226,
      21232,
      21238,
      21244,
      21250,
      21256,
      21262,
      21268,
      21274,
      21280,
      21286,
      21292,
      21298,
      21304,
      21310,
      21316,
      21322
    ],
    "core.event_count": 19,
    "core.first_event_number": 21214,
    "core.last_event_number": 21322,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0186_df-s04-d0_dw_0_20260203T195250.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:16 UTC       justIN version: 01.06.00