justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297917.149@dunegpschedd02.fnal.gov

Jobsub ID297917.149@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:25
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat2026-02-04 01:20:36
From worker nodeHostnamewn-snel-001.farm.nikhef.nl
cpuinfoAMD EPYC 7H12 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:24:21
Input filesvd-protodune:np02vd_raw_run042421_0106_df-s03-d1_dw_0_20260203T142821.hdf5
JobscriptExit code0
Real time1h (6834s)
CPU time51m (3071s = 44%)
Max RSS bytes2610610176 (2489 MiB)
Outputting started2026-02-04 01:18:17
Output files
Finished2026-02-04 01:20:36
Saved logsjustin-logs:297917.149-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

t exception in looking up space point ptr... 0x598f3a70, 0x42b17d60
Caught exception in looking up space point ptr... 0x67744220, 0x48ba2e80
Caught exception in looking up space point ptr... 0x48ba2e80, 0x4c797ff0
Caught exception in looking up space point ptr... 0x4c797ff0, 0x748d3c70
Caught exception in looking up space point ptr... 0x748d3c70, 0x42c9fe70
Caught exception in looking up space point ptr... 0x78c97710, 0x7b7df9d0
Caught exception in looking up space point ptr... 0x7b7df9d0, 0x332e1990
Caught exception in looking up space point ptr... 0x75aab830, 0x78c97710
Caught exception in looking up space point ptr... 0x627f0830, 0x627f0730
Caught exception in looking up space point ptr... 0x627f0630, 0x47d4d750
Caught exception in looking up space point ptr... 0x47d4d750, 0x47d4d950
Caught exception in looking up space point ptr... 0x47d4d950, 0x627f0830
Caught exception in looking up space point ptr... 0x6ed847b0, 0x6ed84620
Caught exception in looking up space point ptr... 0x6ed84620, 0x6ed84940
Caught exception in looking up space point ptr... 0x6ed84940, 0x3fea8930
Caught exception in looking up space point ptr... 0x3fea8930, 0x3fea8ac0
Caught exception in looking up space point ptr... 0x5a4e3230, 0x25f8e280
Caught exception in looking up space point ptr... 0x25f8e280, 0x4da736e0
Caught exception in looking up space point ptr... 0x40b83110, 0x40b83430
Caught exception in looking up space point ptr... 0x40b83430, 0x549a0130
Caught exception in looking up space point ptr... 0x77857430, 0x749accd0
Caught exception in looking up space point ptr... 0x4990db40, 0x77857430
Caught exception in looking up space point ptr... 0x20210860, 0x49851610
Caught exception in looking up space point ptr... 0x49851610, 0x20210860
Caught exception in looking up space point ptr... 0x3ff6bb30, 0x28768b20
Caught exception in looking up space point ptr... 0x28768b20, 0x2d017b90
Caught exception in looking up space point ptr... 0x30f82a00, 0x30f82870
Caught exception in looking up space point ptr... 0x30f82870, 0x30f826e0
Caught exception in looking up space point ptr... 0x30f826e0, 0x612d28e0
Caught exception in looking up space point ptr... 0x612d2c00, 0x30f82a00
Caught exception in looking up space point ptr... 0x4977b050, 0x420928d0
Caught exception in looking up space point ptr... 0x420928d0, 0x56216e10
Caught exception in looking up space point ptr... 0x467035c0, 0x46703810
Caught exception in looking up space point ptr... 0x46703810, 0x467036c0
Caught exception in looking up space point ptr... 0x42cf3700, 0x4aab2240
Caught exception in looking up space point ptr... 0x2e766f90, 0x42cf3700
Caught exception in looking up space point ptr... 0x70ebca20, 0x700ef1f0
Caught exception in looking up space point ptr... 0x3291b890, 0x3e8fead0
Caught exception in looking up space point ptr... 0x3e8fead0, 0x70ebca20
Caught exception in looking up space point ptr... 0x6fb76170, 0x6fb75ec0
Caught exception in looking up space point ptr... 0x6fb75ec0, 0x6fb76170
Caught exception in looking up space point ptr... 0x421a1050, 0x421a0f70
Caught exception in looking up space point ptr... 0x421a0f70, 0x421a1050
Caught exception in looking up space point ptr... 0x76c68de0, 0x53434e30
Caught exception in looking up space point ptr... 0x53434e30, 0x76c68de0
Caught exception in looking up space point ptr... 0x42023860, 0x40a14c80
Caught exception in looking up space point ptr... 0x40a14c80, 0x6ed84c60
Caught exception in looking up space point ptr... 0x6a2cc8b0, 0x6a2cc320
Caught exception in looking up space point ptr... 0x6a2cc320, 0x41a56e50
Caught exception in looking up space point ptr... 0x74c1df30, 0x6fb09790
Caught exception in looking up space point ptr... 0x6fb09790, 0x74c1df30
Caught exception in looking up space point ptr... 0x2d32b0e0, 0x2d32adb0
Caught exception in looking up space point ptr... 0x2d32adb0, 0x2d32b0e0
Caught exception in looking up space point ptr... 0x42478e60, 0x6a2cbfe0
Caught exception in looking up space point ptr... 0x6a2cbfe0, 0x42478e60
Caught exception in looking up space point ptr... 0x6a40eb00, 0x4d729420
Caught exception in looking up space point ptr... 0x4d729420, 0x31f9ced0
Caught exception in looking up space point ptr... 0x4a7b5150, 0x4a7b4e30
Caught exception in looking up space point ptr... 0x4a7b4e30, 0x4a7b4ca0
Caught exception in looking up space point ptr... 0x6e7ab3d0, 0x5e9e1c60
Caught exception in looking up space point ptr... 0x5e9e1c60, 0x6e7ab3d0
Caught exception in looking up space point ptr... 0x2ddb0fe0, 0x756020a0
Caught exception in looking up space point ptr... 0x756020a0, 0x2ddb0fe0
Caught exception in looking up space point ptr... 0x655ec920, 0x25552360
Caught exception in looking up space point ptr... 0x25552360, 0x596a3070
++++>>>> total num hits: 46430, num free: 45144
04-Feb-2026 02:18:07 CET  Closed output file "np02vd_raw_run042421_0106_df-s03-d1_dw_0_20260203T142821_reco_stage0_20260204T011807_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         207.651       248.93        352.266       240.523       35.3789        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        9.045e-05    0.000112693   0.00032462    0.000101621   4.2882e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                82.2582       129.422       224.932       121.886       33.5509        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.564391      0.798753       1.38474      0.754109       0.19177        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.017512      0.0354048     0.272419      0.0218065     0.0503277       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0856285     0.0926549     0.131996      0.0882668     0.0101213       27     
produce:beamevent:BeamEvent                      8.9278e-05    0.00017694    0.00137574    0.000123202   0.00023761       27     
produce:pdvddaphne:DAPHNEReaderPDVD                12.2872       16.3445       24.8075       16.2044       3.13656        27     
produce:ophit:OpHitFinder                         0.0339106     0.0442832     0.0552654     0.0444225    0.00508999       27     
produce:wclsdatavd:WireCellToolkit                 67.2101       79.8814       94.837        79.5567       7.09608        27     
produce:gaushit:GausHitFinder                      0.62998        1.269        2.07637       1.26346      0.367773        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000267924   0.000571433   0.00176973    0.000451028   0.000361894      27     
produce:reco3d:SpacePointSolver                    7.17967       13.0697       20.2943       12.6415       3.48212        27     
produce:hitpdune:DisambigFromSpacePoints          0.0607795     0.128188      0.224144      0.132048      0.0385035       27     
produce:cluster3d:Cluster3D                       0.825758       2.69188       4.71326       2.63231      0.909827        27     
[art]:TriggerResults:TriggerResultInserter        2.143e-05    3.3484e-05    0.000119465   3.0808e-05    1.76345e-05      27     
end_path:out1:RootOutput                          3.757e-06    6.47596e-06   3.2602e-05     4.94e-06     5.82141e-06      27     
end_path:out1:RootOutput(write)                    4.14469       5.11762       6.26038       5.24467      0.532977        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4534.5 MB
  Peak resident set size usage (VmHWM): 2610.61 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0106_df-s03-d1_dw_0_20260203T142821_reco_stage0_20260204T011807_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0106_df-s03-d1_dw_0_20260203T142821_reco_stage0_20260204T011807_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0106_df-s03-d1_dw_0_20260203T142821_reco_stage0_20260204T011807_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161075.0,
    "core.end_time": 1770167889.0,
    "core.events": [
      17241,
      17247,
      17253,
      17259,
      17265,
      17271,
      17277,
      17283,
      17289,
      17295,
      17301,
      17307,
      17313,
      17319,
      17325,
      17331,
      17337,
      17343,
      17349,
      17355,
      17361,
      17367,
      17373,
      17379,
      17385,
      17391,
      17397
    ],
    "core.event_count": 27,
    "core.first_event_number": 17241,
    "core.last_event_number": 17397,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0106_df-s03-d1_dw_0_20260203T142821.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:44:10 UTC       justIN version: 01.06.00