justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.171@dunegpschedd01.fnal.gov

Jobsub ID303653.171@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat2026-02-04 01:40:43
From worker nodeHostnamewn-pijl-013.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:48:33
Input filesvd-protodune:np02vd_raw_run042421_0116_df-s04-d1_dw_0_20260203T143345.hdf5
JobscriptExit code0
Real time1h (6589s)
CPU time52m (3141s = 47%)
Max RSS bytes2597048320 (2476 MiB)
Outputting started2026-02-04 01:38:24
Output files
Finished2026-02-04 01:40:43
Saved logsjustin-logs:303653.171-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

exception in looking up space point ptr... 0x7bc27cc0, 0x7bc28090
Caught exception in looking up space point ptr... 0x3263a410, 0x4cc1c970
Caught exception in looking up space point ptr... 0x4cc1c970, 0x73140950
Caught exception in looking up space point ptr... 0x73140950, 0x5cf5d600
Caught exception in looking up space point ptr... 0x5cf5d600, 0x3263a410
Caught exception in looking up space point ptr... 0x7ba91f90, 0x7ba90bf0
Caught exception in looking up space point ptr... 0x7ba90bf0, 0x7ba8fff0
Caught exception in looking up space point ptr... 0x449a3390, 0x2a922300
Caught exception in looking up space point ptr... 0x2a922300, 0x2767d760
Caught exception in looking up space point ptr... 0x6a3c07c0, 0x77858cf0
Caught exception in looking up space point ptr... 0x825edfe0, 0x6a3c07c0
Caught exception in looking up space point ptr... 0x2dc6a820, 0x430adb90
Caught exception in looking up space point ptr... 0x2dc6a730, 0x2dc6a820
Caught exception in looking up space point ptr... 0x7ba95200, 0x7baa0000
Caught exception in looking up space point ptr... 0x7baa0000, 0x7baa68e0
Caught exception in looking up space point ptr... 0x40a354a0, 0x40a3cd00
Caught exception in looking up space point ptr... 0x40a3cd00, 0x40a354a0
Caught exception in looking up space point ptr... 0x602490a0, 0x60248f50
Caught exception in looking up space point ptr... 0x60248f50, 0x602490a0
Caught exception in looking up space point ptr... 0x47348cc0, 0x48a39c80
Caught exception in looking up space point ptr... 0x48a39c80, 0x47348cc0
Caught exception in looking up space point ptr... 0x73d56770, 0x48ab8ae0
Caught exception in looking up space point ptr... 0x48ab8ae0, 0x32aec4d0
Caught exception in looking up space point ptr... 0x46dd0280, 0x46dc6750
Caught exception in looking up space point ptr... 0x46dc6750, 0x46dd0280
Caught exception in looking up space point ptr... 0x7ba467e0, 0x2712c870
Caught exception in looking up space point ptr... 0x2712c870, 0x7ba467e0
Caught exception in looking up space point ptr... 0x564ec1d0, 0x564f3af0
Caught exception in looking up space point ptr... 0x564f3af0, 0x564f3ff0
Caught exception in looking up space point ptr... 0x522bd4f0, 0x42ff8a40
Caught exception in looking up space point ptr... 0x42ff8a40, 0x42ff8740
Caught exception in looking up space point ptr... 0x47187dc0, 0x6016c730
Caught exception in looking up space point ptr... 0x6016c730, 0x79bba110
Caught exception in looking up space point ptr... 0x2f13a910, 0x337f8100
Caught exception in looking up space point ptr... 0x337f8100, 0x2f13a910
Caught exception in looking up space point ptr... 0x527d5c40, 0x5b1659f0
Caught exception in looking up space point ptr... 0x5b1659f0, 0x309b39b0
Caught exception in looking up space point ptr... 0x643797e0, 0x560e68f0
Caught exception in looking up space point ptr... 0x560e68f0, 0x643797e0
Caught exception in looking up space point ptr... 0x643797e0, 0x41c7d940
Caught exception in looking up space point ptr... 0x41c7d940, 0x643797e0
Caught exception in looking up space point ptr... 0x643797e0, 0x41c7d7b0
Caught exception in looking up space point ptr... 0x41c7d7b0, 0x643797e0
Caught exception in looking up space point ptr... 0x2dc64e40, 0x2dc64fe0
Caught exception in looking up space point ptr... 0x333321a0, 0x2dc64e40
Caught exception in looking up space point ptr... 0x4db43ae0, 0x4db470d0
Caught exception in looking up space point ptr... 0x4db470d0, 0x4db46fa0
Caught exception in looking up space point ptr... 0x40a31f90, 0x40a32a40
Caught exception in looking up space point ptr... 0x40a31d00, 0x40a31f90
Caught exception in looking up space point ptr... 0x441f6840, 0x441f2530
Caught exception in looking up space point ptr... 0x441f2530, 0x441f2420
Caught exception in looking up space point ptr... 0x64334fa0, 0x26f06d20
Caught exception in looking up space point ptr... 0x26f06d20, 0x6b294140
Caught exception in looking up space point ptr... 0x5ea01f20, 0x522f65f0
Caught exception in looking up space point ptr... 0x522f65f0, 0x5ea01f20
Caught exception in looking up space point ptr... 0x27e74be0, 0x27e72660
Caught exception in looking up space point ptr... 0x27e72660, 0x27e74be0
Caught exception in looking up space point ptr... 0x7bc40720, 0x7bc40870
Caught exception in looking up space point ptr... 0x7bc40870, 0x7bc40b10
Caught exception in looking up space point ptr... 0x7bc40c60, 0x7bc409c0
Caught exception in looking up space point ptr... 0x7bc409c0, 0x7bc40720
Caught exception in looking up space point ptr... 0x28131b40, 0x28131cc0
Caught exception in looking up space point ptr... 0x28131cc0, 0x28131df0
Caught exception in looking up space point ptr... 0x56f33d00, 0x55045070
Caught exception in looking up space point ptr... 0x55045070, 0x56f33d00
++++>>>> total num hits: 57258, num free: 54904
04-Feb-2026 02:38:11 CET  Closed output file "np02vd_raw_run042421_0116_df-s04-d1_dw_0_20260203T143345_reco_stage0_20260204T013811_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         193.051       239.676       315.061       236.78        31.9061        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.1257e-05    0.000117501   0.000296818   0.000103417   4.18826e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                74.4168       118.927       181.411       112.006       27.3119        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.590038      0.790066       1.10417      0.782352      0.138639        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176085     0.018672      0.0241834     0.0182806    0.00135931       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.086257      0.088457      0.0912393     0.0883178    0.00122843       27     
produce:beamevent:BeamEvent                      0.000127522   0.000237268   0.00129199    0.000180252   0.00021468       27     
produce:pdvddaphne:DAPHNEReaderPDVD                11.4222       15.5508       21.1209       15.2377       2.88809        27     
produce:ophit:OpHitFinder                         0.0372121     0.0493843     0.0680745     0.0474082     0.0080831       27     
produce:wclsdatavd:WireCellToolkit                 62.4284       81.1421       94.3115       81.1572       7.93111        27     
produce:gaushit:GausHitFinder                     0.632748       1.17513       2.05213       1.09466       0.33754        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000368497    0.0010544    0.00227872    0.000657092   0.000624466      27     
produce:reco3d:SpacePointSolver                    7.16157       13.3547       23.3123       12.3815       3.5409         27     
produce:hitpdune:DisambigFromSpacePoints          0.0856577     0.159663      0.266477      0.148393      0.0558572       27     
produce:cluster3d:Cluster3D                        1.28461        3.169        5.99572       3.00964       1.28348        27     
[art]:TriggerResults:TriggerResultInserter       2.7602e-05    4.98231e-05   0.000109806   4.6029e-05    1.71069e-05      27     
end_path:out1:RootOutput                          3.685e-06    1.81841e-05   0.000301776    6.089e-06    5.57682e-05      27     
end_path:out1:RootOutput(write)                    3.72473       5.17808       6.79572       5.28276      0.782748        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4596.12 MB
  Peak resident set size usage (VmHWM): 2597.05 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0116_df-s04-d1_dw_0_20260203T143345_reco_stage0_20260204T013811_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0116_df-s04-d1_dw_0_20260203T143345_reco_stage0_20260204T013811_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0116_df-s04-d1_dw_0_20260203T143345_reco_stage0_20260204T013811_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162528.0,
    "core.end_time": 1770169094.0,
    "core.events": [
      18863,
      18869,
      18875,
      18881,
      18887,
      18893,
      18899,
      18905,
      18911,
      18917,
      18923,
      18929,
      18935,
      18941,
      18947,
      18953,
      18959,
      18965,
      18971,
      18977,
      18983,
      18989,
      18995,
      19001,
      19007,
      19013,
      19019
    ],
    "core.event_count": 27,
    "core.first_event_number": 18863,
    "core.last_event_number": 19019,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0116_df-s04-d1_dw_0_20260203T143345.hdf5"
    }
  ]
}
justIN time: 2026-02-04 08:56:53 UTC       justIN version: 01.06.00