justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297917.62@dunegpschedd02.fnal.gov

Jobsub ID297917.62@dunegpschedd02.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:48:25
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2026-02-04 01:10:02
From worker nodeHostnamewn-sate-013.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:10:50
Input filesvd-protodune:np02vd_raw_run042421_0045_df-s04-d1_dw_0_20260203T135522.hdf5
JobscriptExit code0
Real time1h (7024s)
CPU time48m (2883s = 41%)
Max RSS bytes2603548672 (2482 MiB)
Outputting started2026-02-04 01:07:56
Output files
Finished2026-02-04 01:10:02
Saved logsjustin-logs:297917.62-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x5828b460, 0x5828b2d0
Caught exception in looking up space point ptr... 0x5828b2d0, 0x5828b460
Caught exception in looking up space point ptr... 0x51a675d0, 0x38b87db0
Caught exception in looking up space point ptr... 0x38b87db0, 0x7367a280
Caught exception in looking up space point ptr... 0x706b4ae0, 0x706b4950
Caught exception in looking up space point ptr... 0x6a2b58c0, 0x706b4ae0
Caught exception in looking up space point ptr... 0x6744b730, 0x6744b5f0
Caught exception in looking up space point ptr... 0x6744b5f0, 0x6744b730
Caught exception in looking up space point ptr... 0x42245330, 0x42241270
Caught exception in looking up space point ptr... 0x42241270, 0x4224eb60
Caught exception in looking up space point ptr... 0x422409e0, 0x422410e0
Caught exception in looking up space point ptr... 0x422410e0, 0x422451a0
Caught exception in looking up space point ptr... 0x3656c6a0, 0x3656c7a0
Caught exception in looking up space point ptr... 0x3656c7a0, 0x365838a0
Caught exception in looking up space point ptr... 0x63da2cb0, 0x63da2e40
Caught exception in looking up space point ptr... 0x63da62a0, 0x63da2cb0
Caught exception in looking up space point ptr... 0x3815e4d0, 0x3815e660
Caught exception in looking up space point ptr... 0x3815e660, 0x7c76ef90
Caught exception in looking up space point ptr... 0x83ef98f0, 0x4ffc6ec0
Caught exception in looking up space point ptr... 0x64f3efb0, 0x83ef98f0
Caught exception in looking up space point ptr... 0x61e264e0, 0x61e0a270
Caught exception in looking up space point ptr... 0x61e0a450, 0x61e12200
Caught exception in looking up space point ptr... 0x61e12200, 0x61e26670
Caught exception in looking up space point ptr... 0x61e26670, 0x61e264e0
Caught exception in looking up space point ptr... 0x5e0ec6b0, 0x51b04f60
Caught exception in looking up space point ptr... 0x51b04f60, 0x5c32ee80
Caught exception in looking up space point ptr... 0x40ddec40, 0x3b94f140
Caught exception in looking up space point ptr... 0x3b94f140, 0x3d2b5480
Caught exception in looking up space point ptr... 0x3d2b5480, 0x412ee930
Caught exception in looking up space point ptr... 0x57c38be0, 0x57c35630
Caught exception in looking up space point ptr... 0x57c35630, 0x57c35ae0
Caught exception in looking up space point ptr... 0x57eaf290, 0x57eaf090
Caught exception in looking up space point ptr... 0x57eb0420, 0x57eaf290
Caught exception in looking up space point ptr... 0x5f0d68e0, 0x6395cac0
Caught exception in looking up space point ptr... 0x39241f00, 0x6395d4e0
Caught exception in looking up space point ptr... 0x6395d4e0, 0x5f0d68e0
Caught exception in looking up space point ptr... 0x655a15a0, 0x776bc380
Caught exception in looking up space point ptr... 0x776bc380, 0x76a1e590
Caught exception in looking up space point ptr... 0x6bf5ffe0, 0x5e055b80
Caught exception in looking up space point ptr... 0x3bf80620, 0x6bf5ffe0
Caught exception in looking up space point ptr... 0x6634d5d0, 0x5a090bb0
Caught exception in looking up space point ptr... 0x5a090bb0, 0x3d6faa20
Caught exception in looking up space point ptr... 0x6634ea90, 0x6f7882c0
Caught exception in looking up space point ptr... 0x6f7882c0, 0x676b55d0
Caught exception in looking up space point ptr... 0x676b5440, 0x676b5120
Caught exception in looking up space point ptr... 0x676b5120, 0x6634ea90
Caught exception in looking up space point ptr... 0x5da4fee0, 0x8da95ae0
Caught exception in looking up space point ptr... 0x5da50010, 0x5da4fdb0
Caught exception in looking up space point ptr... 0x5da4fdb0, 0x5da4fee0
Caught exception in looking up space point ptr... 0x58192040, 0x58191e60
Caught exception in looking up space point ptr... 0x58191e60, 0x58192220
Caught exception in looking up space point ptr... 0x67430c60, 0x6742f710
Caught exception in looking up space point ptr... 0x6742f710, 0x67430c60
Caught exception in looking up space point ptr... 0x69940fc0, 0x73a217f0
Caught exception in looking up space point ptr... 0x73a217f0, 0x58df98b0
Caught exception in looking up space point ptr... 0x58df98b0, 0x61049d20
Caught exception in looking up space point ptr... 0x673fda50, 0x761042a0
Caught exception in looking up space point ptr... 0x761042a0, 0x673fda50
Caught exception in looking up space point ptr... 0x50a90c80, 0x61d144d0
Caught exception in looking up space point ptr... 0x61d144d0, 0x50a90c80
Caught exception in looking up space point ptr... 0x5ff6a140, 0x5dfd36a0
Caught exception in looking up space point ptr... 0x5dfd36a0, 0x7b667af0
Caught exception in looking up space point ptr... 0x39062e40, 0x39067020
Caught exception in looking up space point ptr... 0x39067020, 0x39062e40
Caught exception in looking up space point ptr... 0x85513770, 0x85514780
Caught exception in looking up space point ptr... 0x85514780, 0x85513ac0
++++>>>> total num hits: 60664, num free: 58769
04-Feb-2026 02:07:48 CET  Closed output file "np02vd_raw_run042421_0045_df-s04-d1_dw_0_20260203T135522_reco_stage0_20260204T010748_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         193.52        256.895       330.778       252.084       35.8466        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.6485e-05    0.000107825   0.000289413   9.3596e-05    5.04758e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                90.4054       141.921       204.505       134.689       33.5313        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.622637      0.851054       1.43517      0.819069       0.18179        27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0174304     0.0231512     0.0473272     0.0208896    0.00698277       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0854008     0.0986336     0.325371      0.0867645     0.0455336       27     
produce:beamevent:BeamEvent                      9.8444e-05     0.0001917    0.000993475   0.00014395    0.00016383       27     
produce:pdvddaphne:DAPHNEReaderPDVD                9.17394       17.3106       24.1948       17.5223       3.79851        27     
produce:ophit:OpHitFinder                         0.0291696     0.0467074     0.0557678     0.0470722    0.00623272       27     
produce:wclsdatavd:WireCellToolkit                 75.4692       77.9372       81.9547       78.0843       1.32015        27     
produce:gaushit:GausHitFinder                     0.571521       1.09369       1.56892       1.13647      0.264188        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000219693   0.000403431   0.00215386    0.000325952   0.000350182      27     
produce:reco3d:SpacePointSolver                    6.13906       10.8629       17.031        11.1959       3.09042        27     
produce:hitpdune:DisambigFromSpacePoints          0.0683371     0.118478      0.246242      0.114916      0.0390193       27     
produce:cluster3d:Cluster3D                       0.706502       2.13987       8.36073       1.65826       1.66641        27     
[art]:TriggerResults:TriggerResultInserter       2.7842e-05    3.9205e-05    9.6621e-05    3.4334e-05    1.38291e-05      27     
end_path:out1:RootOutput                          7.254e-06     1.001e-05    3.4014e-05     8.546e-06    5.12129e-06      27     
end_path:out1:RootOutput(write)                    4.0258        4.45701       5.27308       4.44491      0.281556        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4531.36 MB
  Peak resident set size usage (VmHWM): 2603.55 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0045_df-s04-d1_dw_0_20260203T135522_reco_stage0_20260204T010748_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0045_df-s04-d1_dw_0_20260203T135522_reco_stage0_20260204T010748_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0045_df-s04-d1_dw_0_20260203T135522_reco_stage0_20260204T010748_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770160260.0,
    "core.end_time": 1770167270.0,
    "core.events": [
      7343,
      7349,
      7355,
      7361,
      7367,
      7373,
      7379,
      7385,
      7391,
      7397,
      7403,
      7409,
      7415,
      7421,
      7427,
      7433,
      7439,
      7445,
      7451,
      7457,
      7463,
      7469,
      7475,
      7481,
      7487,
      7493,
      7499
    ],
    "core.event_count": 27,
    "core.first_event_number": 7343,
    "core.last_event_number": 7499,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0045_df-s04-d1_dw_0_20260203T135522.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:57:12 UTC       justIN version: 01.06.00