justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.186@dunegpschedd01.fnal.gov

Jobsub ID303651.186@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk_multicore
Last heartbeat2026-02-04 01:23:53
From worker nodeHostnamewn-pep-007.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:23:33
Input filesvd-protodune:np02vd_raw_run042421_0027_df-s03-d1_dw_0_20260203T134537.hdf5
JobscriptExit code0
Real time1h (7095s)
CPU time49m (2980s = 42%)
Max RSS bytes2760646656 (2632 MiB)
Outputting started2026-02-04 01:21:50
Output files
Finished2026-02-04 01:23:53
Saved logsjustin-logs:303651.186-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x66814990
Caught exception in looking up space point ptr... 0x66810230, 0x668140d0
Caught exception in looking up space point ptr... 0x578198c0, 0x72e7d640
Caught exception in looking up space point ptr... 0x72e7d640, 0x578198c0
Caught exception in looking up space point ptr... 0x69c9b1a0, 0x69c9adb0
Caught exception in looking up space point ptr... 0x69c9adb0, 0x69c9b1a0
Caught exception in looking up space point ptr... 0x66002980, 0x6600b4f0
Caught exception in looking up space point ptr... 0x6600b4f0, 0x6600d200
Caught exception in looking up space point ptr... 0x72f57de0, 0x627872f0
Caught exception in looking up space point ptr... 0x72f528a0, 0x72f57de0
Caught exception in looking up space point ptr... 0x57b6aab0, 0x4a0b56e0
Caught exception in looking up space point ptr... 0x4a0b56e0, 0x57b6aab0
Caught exception in looking up space point ptr... 0x421a9760, 0x421b2910
Caught exception in looking up space point ptr... 0x421b2910, 0x421b2a60
Caught exception in looking up space point ptr... 0x7d5cbaf0, 0x7f457770
Caught exception in looking up space point ptr... 0x7f457770, 0x7d5cbaf0
Caught exception in looking up space point ptr... 0x66350310, 0x906c8570
Caught exception in looking up space point ptr... 0x906c8570, 0x6634ba30
Caught exception in looking up space point ptr... 0x8639ef40, 0x5d410710
Caught exception in looking up space point ptr... 0x5d410710, 0x8639ef40
Caught exception in looking up space point ptr... 0x4249ca20, 0x4249c890
Caught exception in looking up space point ptr... 0x4249c890, 0x4249ca20
Caught exception in looking up space point ptr... 0x46977d40, 0x46977bc0
Caught exception in looking up space point ptr... 0x46977bc0, 0x46977c60
Caught exception in looking up space point ptr... 0x7d5d1330, 0x7d5d1010
Caught exception in looking up space point ptr... 0x7d5d1010, 0x7d5d11a0
Caught exception in looking up space point ptr... 0x66357d20, 0x66368ba0
Caught exception in looking up space point ptr... 0x66368ba0, 0x6636b750
Caught exception in looking up space point ptr... 0x6b5bf160, 0x6b5befd0
Caught exception in looking up space point ptr... 0x6b5befd0, 0x6b5bf160
Caught exception in looking up space point ptr... 0x6c2af600, 0x67f0c070
Caught exception in looking up space point ptr... 0x67f0c070, 0x807dbf10
Caught exception in looking up space point ptr... 0x59144520, 0x59143cf0
Caught exception in looking up space point ptr... 0x59143830, 0x59144520
Caught exception in looking up space point ptr... 0x781378b0, 0x78137760
Caught exception in looking up space point ptr... 0x78137760, 0x781378b0
Caught exception in looking up space point ptr... 0x781378b0, 0x781374c0
Caught exception in looking up space point ptr... 0x781374c0, 0x781378b0
Caught exception in looking up space point ptr... 0x4bd2f280, 0x4bd2f130
Caught exception in looking up space point ptr... 0x4bd2f130, 0x4bd2f280
Caught exception in looking up space point ptr... 0x4bd2f280, 0x4bd2ef90
Caught exception in looking up space point ptr... 0x4bd2ef90, 0x4bd2f280
Caught exception in looking up space point ptr... 0x64543eb0, 0x4a5c94a0
Caught exception in looking up space point ptr... 0x4a5c94a0, 0x64543eb0
Caught exception in looking up space point ptr... 0x4af9ce40, 0x4af9cb80
Caught exception in looking up space point ptr... 0x4af9cb80, 0x4af9ce40
Caught exception in looking up space point ptr... 0x4817a800, 0x7d46dfb0
Caught exception in looking up space point ptr... 0x7d46dfb0, 0x4817a800
Caught exception in looking up space point ptr... 0x95b92600, 0x95b98b20
Caught exception in looking up space point ptr... 0x95b98b20, 0x95b905c0
Caught exception in looking up space point ptr... 0x893fe180, 0x65c5f440
Caught exception in looking up space point ptr... 0x65c5f440, 0x4c13ee00
Caught exception in looking up space point ptr... 0x581323b0, 0x731e3c20
Caught exception in looking up space point ptr... 0x731e3c20, 0x6d7133c0
Caught exception in looking up space point ptr... 0x7d4b50c0, 0x7d4b4f30
Caught exception in looking up space point ptr... 0x7d4b4f30, 0x7d4b4a80
Caught exception in looking up space point ptr... 0x7d4b4c10, 0x7d4b4da0
Caught exception in looking up space point ptr... 0x7d4b4da0, 0x7d4b50c0
Caught exception in looking up space point ptr... 0x6241e500, 0x69fc0910
Caught exception in looking up space point ptr... 0x69fc0910, 0x6241e500
Caught exception in looking up space point ptr... 0x7d707e90, 0x7d707cb0
Caught exception in looking up space point ptr... 0x7d707cb0, 0x7d707e90
Caught exception in looking up space point ptr... 0x74278b20, 0x74271ca0
Caught exception in looking up space point ptr... 0x74271ca0, 0x74278b20
Caught exception in looking up space point ptr... 0x918d8380, 0x918d9480
Caught exception in looking up space point ptr... 0x918d9480, 0x6d25a170
++++>>>> total num hits: 59417, num free: 55915
04-Feb-2026 02:21:41 CET  Closed output file "np02vd_raw_run042421_0027_df-s03-d1_dw_0_20260203T134537_reco_stage0_20260204T012141_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         197.821       250.432       325.986       246.218       36.3407        28     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       7.4296e-05    0.000172034   0.000416527   0.000137099   0.000102017      28     
produce:tpcrawdecoder:PDVDTPCReader                96.3176       138.237       197.317       133.536       29.3522        28     
produce:triggerrawdecoder:PDVDTriggerReader4      0.584979      0.814657       1.40028      0.757137       0.20304        28     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0174967     0.0246124     0.0584983     0.021652     0.00887615       28     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0860414     0.101711      0.327658      0.0873282     0.0447886       28     
produce:beamevent:BeamEvent                      0.000111559   0.000300244   0.00160529    0.000280843   0.000259112      28     
produce:pdvddaphne:DAPHNEReaderPDVD                11.5617       18.9736       27.2363       18.6859       4.13839        28     
produce:ophit:OpHitFinder                         0.0303476     0.0384493     0.043446      0.0381164    0.00270407       28     
produce:wclsdatavd:WireCellToolkit                 72.1518       75.6306       83.6889       74.1565       3.5547         28     
produce:gaushit:GausHitFinder                     0.745336       2.81049       17.7404       1.05019       4.1077         28     
produce:nhitsfilter:NumberOfHitsFilter           0.00032459    0.000859701   0.00323632    0.00041612    0.000972763      28     
produce:reco3d:SpacePointSolver                    4.89194       8.49691       13.5436       8.3236        1.71466        23     
produce:hitpdune:DisambigFromSpacePoints          0.0700647     0.106125      0.150184      0.105309      0.020541        23     
produce:cluster3d:Cluster3D                       0.789292       1.50633       3.19728       1.3388       0.498248        23     
[art]:TriggerResults:TriggerResultInserter       2.0054e-05    3.62738e-05   8.4756e-05    2.82585e-05   1.71074e-05      28     
end_path:out1:RootOutput                          4.001e-06    9.39436e-06   2.3361e-05    8.7195e-06    3.29891e-06      28     
end_path:out1:RootOutput(write)                    4.32739       5.46113       7.98344       5.26284      0.978029        28     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4680.86 MB
  Peak resident set size usage (VmHWM): 2760.65 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0027_df-s03-d1_dw_0_20260203T134537_reco_stage0_20260204T012141_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0027_df-s03-d1_dw_0_20260203T134537_reco_stage0_20260204T012141_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0027_df-s03-d1_dw_0_20260203T134537_reco_stage0_20260204T012141_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161022.0,
    "core.end_time": 1770168103.0,
    "core.events": [
      4419,
      4425,
      4431,
      4437,
      4443,
      4449,
      4455,
      4461,
      4467,
      4473,
      4479,
      4485,
      4491,
      4497,
      4503,
      4509,
      4515,
      4521,
      4527,
      4533,
      4539,
      4545,
      4551,
      4557,
      4563,
      4569,
      4575,
      4581
    ],
    "core.event_count": 28,
    "core.first_event_number": 4419,
    "core.last_event_number": 4581,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0027_df-s03-d1_dw_0_20260203T134537.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:47 UTC       justIN version: 01.06.00