justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.126@dunegpschedd01.fnal.gov

Jobsub ID303652.126@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 01:58:13
From worker nodeHostnamewn-pijl-017.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:28:59
Input filesvd-protodune:np02vd_raw_run042433_0320_df-s03-d0_dw_0_20260203T204344.hdf5
JobscriptExit code0
Real time2h (8758s)
CPU time1h (4744s = 54%)
Max RSS bytes3594833920 (3428 MiB)
Outputting started2026-02-04 01:55:09
Output files
Finished2026-02-04 01:58:13
Saved logsjustin-logs:303652.126-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x9e2c7260, 0xb6aebfd0
Caught exception in looking up space point ptr... 0x84cfbf30, 0x649326f0
Caught exception in looking up space point ptr... 0x649326f0, 0x64932880
Caught exception in looking up space point ptr... 0x64932880, 0x70b92620
Caught exception in looking up space point ptr... 0x8326d850, 0x721e3790
Caught exception in looking up space point ptr... 0x721e3790, 0x4f5a4870
Caught exception in looking up space point ptr... 0x4a4876f0, 0x7862b500
Caught exception in looking up space point ptr... 0x7862b500, 0x7862b370
Caught exception in looking up space point ptr... 0x9fbe9f90, 0xa5025820
Caught exception in looking up space point ptr... 0xa5025150, 0x9fbe9f90
Caught exception in looking up space point ptr... 0x8b96ec70, 0x8b978780
Caught exception in looking up space point ptr... 0x8b978780, 0x8b9784f0
Caught exception in looking up space point ptr... 0x7a705300, 0x8f799e10
Caught exception in looking up space point ptr... 0x8f799e10, 0xa7bc6f70
Caught exception in looking up space point ptr... 0xa7bc6f70, 0x8f81ea80
Caught exception in looking up space point ptr... 0x98cac0c0, 0x98c7f010
Caught exception in looking up space point ptr... 0x98cac250, 0x98cac0c0
Caught exception in looking up space point ptr... 0x50275f00, 0x8ecfc2c0
Caught exception in looking up space point ptr... 0x50280870, 0x50275f00
Caught exception in looking up space point ptr... 0x6dafc710, 0x6dafd070
Caught exception in looking up space point ptr... 0x9baf6e60, 0x6dafc710
Caught exception in looking up space point ptr... 0x437c1990, 0x437af600
Caught exception in looking up space point ptr... 0x437c1ae0, 0x437c1990
Caught exception in looking up space point ptr... 0x86208c70, 0xbc45c7d0
Caught exception in looking up space point ptr... 0x86208e00, 0x86208c70
Caught exception in looking up space point ptr... 0x680c8fc0, 0x680c31a0
Caught exception in looking up space point ptr... 0x680c31a0, 0x4cf39240
Caught exception in looking up space point ptr... 0x861adf80, 0x89079dd0
Caught exception in looking up space point ptr... 0x861ae110, 0x861adf80
Caught exception in looking up space point ptr... 0x63aefb00, 0x63ada320
Caught exception in looking up space point ptr... 0x63aefc90, 0x63aefb00
Caught exception in looking up space point ptr... 0x49605c60, 0xb0588b50
Caught exception in looking up space point ptr... 0x49605df0, 0x49605c60
Caught exception in looking up space point ptr... 0xb11ae6d0, 0x6ebf7cc0
Caught exception in looking up space point ptr... 0x6ebf7cc0, 0x9ab166e0
Caught exception in looking up space point ptr... 0x695c25e0, 0x695d1040
Caught exception in looking up space point ptr... 0x695d1040, 0x82ace660
Caught exception in looking up space point ptr... 0x47e44540, 0x47e33880
Caught exception in looking up space point ptr... 0x47e33880, 0x47e22bc0
Caught exception in looking up space point ptr... 0xa5083030, 0x48b6a1f0
Caught exception in looking up space point ptr... 0x48b4b1d0, 0xa5083030
Caught exception in looking up space point ptr... 0x9100b0f0, 0x4cad6810
Caught exception in looking up space point ptr... 0x4cad6810, 0x80be8db0
Caught exception in looking up space point ptr... 0x80be8db0, 0x7a3ca790
Caught exception in looking up space point ptr... 0x7a3ca790, 0x9100b0f0
Caught exception in looking up space point ptr... 0x62676360, 0x62676190
Caught exception in looking up space point ptr... 0x4a64b700, 0x62676360
Caught exception in looking up space point ptr... 0x6954efc0, 0x7402c3f0
Caught exception in looking up space point ptr... 0x7402c3f0, 0x7402d230
Caught exception in looking up space point ptr... 0xa5061120, 0xa506d490
Caught exception in looking up space point ptr... 0xa506d490, 0xa5061120
Caught exception in looking up space point ptr... 0x4df1e130, 0x4df1dfa0
Caught exception in looking up space point ptr... 0x4df1dfa0, 0x6f5c6770
Caught exception in looking up space point ptr... 0x6f5c6770, 0x6f5c6670
Caught exception in looking up space point ptr... 0x6f5c6570, 0x4df1e130
Caught exception in looking up space point ptr... 0x4b4cdb90, 0x884ae000
Caught exception in looking up space point ptr... 0x884ae000, 0x4b4cdd20
Caught exception in looking up space point ptr... 0x628e5380, 0x628e4e90
Caught exception in looking up space point ptr... 0x628e4e90, 0x8bb50b10
Caught exception in looking up space point ptr... 0x97854b20, 0x628e5130
Caught exception in looking up space point ptr... 0x628e5130, 0x628e5380
Caught exception in looking up space point ptr... 0x712ff010, 0x71301920
Caught exception in looking up space point ptr... 0x7130e060, 0x712ff010
Caught exception in looking up space point ptr... 0x40dc5f60, 0x40dc6000
Caught exception in looking up space point ptr... 0x40dc6000, 0x40dc5f60
Caught exception in looking up space point ptr... 0x7107d710, 0x7107d580
Caught exception in looking up space point ptr... 0x7107d580, 0xb1161f00
++++>>>> total num hits: 528666, num free: 519069
04-Feb-2026 02:54:40 CET  Closed output file "np02vd_raw_run042433_0320_df-s03-d0_dw_0_20260203T204344_reco_stage0_20260204T015440_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         350.043       454.826       527.618       457.852       44.9502        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.8032e-05    0.000109439   0.000289788   9.2771e-05    4.62232e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                 146.8        200.643       294.885       197.05        40.2033        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.70619       1.14098       4.12565       1.00991      0.721706        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0176509     0.026742      0.0718979     0.0225324     0.013472        19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0175973     0.0217789     0.0430508     0.0178646    0.00911509       19     
produce:beamevent:BeamEvent                      0.000116145   0.000198722   0.00108588    0.000153993   0.000210085      19     
produce:pdvddaphne:DAPHNEReaderPDVD                13.0838       17.5775       23.2504       17.0068       2.87786        19     
produce:ophit:OpHitFinder                         0.0321112     0.0479461     0.0549446     0.0480258    0.00510819       19     
produce:wclsdatavd:WireCellToolkit                 118.02        139.355       158.622       138.688       9.71743        19     
produce:gaushit:GausHitFinder                      1.09398       2.71588       3.77253       2.70927      0.627806        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000266523   0.000721599   0.00130257    0.000683473   0.000301479      19     
produce:reco3d:SpacePointSolver                    11.3489       16.6077       22.0487       17.1247       2.70333        19     
produce:hitpdune:DisambigFromSpacePoints          0.250489      0.348878       0.54505      0.313622      0.089149        19     
produce:cluster3d:Cluster3D                        2.67731       67.2166       80.147        68.4112       16.523         19     
[art]:TriggerResults:TriggerResultInserter       3.0937e-05    4.61381e-05   0.000113151    4.007e-05    1.83346e-05      19     
end_path:out1:RootOutput                          4.727e-06    8.35416e-06   2.9164e-05     6.619e-06    5.49659e-06      19     
end_path:out1:RootOutput(write)                    6.66093       9.05044       10.0921       9.43213      0.933086        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5542.52 MB
  Peak resident set size usage (VmHWM): 3594.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0320_df-s03-d0_dw_0_20260203T204344_reco_stage0_20260204T015440_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0320_df-s03-d0_dw_0_20260203T204344_reco_stage0_20260204T015440_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0320_df-s03-d0_dw_0_20260203T204344_reco_stage0_20260204T015440_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161351.0,
    "core.end_time": 1770170084.0,
    "core.events": [
      36488,
      36494,
      36500,
      36506,
      36512,
      36518,
      36524,
      36530,
      36536,
      36542,
      36548,
      36554,
      36560,
      36566,
      36572,
      36578,
      36584,
      36590,
      36596
    ],
    "core.event_count": 19,
    "core.first_event_number": 36488,
    "core.last_event_number": 36596,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0320_df-s03-d0_dw_0_20260203T204344.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:03 UTC       justIN version: 01.06.00