justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297916.5@dunegpschedd02.fnal.gov

Jobsub ID297916.5@dunegpschedd02.fnal.gov
Workflow ID12718
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-03 23:27:38
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:34
Input filesvd-protodune:np02vd_raw_run042422_0082_df-s03-d1_dw_0_20260203T155907.hdf5
JobscriptExit code0
Real time37m (2241s)
CPU time35m (2124s = 94%)
Max RSS bytes2637410304 (2515 MiB)
Outputting started2026-02-03 23:24:56
Output files
Finished2026-02-03 23:27:30
Saved logsjustin-logs:297916.5-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

aught exception in looking up space point ptr... 0x2c367600, 0x29252180
Caught exception in looking up space point ptr... 0x35733820, 0x55f86470
Caught exception in looking up space point ptr... 0x55f86470, 0x4fb05650
Caught exception in looking up space point ptr... 0x4fb05650, 0x3676aa90
Caught exception in looking up space point ptr... 0x3676aa90, 0x35733820
Caught exception in looking up space point ptr... 0x63966590, 0x3daf9200
Caught exception in looking up space point ptr... 0x3daf9200, 0x42bda5b0
Caught exception in looking up space point ptr... 0x126ff170, 0x126fdaa0
Caught exception in looking up space point ptr... 0x126fdcc0, 0x126ff170
Caught exception in looking up space point ptr... 0x4cdd6fd0, 0xd914740
Caught exception in looking up space point ptr... 0xd914740, 0x295ce100
Caught exception in looking up space point ptr... 0x20750040, 0x207555e0
Caught exception in looking up space point ptr... 0x207555e0, 0x207550c0
Caught exception in looking up space point ptr... 0x207504c0, 0x20750040
Caught exception in looking up space point ptr... 0x5f8ab100, 0x5f8ab2e0
Caught exception in looking up space point ptr... 0x5f8ab2e0, 0x5f8ab470
Caught exception in looking up space point ptr... 0x2b4c5980, 0x598f0820
Caught exception in looking up space point ptr... 0x598f0820, 0x2b4c5980
Caught exception in looking up space point ptr... 0x46e989b0, 0x46e99840
Caught exception in looking up space point ptr... 0x46e99840, 0x46e99a20
Caught exception in looking up space point ptr... 0x46e99a20, 0x46e98cb0
Caught exception in looking up space point ptr... 0x11652c70, 0xf5e2db0
Caught exception in looking up space point ptr... 0xf5e2db0, 0xf5ed540
Caught exception in looking up space point ptr... 0x598dfa30, 0x598cfae0
Caught exception in looking up space point ptr... 0x598d0420, 0x598dfa30
Caught exception in looking up space point ptr... 0x1163a820, 0x1163a920
Caught exception in looking up space point ptr... 0x1163a920, 0x1163aa20
Caught exception in looking up space point ptr... 0x349686c0, 0x305119f0
Caught exception in looking up space point ptr... 0x305119f0, 0x30511bf0
Caught exception in looking up space point ptr... 0x416b5f20, 0x416b73d0
Caught exception in looking up space point ptr... 0x416b73d0, 0x416b74c0
Caught exception in looking up space point ptr... 0x42798e50, 0x42798fa0
Caught exception in looking up space point ptr... 0x42798fa0, 0x4279e5a0
Caught exception in looking up space point ptr... 0x31257aa0, 0x3125eea0
Caught exception in looking up space point ptr... 0x3125eea0, 0x1270bc80
Caught exception in looking up space point ptr... 0x53130260, 0x5312f8b0
Caught exception in looking up space point ptr... 0x5312f8b0, 0x5312d9b0
Caught exception in looking up space point ptr... 0x1270c850, 0x1270c410
Caught exception in looking up space point ptr... 0x1270c410, 0x1270c850
Caught exception in looking up space point ptr... 0x44a65e90, 0x43ca1600
Caught exception in looking up space point ptr... 0x4f7a3e10, 0x44a65e90
Caught exception in looking up space point ptr... 0x502213b0, 0x581e5d00
Caught exception in looking up space point ptr... 0x3092bc80, 0x502213b0
Caught exception in looking up space point ptr... 0x3b9fcd20, 0x4da6afe0
Caught exception in looking up space point ptr... 0x4da6afe0, 0x4da6aaf0
Caught exception in looking up space point ptr... 0x416b65c0, 0x416af010
Caught exception in looking up space point ptr... 0x416af010, 0x416b3f90
Caught exception in looking up space point ptr... 0x13681b20, 0x4ac8cef0
Caught exception in looking up space point ptr... 0x4ac8cef0, 0x4ac8cdf0
Caught exception in looking up space point ptr... 0x3d1ba660, 0x2799d8a0
Caught exception in looking up space point ptr... 0x3c906d40, 0x3d1ba660
Caught exception in looking up space point ptr... 0x31260670, 0x3125e440
Caught exception in looking up space point ptr... 0x3125e440, 0x31260670
Caught exception in looking up space point ptr... 0x45af3580, 0x45af31f0
Caught exception in looking up space point ptr... 0x45af31f0, 0x45af3320
Caught exception in looking up space point ptr... 0x46e8d680, 0x46e8a910
Caught exception in looking up space point ptr... 0x46e8df70, 0x46e8d680
Caught exception in looking up space point ptr... 0x2074f3b0, 0x2074f210
Caught exception in looking up space point ptr... 0x2074f210, 0x2074f6d0
Caught exception in looking up space point ptr... 0x295ce2d0, 0x365223b0
Caught exception in looking up space point ptr... 0x365223b0, 0x63961ff0
Caught exception in looking up space point ptr... 0x419534d0, 0x334aa070
Caught exception in looking up space point ptr... 0x334aa070, 0x419534d0
Caught exception in looking up space point ptr... 0x344cbe90, 0x344ba9d0
Caught exception in looking up space point ptr... 0x344ba9d0, 0x344cbe90
++++>>>> total num hits: 61576, num free: 59568
04-Feb-2026 00:24:43 CET  Closed output file "np02vd_raw_run042422_0082_df-s03-d1_dw_0_20260203T155907_reco_stage0_20260203T232443_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         68.6391       80.7113       97.3024       78.5104       7.95309        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.0161e-05    8.98142e-05   0.000226032    7.832e-05    4.21033e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.84685       4.51204       5.86946       4.43631      0.361765        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0109009     0.0266767     0.0777463     0.0205249     0.0175113       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000416832   0.00135726    0.00984268    0.000598923   0.00210716       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00156771    0.00230901    0.00376143    0.00217547    0.000528075      27     
produce:beamevent:BeamEvent                      5.7441e-05    0.000125883   0.000963255    9.17e-05     0.000165774      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.2724        2.80336       3.28714       2.78176      0.256357        27     
produce:ophit:OpHitFinder                          0.02117      0.0363055     0.0497125     0.0371031    0.00719738       27     
produce:wclsdatavd:WireCellToolkit                 51.8149       59.9505       72.7084       58.2623       6.0526         27     
produce:gaushit:GausHitFinder                     0.514185      0.864743       1.4707       0.780892      0.273211        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000142901   0.000232955   0.000454772   0.000213521   6.93483e-05      27     
produce:reco3d:SpacePointSolver                    4.13946       7.55841       12.1917       7.73006       1.97362        27     
produce:hitpdune:DisambigFromSpacePoints          0.045596      0.0799085     0.163395      0.0763566     0.027689        27     
produce:cluster3d:Cluster3D                       0.599697       1.48768       3.69242       1.35283       0.6562         27     
[art]:TriggerResults:TriggerResultInserter        1.709e-05    3.35391e-05   0.00010483    2.9531e-05    1.66647e-05      27     
end_path:out1:RootOutput                            3e-06      7.32963e-06    2.664e-05     6.75e-06     4.88376e-06      27     
end_path:out1:RootOutput(write)                    2.73912       3.36882       4.22071       3.27039      0.440892        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4567.96 MB
  Peak resident set size usage (VmHWM): 2637.41 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042422_0082_df-s03-d1_dw_0_20260203T155907_reco_stage0_20260203T232443_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042422_0082_df-s03-d1_dw_0_20260203T155907_reco_stage0_20260203T232443_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042422_0082_df-s03-d1_dw_0_20260203T155907_reco_stage0_20260203T232443_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158864.0,
    "core.end_time": 1770161085.0,
    "core.events": [
      13299,
      13305,
      13311,
      13317,
      13323,
      13329,
      13335,
      13341,
      13347,
      13353,
      13359,
      13365,
      13371,
      13377,
      13383,
      13389,
      13395,
      13401,
      13407,
      13413,
      13419,
      13425,
      13431,
      13437,
      13443,
      13449,
      13455
    ],
    "core.event_count": 27,
    "core.first_event_number": 13299,
    "core.last_event_number": 13455,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42422
    ],
    "core.runs_subruns": [
      4242200001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042422_0082_df-s03-d1_dw_0_20260203T155907.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:24 UTC       justIN version: 01.06.00