justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.91@dunegpschedd01.fnal.gov

Jobsub ID303653.91@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:13:26
From worker nodeHostnameb9p35p0023.cern.ch
cpuinfoAMD EPYC 9655P 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:41:09
Input filesvd-protodune:np02vd_raw_run042421_0037_df-s02-d1_dw_0_20260203T135101.hdf5
JobscriptExit code0
Real time30m (1813s)
CPU time28m (1717s = 94%)
Max RSS bytes2628698112 (2506 MiB)
Outputting started2026-02-04 00:11:23
Output files
Finished2026-02-04 00:13:26
Saved logsjustin-logs:303653.91-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

king up space point ptr... 0x5a5d5ab0, 0x5a5db650
Caught exception in looking up space point ptr... 0x5a5db650, 0x5a5d5d10
Caught exception in looking up space point ptr... 0x5a5d5d10, 0x5a5d2c30
Caught exception in looking up space point ptr... 0x5a5d2c30, 0x5a5d5ab0
Caught exception in looking up space point ptr... 0x2215a9b0, 0x81cac80
Caught exception in looking up space point ptr... 0x281794d0, 0x2215a9b0
Caught exception in looking up space point ptr... 0x42ac5720, 0x42ac4300
Caught exception in looking up space point ptr... 0x42abef80, 0x42ac5720
Caught exception in looking up space point ptr... 0x215eaa40, 0xcaa3820
Caught exception in looking up space point ptr... 0xcaa3820, 0x215eaa40
Caught exception in looking up space point ptr... 0x64b303d0, 0x64b2fcf0
Caught exception in looking up space point ptr... 0x64b2fcf0, 0x64b303d0
Caught exception in looking up space point ptr... 0x2bd38290, 0x2bd37040
Caught exception in looking up space point ptr... 0x2bd37040, 0x2bd386c0
Caught exception in looking up space point ptr... 0x64eb4350, 0x2ed79740
Caught exception in looking up space point ptr... 0x41366f20, 0x4ca01a60
Caught exception in looking up space point ptr... 0x4ca01a60, 0x64eb4350
Caught exception in looking up space point ptr... 0x408150a0, 0x2b0d09f0
Caught exception in looking up space point ptr... 0x2b0d09f0, 0x408150a0
Caught exception in looking up space point ptr... 0x99c5ca0, 0x99c5e30
Caught exception in looking up space point ptr... 0x99c5e30, 0x99c5ca0
Caught exception in looking up space point ptr... 0x55b63ef0, 0x42818490
Caught exception in looking up space point ptr... 0x42818490, 0x21df39b0
Caught exception in looking up space point ptr... 0x21df39b0, 0x7f43d60
Caught exception in looking up space point ptr... 0x7f43d60, 0x35ba94e0
Caught exception in looking up space point ptr... 0x35ba94e0, 0x55b63ef0
Caught exception in looking up space point ptr... 0x2d429750, 0x2d42aae0
Caught exception in looking up space point ptr... 0x2d41e090, 0x2d429750
Caught exception in looking up space point ptr... 0x1137bfe0, 0x2d435970
Caught exception in looking up space point ptr... 0x2eec9570, 0x1137bfe0
Caught exception in looking up space point ptr... 0x3a7fa810, 0x3a808030
Caught exception in looking up space point ptr... 0x3a808030, 0x3a7fa810
Caught exception in looking up space point ptr... 0x41b8f240, 0x41b8f0b0
Caught exception in looking up space point ptr... 0x41b8f0b0, 0x41b8f240
Caught exception in looking up space point ptr... 0x410618f0, 0x41061a80
Caught exception in looking up space point ptr... 0x41061a80, 0x410618f0
Caught exception in looking up space point ptr... 0x2bcbd820, 0x2bcbd370
Caught exception in looking up space point ptr... 0x2bcbd370, 0x2bcbd820
Caught exception in looking up space point ptr... 0xdfe4db0, 0x107883b0
Caught exception in looking up space point ptr... 0xdfe4c80, 0xdfe4db0
Caught exception in looking up space point ptr... 0x2bcb4f00, 0x38cf7470
Caught exception in looking up space point ptr... 0x38cf7470, 0x2bcb4f00
Caught exception in looking up space point ptr... 0x15241940, 0x2d2ef240
Caught exception in looking up space point ptr... 0x2d2ef240, 0x15241940
Caught exception in looking up space point ptr... 0x99c7a40, 0x99c7d60
Caught exception in looking up space point ptr... 0x99c7d60, 0x99c7a40
Caught exception in looking up space point ptr... 0x87e8bf0, 0x87e88d0
Caught exception in looking up space point ptr... 0x87e88d0, 0x87e8bf0
Caught exception in looking up space point ptr... 0x976ac70, 0x2caa3bc0
Caught exception in looking up space point ptr... 0x2caa3bc0, 0x976ac70
Caught exception in looking up space point ptr... 0x2bd49630, 0x2bd49500
Caught exception in looking up space point ptr... 0x2bd49500, 0x2bd49630
Caught exception in looking up space point ptr... 0x99e0100, 0x99dffb0
Caught exception in looking up space point ptr... 0x99dffb0, 0x99dfd60
Caught exception in looking up space point ptr... 0x42ac4c50, 0x11155da0
Caught exception in looking up space point ptr... 0x11155da0, 0x42ac4c50
Caught exception in looking up space point ptr... 0x38d05220, 0x38d0cc60
Caught exception in looking up space point ptr... 0x38d0cc60, 0x38d05220
Caught exception in looking up space point ptr... 0x43d41bd0, 0x2e1a8d10
Caught exception in looking up space point ptr... 0x2e1a8d10, 0x43d41bd0
Caught exception in looking up space point ptr... 0x40d63c50, 0x40d63b00
Caught exception in looking up space point ptr... 0x40d63b00, 0x40d63c50
Caught exception in looking up space point ptr... 0x1041d6e0, 0x1040ebe0
Caught exception in looking up space point ptr... 0x1041f600, 0x1041d6e0
Caught exception in looking up space point ptr... 0x32c89e30, 0x377a9750
Caught exception in looking up space point ptr... 0x377a9750, 0x32c89e30
++++>>>> total num hits: 74370, num free: 72041
04-Feb-2026 01:11:17 CET  Closed output file "np02vd_raw_run042421_0037_df-s02-d1_dw_0_20260203T135101_reco_stage0_20260204T001117_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         54.8852       65.4829       75.3223       65.4045       4.76264        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.857e-05    7.14139e-05   0.00018944    6.7271e-05    2.43139e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.06348       4.6406        9.44976       4.45083      0.963423        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0070032     0.0196253     0.149068     0.00916236     0.0275403       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000293621   0.000439415   0.000731952   0.000435441   8.78653e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00101428    0.00128162    0.00161013    0.00127568    0.000160239      27     
produce:beamevent:BeamEvent                       5.997e-05    0.000101848   0.000735302   7.8721e-05    0.000124621      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.53936       2.01045       2.26168       2.04929      0.186384        27     
produce:ophit:OpHitFinder                         0.0200842     0.0322013     0.0399264     0.0324603    0.00422424       27     
produce:wclsdatavd:WireCellToolkit                 40.1602       45.574        52.311        45.4548       2.73512        27     
produce:gaushit:GausHitFinder                     0.418292      0.702081       1.0212       0.693946      0.160201        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000108051   0.000160049   0.000378731   0.000151721   4.90169e-05      27     
produce:reco3d:SpacePointSolver                    4.45468       7.76405       11.4045       7.34451       1.97357        27     
produce:hitpdune:DisambigFromSpacePoints          0.0377274     0.0696189     0.123921      0.0602229     0.0260497       27     
produce:cluster3d:Cluster3D                       0.574743       1.17221       1.99643       1.12468      0.416088        27     
[art]:TriggerResults:TriggerResultInserter        2.078e-05    2.82538e-05    6.922e-05     2.679e-05    8.76514e-06      27     
end_path:out1:RootOutput                          4.34e-06     5.76078e-06    1.736e-05     5.28e-06     2.32208e-06      27     
end_path:out1:RootOutput(write)                    2.87691       3.4748        4.34379       3.37808      0.394884        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4648.67 MB
  Peak resident set size usage (VmHWM): 2628.7 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0037_df-s02-d1_dw_0_20260203T135101_reco_stage0_20260204T001117_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0037_df-s02-d1_dw_0_20260203T135101_reco_stage0_20260204T001117_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0037_df-s02-d1_dw_0_20260203T135101_reco_stage0_20260204T001117_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162075.0,
    "core.end_time": 1770163878.0,
    "core.events": [
      6037,
      6043,
      6049,
      6055,
      6061,
      6067,
      6073,
      6079,
      6085,
      6091,
      6097,
      6103,
      6109,
      6115,
      6121,
      6127,
      6133,
      6139,
      6145,
      6151,
      6157,
      6163,
      6169,
      6175,
      6181,
      6187,
      6193
    ],
    "core.event_count": 27,
    "core.first_event_number": 6037,
    "core.last_event_number": 6193,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0037_df-s02-d1_dw_0_20260203T135101.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:24 UTC       justIN version: 01.06.00