justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.156@dunegpschedd01.fnal.gov

Jobsub ID303647.156@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-03 23:24:13
From worker nodeHostnameb9p10p5730.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:37
Input filesvd-protodune:np02vd_raw_run042421_0100_df-s03-d1_dw_0_20260203T142506.hdf5
JobscriptExit code0
Real time34m (2063s)
CPU time33m (1990s = 96%)
Max RSS bytes2637529088 (2515 MiB)
Outputting started2026-02-03 23:22:01
Output files
Finished2026-02-03 23:24:13
Saved logsjustin-logs:303647.156-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

450, 0x5b3d01d0
Caught exception in looking up space point ptr... 0x5b3d01d0, 0x2a5839b0
Caught exception in looking up space point ptr... 0x16c8c200, 0x16c8c100
Caught exception in looking up space point ptr... 0x16c8c100, 0x16c92670
Caught exception in looking up space point ptr... 0x2b260000, 0x2b25f9c0
Caught exception in looking up space point ptr... 0x2b25f9c0, 0x2b25fce0
Caught exception in looking up space point ptr... 0x5cbbece0, 0x3d2eeeb0
Caught exception in looking up space point ptr... 0x3d2eeeb0, 0x5cbbece0
Caught exception in looking up space point ptr... 0x355fbb30, 0x355fa520
Caught exception in looking up space point ptr... 0x355fa520, 0x355fbb30
Caught exception in looking up space point ptr... 0x355fbb30, 0x355f8150
Caught exception in looking up space point ptr... 0x355f8150, 0x355fbb30
Caught exception in looking up space point ptr... 0x94d1d50, 0x14ba8a10
Caught exception in looking up space point ptr... 0x94ca520, 0x94d1d50
Caught exception in looking up space point ptr... 0x937f4c0, 0x58767370
Caught exception in looking up space point ptr... 0x58767370, 0x937f4c0
Caught exception in looking up space point ptr... 0x14b1f140, 0x14b214d0
Caught exception in looking up space point ptr... 0x14b214d0, 0x14b1eda0
Caught exception in looking up space point ptr... 0x5b0ce380, 0x355f06a0
Caught exception in looking up space point ptr... 0x355f06a0, 0x355f0830
Caught exception in looking up space point ptr... 0x16b65ba0, 0x16b65880
Caught exception in looking up space point ptr... 0x16b65880, 0x16b656a0
Caught exception in looking up space point ptr... 0x40d9c920, 0x2f121110
Caught exception in looking up space point ptr... 0x2f121110, 0x40d9c920
Caught exception in looking up space point ptr... 0x40d9c920, 0x2f121230
Caught exception in looking up space point ptr... 0x2f121230, 0x40d9c920
Caught exception in looking up space point ptr... 0x40d9c920, 0x57ce34d0
Caught exception in looking up space point ptr... 0x57ce34d0, 0x40d9c920
Caught exception in looking up space point ptr... 0x2905bfa0, 0x16c8d680
Caught exception in looking up space point ptr... 0x16c8d680, 0xa3a5f70
Caught exception in looking up space point ptr... 0x50de5410, 0x50de1430
Caught exception in looking up space point ptr... 0x50de1430, 0x50de6590
Caught exception in looking up space point ptr... 0x45308ce0, 0x45308b40
Caught exception in looking up space point ptr... 0x45308b40, 0x45308ce0
Caught exception in looking up space point ptr... 0xfb92330, 0x1020a810
Caught exception in looking up space point ptr... 0x1020a810, 0xfb92330
Caught exception in looking up space point ptr... 0x372647f0, 0x372a1920
Caught exception in looking up space point ptr... 0x372a1920, 0x372647f0
Caught exception in looking up space point ptr... 0x2906e5a0, 0x16c6fbe0
Caught exception in looking up space point ptr... 0x16c6fbe0, 0x2906e5a0
Caught exception in looking up space point ptr... 0x11902720, 0x11902820
Caught exception in looking up space point ptr... 0x11902820, 0x11902720
Caught exception in looking up space point ptr... 0x5edcc6c0, 0x4a55b660
Caught exception in looking up space point ptr... 0x4a55b660, 0x2e57e820
Caught exception in looking up space point ptr... 0xfeb3590, 0xfeb30e0
Caught exception in looking up space point ptr... 0xfeb30e0, 0xfeb3270
Caught exception in looking up space point ptr... 0xad6a4f0, 0xad6ae50
Caught exception in looking up space point ptr... 0xad6a5d0, 0xad6a4f0
Caught exception in looking up space point ptr... 0x492b4a70, 0x492b4940
Caught exception in looking up space point ptr... 0x492b4940, 0x492b4a70
Caught exception in looking up space point ptr... 0x50cfae40, 0x96fd580
Caught exception in looking up space point ptr... 0x96fd580, 0x50cfae40
Caught exception in looking up space point ptr... 0x2b5a8aa0, 0x578bc460
Caught exception in looking up space point ptr... 0x578bc460, 0x578bc640
Caught exception in looking up space point ptr... 0x299fa950, 0x45e64f50
Caught exception in looking up space point ptr... 0x45e64f50, 0x299fa950
Caught exception in looking up space point ptr... 0x708c3c00, 0x3dfd40e0
Caught exception in looking up space point ptr... 0x3dfd40e0, 0x708c3c00
Caught exception in looking up space point ptr... 0xe2ff110, 0xe2ff430
Caught exception in looking up space point ptr... 0xe2ff430, 0xe2ff110
Caught exception in looking up space point ptr... 0x14806840, 0x147f9930
Caught exception in looking up space point ptr... 0x147f9930, 0x147f9f20
Caught exception in looking up space point ptr... 0x25d267a0, 0x31c937d0
Caught exception in looking up space point ptr... 0x31c937d0, 0x25d268e0
Caught exception in looking up space point ptr... 0x14c49070, 0x4e7e9110
Caught exception in looking up space point ptr... 0x4e7e9110, 0x14c49070
++++>>>> total num hits: 81174, num free: 76667
04-Feb-2026 00:21:54 CET  Closed output file "np02vd_raw_run042421_0100_df-s03-d1_dw_0_20260203T142506_reco_stage0_20260203T232154_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         66.8431       74.273        81.6698       72.8603       3.99177        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.294e-05    9.75549e-05   0.000478965   6.7691e-05    8.87558e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.05221       3.76594       6.0816        3.65009      0.517455        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0102167     0.0171005     0.046676      0.0143605    0.00721234       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000387565   0.000522986   0.000918292   0.000470766   0.00014198       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00143693    0.00193965    0.00312896    0.00183873    0.000351063      27     
produce:beamevent:BeamEvent                      5.1621e-05    0.000123139   0.00115381    7.9211e-05     0.0002044       27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.09723       2.56111       2.99714       2.59898      0.259322        27     
produce:ophit:OpHitFinder                         0.0237743     0.0355085     0.0479095     0.0344504    0.00544954       27     
produce:wclsdatavd:WireCellToolkit                 47.5957       53.2086       61.3941       52.0245       4.20535        27     
produce:gaushit:GausHitFinder                      0.57883       0.80215       1.13631      0.808217      0.142027        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000136862   0.000232202   0.000527827   0.000214192   7.22964e-05      27     
produce:reco3d:SpacePointSolver                    4.84197       8.9567        13.626        8.8886        2.0984         27     
produce:hitpdune:DisambigFromSpacePoints          0.0517203     0.0810216      0.14853      0.0763261     0.0232318       27     
produce:cluster3d:Cluster3D                       0.948109       1.54027       2.57219       1.44782      0.395685        27     
[art]:TriggerResults:TriggerResultInserter        1.876e-05    3.12608e-05   0.000157292    2.573e-05    2.52334e-05      27     
end_path:out1:RootOutput                          3.53e-06     8.31037e-06    3.722e-05     5.32e-06     7.18977e-06      27     
end_path:out1:RootOutput(write)                    2.79844       3.28309       4.06207       3.11586      0.365364        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4600.92 MB
  Peak resident set size usage (VmHWM): 2637.53 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0100_df-s03-d1_dw_0_20260203T142506_reco_stage0_20260203T232154_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0100_df-s03-d1_dw_0_20260203T142506_reco_stage0_20260203T232154_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0100_df-s03-d1_dw_0_20260203T142506_reco_stage0_20260203T232154_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158869.0,
    "core.end_time": 1770160916.0,
    "core.events": [
      16269,
      16275,
      16281,
      16287,
      16293,
      16299,
      16305,
      16311,
      16317,
      16323,
      16329,
      16335,
      16341,
      16347,
      16353,
      16359,
      16365,
      16371,
      16377,
      16383,
      16389,
      16395,
      16401,
      16407,
      16413,
      16419,
      16425
    ],
    "core.event_count": 27,
    "core.first_event_number": 16269,
    "core.last_event_number": 16425,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0100_df-s03-d1_dw_0_20260203T142506.hdf5"
    }
  ]
}
justIN time: 2026-02-04 12:05:35 UTC       justIN version: 01.06.00