justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.97@dunegpschedd01.fnal.gov

Jobsub ID303647.97@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-03 23:28:58
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:30
Input filesvd-protodune:np02vd_raw_run042421_0041_df-s02-d1_dw_0_20260203T135310.hdf5
JobscriptExit code0
Real time38m (2337s)
CPU time37m (2237s = 95%)
Max RSS bytes2687135744 (2562 MiB)
Outputting started2026-02-03 23:26:28
Output files
Finished2026-02-03 23:28:58
Saved logsjustin-logs:303647.97-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ing up space point ptr... 0xe6eebd0, 0xe6f0cb0
Caught exception in looking up space point ptr... 0x8da94f0, 0x3a526720
Caught exception in looking up space point ptr... 0x3a526720, 0x401361b0
Caught exception in looking up space point ptr... 0x15545d40, 0x8da94f0
Caught exception in looking up space point ptr... 0x3702a280, 0x3702a150
Caught exception in looking up space point ptr... 0x3702c3f0, 0x3702a280
Caught exception in looking up space point ptr... 0x5e8b3cd0, 0x330afb00
Caught exception in looking up space point ptr... 0x330afb00, 0x103dc530
Caught exception in looking up space point ptr... 0x3697c670, 0x3697e8e0
Caught exception in looking up space point ptr... 0x3697e8e0, 0x60a3bb90
Caught exception in looking up space point ptr... 0xec20d80, 0x48ed7320
Caught exception in looking up space point ptr... 0x48ed7320, 0x48ef0c40
Caught exception in looking up space point ptr... 0x2a895010, 0x489e8990
Caught exception in looking up space point ptr... 0x489e8990, 0x2a895010
Caught exception in looking up space point ptr... 0x2a895010, 0x372e03c0
Caught exception in looking up space point ptr... 0x372e03c0, 0x2a895010
Caught exception in looking up space point ptr... 0x63ca5b70, 0x4fed1e30
Caught exception in looking up space point ptr... 0x4fed1e30, 0x11dcde90
Caught exception in looking up space point ptr... 0x584f4330, 0x206b1c00
Caught exception in looking up space point ptr... 0x584f40d0, 0x584f4330
Caught exception in looking up space point ptr... 0x1518e100, 0x9037240
Caught exception in looking up space point ptr... 0x1518df60, 0x1518e100
Caught exception in looking up space point ptr... 0x3434c000, 0x8e07890
Caught exception in looking up space point ptr... 0x8e07890, 0x2a64eac0
Caught exception in looking up space point ptr... 0x95729f0, 0x956b9c0
Caught exception in looking up space point ptr... 0x956b9c0, 0x956d890
Caught exception in looking up space point ptr... 0x256ab820, 0x256ab500
Caught exception in looking up space point ptr... 0x256ab500, 0x256b09c0
Caught exception in looking up space point ptr... 0x256b09c0, 0x256b0e70
Caught exception in looking up space point ptr... 0x256b0e70, 0x256ab820
Caught exception in looking up space point ptr... 0x15192580, 0x15192760
Caught exception in looking up space point ptr... 0x15192760, 0x4c027b50
Caught exception in looking up space point ptr... 0x335b5880, 0x335b3d50
Caught exception in looking up space point ptr... 0x335b56f0, 0x335b5880
Caught exception in looking up space point ptr... 0x151dd3b0, 0x22a19760
Caught exception in looking up space point ptr... 0x22a19760, 0x4080cea0
Caught exception in looking up space point ptr... 0x4963af10, 0x10482230
Caught exception in looking up space point ptr... 0x10482230, 0x4963af10
Caught exception in looking up space point ptr... 0x9760580, 0x9760650
Caught exception in looking up space point ptr... 0x9760650, 0x9760580
Caught exception in looking up space point ptr... 0x4aba4750, 0x6cdd2010
Caught exception in looking up space point ptr... 0x6cdd2010, 0x27431920
Caught exception in looking up space point ptr... 0x5eba02a0, 0x14e5fd90
Caught exception in looking up space point ptr... 0x4f582fc0, 0x5eba02a0
Caught exception in looking up space point ptr... 0x22a0b110, 0xa49fff0
Caught exception in looking up space point ptr... 0xa49fff0, 0x22a0b110
Caught exception in looking up space point ptr... 0x48676580, 0x48676680
Caught exception in looking up space point ptr... 0x48676680, 0x48676580
Caught exception in looking up space point ptr... 0x12aa5c10, 0x12aa5970
Caught exception in looking up space point ptr... 0x12aa5970, 0x12aa4eb0
Caught exception in looking up space point ptr... 0x117f3710, 0x117f38a0
Caught exception in looking up space point ptr... 0x117f38a0, 0x117f3710
Caught exception in looking up space point ptr... 0x61010150, 0x61009ab0
Caught exception in looking up space point ptr... 0x61009ab0, 0x61010150
Caught exception in looking up space point ptr... 0x5a32d490, 0x5a32d2b0
Caught exception in looking up space point ptr... 0x5a32d2b0, 0x5a32d490
Caught exception in looking up space point ptr... 0x1109fff0, 0x110a0150
Caught exception in looking up space point ptr... 0x10433370, 0x1109fff0
Caught exception in looking up space point ptr... 0x4fee4290, 0x4fee9df0
Caught exception in looking up space point ptr... 0x4fee9df0, 0x4fee4290
Caught exception in looking up space point ptr... 0x3034d310, 0x3034d800
Caught exception in looking up space point ptr... 0x3034d800, 0x3034d310
Caught exception in looking up space point ptr... 0xa4d2bc0, 0xa4d2970
Caught exception in looking up space point ptr... 0xa4d2970, 0xa4d2bc0
Caught exception in looking up space point ptr... 0xa4d5ba0, 0xa4d59c0
Caught exception in looking up space point ptr... 0xa4d59c0, 0xa4d5ba0
++++>>>> total num hits: 70025, num free: 68169
04-Feb-2026 00:26:22 CET  Closed output file "np02vd_raw_run042421_0041_df-s02-d1_dw_0_20260203T135310_reco_stage0_20260203T232622_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         68.8806       84.6372       93.9656       84.0234       5.56577        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.5681e-05    9.6985e-05    0.000209471   8.3711e-05    3.83413e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.5696        4.08746       5.40887       4.08223      0.333075        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0101958     0.020621      0.134324      0.0127479     0.0234204       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000408282   0.000558331   0.00119624    0.000536102   0.000142787      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00151689    0.00187915    0.00260254    0.00184078    0.000241721      27     
produce:beamevent:BeamEvent                      5.8031e-05    0.000132757   0.000821335   9.9721e-05    0.000138739      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.24876       2.7999        3.12428       2.8418       0.219535        27     
produce:ophit:OpHitFinder                         0.0280846     0.0385188     0.0531785     0.0385263    0.00603851       27     
produce:wclsdatavd:WireCellToolkit                 51.9021       59.7505       68.3444       59.2379       4.33323        27     
produce:gaushit:GausHitFinder                     0.420269      0.904851       1.32887      0.850402      0.219413        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000198891   0.00026803    0.000486072   0.000246411   5.88113e-05      27     
produce:reco3d:SpacePointSolver                    6.43799       11.7235       17.5592       12.0557       2.32453        27     
produce:hitpdune:DisambigFromSpacePoints          0.0512657     0.0941997     0.158968      0.0834425     0.0266465       27     
produce:cluster3d:Cluster3D                       0.558174       1.68279       2.67413       1.64842      0.516981        27     
[art]:TriggerResults:TriggerResultInserter        2.516e-05    3.56227e-05   0.00012998     2.987e-05     2.017e-05       27     
end_path:out1:RootOutput                          4.261e-06    8.66378e-06    3.851e-05     6.531e-06    7.15525e-06      27     
end_path:out1:RootOutput(write)                    2.88326       3.50858       4.13316       3.34558      0.406452        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4620.58 MB
  Peak resident set size usage (VmHWM): 2687.14 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0041_df-s02-d1_dw_0_20260203T135310_reco_stage0_20260203T232622_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0041_df-s02-d1_dw_0_20260203T135310_reco_stage0_20260203T232622_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0041_df-s02-d1_dw_0_20260203T135310_reco_stage0_20260203T232622_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158859.0,
    "core.end_time": 1770161183.0,
    "core.events": [
      6685,
      6691,
      6697,
      6703,
      6709,
      6715,
      6721,
      6727,
      6733,
      6739,
      6745,
      6751,
      6757,
      6763,
      6769,
      6775,
      6781,
      6787,
      6793,
      6799,
      6805,
      6811,
      6817,
      6823,
      6829,
      6835,
      6841
    ],
    "core.event_count": 27,
    "core.first_event_number": 6685,
    "core.last_event_number": 6841,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0041_df-s02-d1_dw_0_20260203T135310.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:57:59 UTC       justIN version: 01.06.00