justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.142@dunegpschedd02.fnal.gov

Jobsub ID297921.142@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce504
Last heartbeat2026-02-04 00:39:35
From worker nodeHostnameb9p12p0289.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:03
Input filesvd-protodune:np02vd_raw_run042433_0367_df-s04-d0_dw_0_20260203T210137.hdf5
JobscriptExit code0
Real time41m (2470s)
CPU time39m (2387s = 96%)
Max RSS bytes3419824128 (3261 MiB)
Outputting started2026-02-04 00:37:15
Output files
Finished2026-02-04 00:39:35
Saved logsjustin-logs:297921.142-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

space point ptr... 0x5de6cc80, 0x5de857a0
Caught exception in looking up space point ptr... 0x5de857a0, 0x5de78fb0
Caught exception in looking up space point ptr... 0x5c5613d0, 0x53e757f0
Caught exception in looking up space point ptr... 0x53e757f0, 0x5c5613d0
Caught exception in looking up space point ptr... 0x3b259b20, 0x3b2598e0
Caught exception in looking up space point ptr... 0x3b2598e0, 0x3b259b20
Caught exception in looking up space point ptr... 0x3b2831c0, 0x3b282ac0
Caught exception in looking up space point ptr... 0x3b282ac0, 0x3b2845b0
Caught exception in looking up space point ptr... 0x51ed1810, 0x51ed1510
Caught exception in looking up space point ptr... 0x51ed1510, 0x51ed1610
Caught exception in looking up space point ptr... 0x184f7af0, 0x6800fa30
Caught exception in looking up space point ptr... 0x6800fa30, 0x488150b0
Caught exception in looking up space point ptr... 0x4e2092b0, 0x41052440
Caught exception in looking up space point ptr... 0x7cbabc70, 0x4e2092b0
Caught exception in looking up space point ptr... 0x37893ae0, 0x3d418000
Caught exception in looking up space point ptr... 0x3d418000, 0x37893ae0
Caught exception in looking up space point ptr... 0x56933850, 0x67eddaf0
Caught exception in looking up space point ptr... 0x67eddaf0, 0x13e4bb00
Caught exception in looking up space point ptr... 0x13e4bb00, 0x1b173190
Caught exception in looking up space point ptr... 0x1b2ddf30, 0x3bd62a30
Caught exception in looking up space point ptr... 0x3bd62a30, 0x1b2ddf30
Caught exception in looking up space point ptr... 0x960e0d0, 0x6745f750
Caught exception in looking up space point ptr... 0x6ab5e2d0, 0x960e0d0
Caught exception in looking up space point ptr... 0x55a12850, 0x402e4a70
Caught exception in looking up space point ptr... 0x3c64d440, 0x55a12850
Caught exception in looking up space point ptr... 0x110d1480, 0x110ccda0
Caught exception in looking up space point ptr... 0x110ccda0, 0x110c91e0
Caught exception in looking up space point ptr... 0x5de83560, 0x5de8de80
Caught exception in looking up space point ptr... 0x5de8de80, 0x5de8e1a0
Caught exception in looking up space point ptr... 0x5dda70b0, 0x5dda6f20
Caught exception in looking up space point ptr... 0x5dda6f20, 0x5dda6d90
Caught exception in looking up space point ptr... 0x5dda6d90, 0x5dda3360
Caught exception in looking up space point ptr... 0x5dda3360, 0x5dda3500
Caught exception in looking up space point ptr... 0x6977a9d0, 0x6977dfc0
Caught exception in looking up space point ptr... 0x6977dfc0, 0x6977a9d0
Caught exception in looking up space point ptr... 0x5dd98130, 0x5dd97fe0
Caught exception in looking up space point ptr... 0x5dd97fe0, 0x5dd98130
Caught exception in looking up space point ptr... 0x1aa46b30, 0x39adf580
Caught exception in looking up space point ptr... 0x39adf580, 0x1aa46b30
Caught exception in looking up space point ptr... 0xb4a65c0, 0xb4a6430
Caught exception in looking up space point ptr... 0xb4a6430, 0xb4a65c0
Caught exception in looking up space point ptr... 0x345bff40, 0x345bfdb0
Caught exception in looking up space point ptr... 0x345bfdb0, 0x345bfc20
Caught exception in looking up space point ptr... 0x5c630910, 0x4bbedb70
Caught exception in looking up space point ptr... 0x4bbedb70, 0x5c630910
Caught exception in looking up space point ptr... 0x361a5740, 0x39efd880
Caught exception in looking up space point ptr... 0x39efd880, 0x39efd730
Caught exception in looking up space point ptr... 0x18d0bef0, 0x18d0a5b0
Caught exception in looking up space point ptr... 0x18d0a5b0, 0x18d0af20
Caught exception in looking up space point ptr... 0xafe79e0, 0xafd06c0
Caught exception in looking up space point ptr... 0xafd06c0, 0xafe6dd0
Caught exception in looking up space point ptr... 0xe26ec60, 0xe26ead0
Caught exception in looking up space point ptr... 0xe26ead0, 0xe26ec60
Caught exception in looking up space point ptr... 0xe26ec60, 0xe26e940
Caught exception in looking up space point ptr... 0xe26e940, 0xe26ec60
Caught exception in looking up space point ptr... 0x63c98640, 0x63c98320
Caught exception in looking up space point ptr... 0x63c98320, 0x63c98640
Caught exception in looking up space point ptr... 0x363dd990, 0x38e57500
Caught exception in looking up space point ptr... 0x38e57500, 0x363dd990
Caught exception in looking up space point ptr... 0x136febd0, 0x3955c860
Caught exception in looking up space point ptr... 0x3955c860, 0x39f18890
Caught exception in looking up space point ptr... 0x2cae0850, 0x3e45fac0
Caught exception in looking up space point ptr... 0x3e45fac0, 0x2cae0850
Caught exception in looking up space point ptr... 0x63d4f710, 0x63d4f7b0
Caught exception in looking up space point ptr... 0x63d4f7b0, 0x63d4f710
Caught exception in looking up space point ptr... 0x33f57d90, 0x63ca4890
Caught exception in looking up space point ptr... 0x63ca4890, 0x33f57d90
++++>>>> total num hits: 63130, num free: 60499
04-Feb-2026 01:37:07 CET  Closed output file "np02vd_raw_run042433_0367_df-s04-d0_dw_0_20260203T210137_reco_stage0_20260204T003707_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         110.288       127.193       138.024       126.833       7.87758        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.31e-05     9.12492e-05   0.000236011    8.186e-05    4.41226e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.25818       6.1488        6.68689       6.2809       0.386886        19     
produce:triggerrawdecoder:PDVDTriggerReader4     0.00894979     0.0175305     0.0398598     0.0131242     0.0101427       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000361281   0.000544732    0.0010855    0.00049191    0.000166285      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000359551   0.000489613   0.000720833   0.000457471   0.00010532       19     
produce:beamevent:BeamEvent                       4.917e-05    0.000129255   0.000945684    7.711e-05    0.00019464       19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.05855       2.54964       3.18196       2.42171      0.307191        19     
produce:ophit:OpHitFinder                         0.0213963     0.0325631     0.0420043     0.0339636    0.00554395       19     
produce:wclsdatavd:WireCellToolkit                 84.315        92.5343       98.5029       93.5382       4.02539        19     
produce:gaushit:GausHitFinder                     0.761722       1.18846       1.69064       1.10837      0.270382        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000240872   0.000356896   0.000728593   0.000335131   0.000109024      19     
produce:reco3d:SpacePointSolver                    9.63552       17.3666       25.5706       17.9662       4.87079        19     
produce:hitpdune:DisambigFromSpacePoints          0.108011      0.158305       0.23679      0.152448      0.0342069       19     
produce:cluster3d:Cluster3D                        1.31539       2.33239       4.47277       2.16902      0.763535        19     
[art]:TriggerResults:TriggerResultInserter       2.4241e-05    3.15091e-05    0.0001063    2.6141e-05    1.78478e-05      19     
end_path:out1:RootOutput                          3.59e-06     6.66742e-06   2.8501e-05     4.98e-06     5.3847e-06       19     
end_path:out1:RootOutput(write)                    4.20031       4.8307        6.34527       4.54658      0.638292        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5431.98 MB
  Peak resident set size usage (VmHWM): 3419.82 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0367_df-s04-d0_dw_0_20260203T210137_reco_stage0_20260204T003707_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0367_df-s04-d0_dw_0_20260203T210137_reco_stage0_20260204T003707_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0367_df-s04-d0_dw_0_20260203T210137_reco_stage0_20260204T003707_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162973.0,
    "core.end_time": 1770165428.0,
    "core.events": [
      41848,
      41854,
      41860,
      41866,
      41872,
      41878,
      41884,
      41890,
      41896,
      41902,
      41908,
      41914,
      41920,
      41926,
      41932,
      41938,
      41944,
      41950,
      41956
    ],
    "core.event_count": 19,
    "core.first_event_number": 41848,
    "core.last_event_number": 41956,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0367_df-s04-d0_dw_0_20260203T210137.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:55 UTC       justIN version: 01.06.00