justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.149@dunegpschedd01.fnal.gov

Jobsub ID303652.149@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2026-02-04 00:31:01
From worker nodeHostnameb9p10p7838.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:32:16
Input filesvd-protodune:np02vd_raw_run042433_0138_df-s04-d0_dw_0_20260203T193434.hdf5
JobscriptExit code0
Real time56m (3376s)
CPU time54m (3249s = 96%)
Max RSS bytes3425292288 (3266 MiB)
Outputting started2026-02-04 00:28:34
Output files
Finished2026-02-04 00:31:01
Saved logsjustin-logs:303652.149-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tr... 0x54af1cb0, 0x54aef860
Caught exception in looking up space point ptr... 0x7066e930, 0x7066e7e0
Caught exception in looking up space point ptr... 0x7066e7e0, 0x7066e540
Caught exception in looking up space point ptr... 0x60a45ad0, 0x10ef1ea0
Caught exception in looking up space point ptr... 0x10ef1ea0, 0x60a45ad0
Caught exception in looking up space point ptr... 0x3ac5f570, 0x3ac5f470
Caught exception in looking up space point ptr... 0x3ac5f470, 0x66888340
Caught exception in looking up space point ptr... 0x66888340, 0x668884d0
Caught exception in looking up space point ptr... 0x668884d0, 0x3ac5f6c0
Caught exception in looking up space point ptr... 0x123c6c40, 0x123c6920
Caught exception in looking up space point ptr... 0x123c6920, 0x123c6c40
Caught exception in looking up space point ptr... 0x109afc60, 0x109af940
Caught exception in looking up space point ptr... 0x109af940, 0x109af7b0
Caught exception in looking up space point ptr... 0x3b980470, 0x3b9801e0
Caught exception in looking up space point ptr... 0x3b9801e0, 0x3b925c70
Caught exception in looking up space point ptr... 0x3b925c70, 0x34fae6e0
Caught exception in looking up space point ptr... 0x34fae6e0, 0x3b9482b0
Caught exception in looking up space point ptr... 0x3b9482b0, 0x55d14340
Caught exception in looking up space point ptr... 0x6d512380, 0x6d5121f0
Caught exception in looking up space point ptr... 0x6d5121f0, 0x6d512380
Caught exception in looking up space point ptr... 0x47ad80d0, 0x930a8b0
Caught exception in looking up space point ptr... 0x381abb30, 0x47ad80d0
Caught exception in looking up space point ptr... 0x3077acc0, 0x30782a90
Caught exception in looking up space point ptr... 0x30782a90, 0x3077acc0
Caught exception in looking up space point ptr... 0x3ecd8f30, 0x14e36000
Caught exception in looking up space point ptr... 0xd7c5d30, 0x3ecd8f30
Caught exception in looking up space point ptr... 0x5b4b1e40, 0x5b4b1fd0
Caught exception in looking up space point ptr... 0x5b4b1fd0, 0x5b4b1e40
Caught exception in looking up space point ptr... 0x1668a720, 0x1668bc30
Caught exception in looking up space point ptr... 0x1668bc30, 0x1668a720
Caught exception in looking up space point ptr... 0x6115f290, 0x6115f000
Caught exception in looking up space point ptr... 0x6115f000, 0x6115f520
Caught exception in looking up space point ptr... 0x6115f520, 0x3eb3d630
Caught exception in looking up space point ptr... 0x7288c6f0, 0x667b9100
Caught exception in looking up space point ptr... 0x667b9100, 0x667b9310
Caught exception in looking up space point ptr... 0x667b9310, 0x667b9520
Caught exception in looking up space point ptr... 0x667b9520, 0x7288ca80
Caught exception in looking up space point ptr... 0x2dc5f6a0, 0x2dc5e910
Caught exception in looking up space point ptr... 0x2dc5f970, 0x2dc5f6a0
Caught exception in looking up space point ptr... 0x612c7fd0, 0x612c7520
Caught exception in looking up space point ptr... 0x612c7520, 0x612c7fd0
Caught exception in looking up space point ptr... 0x62a575f0, 0x62a556e0
Caught exception in looking up space point ptr... 0x62a556e0, 0x62a55870
Caught exception in looking up space point ptr... 0x3cfa8730, 0x3cfa8690
Caught exception in looking up space point ptr... 0x3cfa8690, 0x3cfa8730
Caught exception in looking up space point ptr... 0x3cfa8730, 0x3cfa85f0
Caught exception in looking up space point ptr... 0x3cfa85f0, 0x3cfa8730
Caught exception in looking up space point ptr... 0x6ec546e0, 0x827ec550
Caught exception in looking up space point ptr... 0x827ec550, 0x6ec546e0
Caught exception in looking up space point ptr... 0x849a560, 0x3063adf0
Caught exception in looking up space point ptr... 0x3063adf0, 0x849a560
Caught exception in looking up space point ptr... 0x6aff7dc0, 0x6affd7a0
Caught exception in looking up space point ptr... 0x6affd7a0, 0x6aff7dc0
Caught exception in looking up space point ptr... 0x6aff7dc0, 0x1494dc80
Caught exception in looking up space point ptr... 0x1494dc80, 0x6aff7dc0
Caught exception in looking up space point ptr... 0x2dc2f7d0, 0x126b5ce0
Caught exception in looking up space point ptr... 0x126b5ce0, 0x2dc2f7d0
Caught exception in looking up space point ptr... 0x2f1c89d0, 0x7b89e60
Caught exception in looking up space point ptr... 0x7b89e60, 0x2f1c89d0
Caught exception in looking up space point ptr... 0x6b6d8010, 0x6b6d7b60
Caught exception in looking up space point ptr... 0x6b6d7b60, 0x6b6d7cf0
Caught exception in looking up space point ptr... 0x13ddbaf0, 0x826dc010
Caught exception in looking up space point ptr... 0x826dc010, 0x13ddbaf0
Caught exception in looking up space point ptr... 0x86715640, 0x86716120
Caught exception in looking up space point ptr... 0x86716120, 0x86715e70
Caught exception in looking up space point ptr... 0x60b294a0, 0x19392720
Caught exception in looking up space point ptr... 0x19392720, 0x60b294a0
++++>>>> total num hits: 296520, num free: 288707
04-Feb-2026 01:28:26 CET  Closed output file "np02vd_raw_run042433_0138_df-s04-d0_dw_0_20260203T193434_reco_stage0_20260204T002826_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         150.226       174.039       207.04        172.356       15.628         19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.608e-05    0.000110937   0.000251202    9.267e-05    5.01817e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.16559       7.16103       12.8277       6.77242       1.39385        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0113909     0.0221496     0.0724041     0.0196793     0.0132563       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000424811   0.000616787   0.00144644    0.000569526   0.000214588      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000418581   0.00059233    0.00168698    0.000533173   0.000265864      19     
produce:beamevent:BeamEvent                      7.3891e-05    0.000153077   0.000870629   9.8461e-05    0.000174438      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.39533       3.04563       3.68049       3.00967      0.323444        19     
produce:ophit:OpHitFinder                         0.0258423     0.0392111     0.0497522     0.0404479    0.00628168       19     
produce:wclsdatavd:WireCellToolkit                 103.85        126.469       153.376       126.625       12.355         19     
produce:gaushit:GausHitFinder                     0.916136       1.61261       2.44952       1.49996      0.421904        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000317033   0.000456732   0.000723245   0.000462581   9.4699e-05       19     
produce:reco3d:SpacePointSolver                    15.5765       25.4603       37.4116       24.8935       5.42288        19     
produce:hitpdune:DisambigFromSpacePoints          0.143991      0.242088      0.336857      0.237264      0.0521621       19     
produce:cluster3d:Cluster3D                        1.9006        4.21479       23.4362       2.98095       4.63346        19     
[art]:TriggerResults:TriggerResultInserter        2.733e-05    4.01877e-05   0.000130861    3.429e-05    2.24101e-05      19     
end_path:out1:RootOutput                          5.74e-06     1.04795e-05   3.9231e-05     8.39e-06     7.18235e-06      19     
end_path:out1:RootOutput(write)                    4.41679       5.73541       7.16147       5.70526      0.767225        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5415.81 MB
  Peak resident set size usage (VmHWM): 3425.29 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0138_df-s04-d0_dw_0_20260203T193434_reco_stage0_20260204T002826_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0138_df-s04-d0_dw_0_20260203T193434_reco_stage0_20260204T002826_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0138_df-s04-d0_dw_0_20260203T193434_reco_stage0_20260204T002826_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161548.0,
    "core.end_time": 1770164907.0,
    "core.events": [
      15742,
      15748,
      15754,
      15760,
      15766,
      15772,
      15778,
      15784,
      15790,
      15796,
      15802,
      15808,
      15814,
      15820,
      15826,
      15832,
      15838,
      15844,
      15850
    ],
    "core.event_count": 19,
    "core.first_event_number": 15742,
    "core.last_event_number": 15850,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0138_df-s04-d0_dw_0_20260203T193434.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:42:23 UTC       justIN version: 01.06.00