justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.11@dunegpschedd01.fnal.gov

Jobsub ID303652.11@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:21:31
From worker nodeHostnameb9p10p5730.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:25:13
Input filesvd-protodune:np02vd_raw_run042433_0543_df-s04-d0_dw_0_20260203T220830.hdf5
JobscriptExit code0
Real time53m (3217s)
CPU time51m (3116s = 96%)
Max RSS bytes3645612032 (3476 MiB)
Outputting started2026-02-04 00:18:52
Output files
Finished2026-02-04 00:21:31
Saved logsjustin-logs:303652.11-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x59d02bc0, 0x59ce5ab0
Caught exception in looking up space point ptr... 0x3f4fd840, 0x3f4fd980
Caught exception in looking up space point ptr... 0x3f4fd980, 0x3f4fd840
Caught exception in looking up space point ptr... 0x3f4fd840, 0x3f4fcf10
Caught exception in looking up space point ptr... 0x3f4fcf10, 0x3f4fd840
Caught exception in looking up space point ptr... 0x59e7a540, 0x59e79f00
Caught exception in looking up space point ptr... 0x59e7a9f0, 0x59e7a540
Caught exception in looking up space point ptr... 0x51f5e8b0, 0x51f60c60
Caught exception in looking up space point ptr... 0x51f60c60, 0x51f5e8b0
Caught exception in looking up space point ptr... 0x51f5e8b0, 0x10a5d420
Caught exception in looking up space point ptr... 0x10a5d420, 0x51f5e8b0
Caught exception in looking up space point ptr... 0x51f5e8b0, 0x51f64df0
Caught exception in looking up space point ptr... 0x51f64df0, 0x51f5e8b0
Caught exception in looking up space point ptr... 0x19449410, 0x193fbe80
Caught exception in looking up space point ptr... 0x41e3d350, 0x19449410
Caught exception in looking up space point ptr... 0x182179f0, 0x2c8a5210
Caught exception in looking up space point ptr... 0x2c8a5210, 0x2c8ba5e0
Caught exception in looking up space point ptr... 0x10026f10, 0x10026410
Caught exception in looking up space point ptr... 0x10026410, 0x100256c0
Caught exception in looking up space point ptr... 0x100256c0, 0x10025950
Caught exception in looking up space point ptr... 0x78dd8500, 0x78dc9770
Caught exception in looking up space point ptr... 0x78dc9770, 0x78dd8650
Caught exception in looking up space point ptr... 0x1842a2e0, 0x60da2bf0
Caught exception in looking up space point ptr... 0x60da2bf0, 0x46fec360
Caught exception in looking up space point ptr... 0x31b4d4e0, 0x31b4d800
Caught exception in looking up space point ptr... 0x31b4d800, 0x31b4d4e0
Caught exception in looking up space point ptr... 0x41e44210, 0x51142fb0
Caught exception in looking up space point ptr... 0x51142fb0, 0x34a81970
Caught exception in looking up space point ptr... 0x914489d0, 0x91448830
Caught exception in looking up space point ptr... 0x91448830, 0x914489d0
Caught exception in looking up space point ptr... 0x520f12b0, 0x55bc4c00
Caught exception in looking up space point ptr... 0x55bc4c00, 0x520f1120
Caught exception in looking up space point ptr... 0x520f1120, 0x520f0e40
Caught exception in looking up space point ptr... 0x520f0e40, 0x520f12b0
Caught exception in looking up space point ptr... 0x34ce0e30, 0x60c438d0
Caught exception in looking up space point ptr... 0x60c438d0, 0x34ce0e30
Caught exception in looking up space point ptr... 0x51f73e30, 0x51f74510
Caught exception in looking up space point ptr... 0x51f74510, 0x51f74280
Caught exception in looking up space point ptr... 0x51f74280, 0x51f74840
Caught exception in looking up space point ptr... 0x4fe03f20, 0x4fe040b0
Caught exception in looking up space point ptr... 0x4fe040b0, 0x4fe03f20
Caught exception in looking up space point ptr... 0x2f25bf50, 0x2f25c050
Caught exception in looking up space point ptr... 0x2f25c050, 0x2f25bf50
Caught exception in looking up space point ptr... 0xfa109b0, 0xfa10d20
Caught exception in looking up space point ptr... 0xfa10d20, 0xfa10eb0
Caught exception in looking up space point ptr... 0x51f13080, 0x51f13260
Caught exception in looking up space point ptr... 0x51f13260, 0x51f13080
Caught exception in looking up space point ptr... 0x51f26780, 0x51f259e0
Caught exception in looking up space point ptr... 0x51f259e0, 0x51f29150
Caught exception in looking up space point ptr... 0x384a9eb0, 0x384ab300
Caught exception in looking up space point ptr... 0x3f244980, 0x384a9eb0
Caught exception in looking up space point ptr... 0x3f1139a0, 0x3f766fb0
Caught exception in looking up space point ptr... 0x3f766fb0, 0x3f1139a0
Caught exception in looking up space point ptr... 0x19d52820, 0x19d52b40
Caught exception in looking up space point ptr... 0x19d52b40, 0x19d52820
Caught exception in looking up space point ptr... 0x158e6370, 0x804edf30
Caught exception in looking up space point ptr... 0x804edf30, 0x158e6370
Caught exception in looking up space point ptr... 0x158e6370, 0x17381800
Caught exception in looking up space point ptr... 0x17381800, 0x158e6370
Caught exception in looking up space point ptr... 0xa599610, 0xa59a450
Caught exception in looking up space point ptr... 0xa59a450, 0xa599610
Caught exception in looking up space point ptr... 0x2f52ff50, 0x69c08d70
Caught exception in looking up space point ptr... 0x69c08d70, 0x2f52ff50
Caught exception in looking up space point ptr... 0x51f05760, 0x9145f290
Caught exception in looking up space point ptr... 0x9145f290, 0x51f05760
Caught exception in looking up space point ptr... 0x55622690, 0x41d503a0
Caught exception in looking up space point ptr... 0x41d503a0, 0x55622690
++++>>>> total num hits: 545912, num free: 537548
04-Feb-2026 01:18:43 CET  Closed output file "np02vd_raw_run042433_0543_df-s04-d0_dw_0_20260203T220830_reco_stage0_20260204T001843_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         113.143       166.776       192.707       171.301       22.5603        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.2831e-05    9.04138e-05   0.000166272   7.7521e-05    3.93812e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.87384       6.70255       7.44573       6.64259      0.469474        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0106941     0.0159777     0.0393475     0.011927     0.00805938       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000428935   0.000540219   0.00075542    0.000514846   9.20655e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000396554   0.000533007   0.000880311   0.000505476   0.000123625      19     
produce:beamevent:BeamEvent                      7.2861e-05    0.00014002    0.000868331   0.000102841   0.000172837      19     
produce:pdvddaphne:DAPHNEReaderPDVD                1.98266       2.51949       3.18671       2.42825      0.359451        19     
produce:ophit:OpHitFinder                         0.0206524     0.0338422     0.0412663     0.0351406    0.00549131       19     
produce:wclsdatavd:WireCellToolkit                 83.6527       96.0681       115.785       93.7216       7.43545        19     
produce:gaushit:GausHitFinder                      0.84488       1.91014       3.14386       1.83948      0.515549        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000293183   0.00038725    0.000508107   0.000376845   7.11986e-05      19     
produce:reco3d:SpacePointSolver                    14.5415       19.8368        28.71        19.907        3.2959         19     
produce:hitpdune:DisambigFromSpacePoints          0.118909      0.273898      0.535521      0.251771      0.0877431       19     
produce:cluster3d:Cluster3D                        1.60219       33.6692       51.447        41.3347       17.9615        19     
[art]:TriggerResults:TriggerResultInserter        2.681e-05    4.52832e-05   0.000135022   3.5701e-05    2.42471e-05      19     
end_path:out1:RootOutput                          3.99e-06     8.35695e-06   3.5091e-05     7.29e-06     6.51372e-06      19     
end_path:out1:RootOutput(write)                    4.11181       5.71599       8.17723       5.45445       1.0551         19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5697.57 MB
  Peak resident set size usage (VmHWM): 3645.61 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0543_df-s04-d0_dw_0_20260203T220830_reco_stage0_20260204T001843_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0543_df-s04-d0_dw_0_20260203T220830_reco_stage0_20260204T001843_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0543_df-s04-d0_dw_0_20260203T220830_reco_stage0_20260204T001843_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161122.0,
    "core.end_time": 1770164325.0,
    "core.events": [
      61912,
      61918,
      61924,
      61930,
      61936,
      61942,
      61948,
      61954,
      61960,
      61966,
      61972,
      61978,
      61984,
      61990,
      61996,
      62002,
      62008,
      62014,
      62020
    ],
    "core.event_count": 19,
    "core.first_event_number": 61912,
    "core.last_event_number": 62020,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0543_df-s04-d0_dw_0_20260203T220830.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:03 UTC       justIN version: 01.06.00