justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298011.2@dunegpschedd02.fnal.gov

Jobsub ID298011.2@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:38:31
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2026-02-04 03:03:22
From worker nodeHostnamewn2209251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting
Started2026-02-04 01:39:13
Input filesvd-protodune:np02vd_raw_run042433_0090_df-s04-d0_dw_0_20260203T191619.hdf5
JobscriptExit code0
Real time1h (5044s)
CPU time1h (4555s = 90%)
Max RSS bytes5412433920 (5161 MiB)
Outputting started2026-02-04 03:03:22
Output files
Finished 
List job events     Live HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x7695ea0, 0x4349ac40
Caught exception in looking up space point ptr... 0x4349ac40, 0x45685720
Caught exception in looking up space point ptr... 0x569baac0, 0x7695ea0
Caught exception in looking up space point ptr... 0x6d27cfe0, 0x6d267160
Caught exception in looking up space point ptr... 0x6d267160, 0x6d268b30
Caught exception in looking up space point ptr... 0x6d268b30, 0x6d266d80
Caught exception in looking up space point ptr... 0x6d266d80, 0x6d26b690
Caught exception in looking up space point ptr... 0x6d2745c0, 0x6d27cfe0
Caught exception in looking up space point ptr... 0x380fa350, 0x380fa710
Caught exception in looking up space point ptr... 0x380fa710, 0x380fa350
Caught exception in looking up space point ptr... 0x33c11d80, 0x33c11f10
Caught exception in looking up space point ptr... 0x33c11f10, 0x33c120a0
Caught exception in looking up space point ptr... 0x3b257b80, 0x4da3c600
Caught exception in looking up space point ptr... 0x50916fa0, 0x3b257b80
Caught exception in looking up space point ptr... 0x5b75d0d0, 0x5b75eb70
Caught exception in looking up space point ptr... 0x5b75eb70, 0x37b085f0
Caught exception in looking up space point ptr... 0x9e58810, 0x39ab1af0
Caught exception in looking up space point ptr... 0x39ab1af0, 0x39ab1de0
Caught exception in looking up space point ptr... 0x39ab1de0, 0x39ab1800
Caught exception in looking up space point ptr... 0x39ab1800, 0x10955280
Caught exception in looking up space point ptr... 0x10955280, 0x9e55ab0
Caught exception in looking up space point ptr... 0x4563f470, 0x4036e390
Caught exception in looking up space point ptr... 0x4036e390, 0x4563fa50
Caught exception in looking up space point ptr... 0x39f4f6f0, 0x3c8c7c30
Caught exception in looking up space point ptr... 0x3c8c7c30, 0x39f4f990
Caught exception in looking up space point ptr... 0x50f14a30, 0x45642d30
Caught exception in looking up space point ptr... 0x50f12230, 0x50f14a30
Caught exception in looking up space point ptr... 0x30d20de0, 0x30d20f70
Caught exception in looking up space point ptr... 0x30d20f70, 0x30d21100
Caught exception in looking up space point ptr... 0x4bc94540, 0x50be8870
Caught exception in looking up space point ptr... 0x4065dbe0, 0x4bc94540
Caught exception in looking up space point ptr... 0x56a1a0d0, 0x56a1bc50
Caught exception in looking up space point ptr... 0x56a1bc50, 0x56a22690
Caught exception in looking up space point ptr... 0x3fd168a0, 0x3cc08b70
Caught exception in looking up space point ptr... 0x3cc079e0, 0x3fd168a0
Caught exception in looking up space point ptr... 0xc0a5d50, 0xc0a5c00
Caught exception in looking up space point ptr... 0xc0a5c00, 0xc0a5d50
Caught exception in looking up space point ptr... 0x6f22af20, 0x6f22b2e0
Caught exception in looking up space point ptr... 0x6f22b2e0, 0x6f22b4c0
Caught exception in looking up space point ptr... 0x39072af0, 0x390711f0
Caught exception in looking up space point ptr... 0x390711f0, 0x39072af0
Caught exception in looking up space point ptr... 0x7d122c00, 0x7d122d40
Caught exception in looking up space point ptr... 0x7d122d40, 0x7d122c00
Caught exception in looking up space point ptr... 0x3b44b110, 0x5e3ad1d0
Caught exception in looking up space point ptr... 0x5e3ad1d0, 0x4a9f8ad0
Caught exception in looking up space point ptr... 0x3585f930, 0x4d841870
Caught exception in looking up space point ptr... 0x38660930, 0x3585f930
Caught exception in looking up space point ptr... 0x7679ca0, 0x3b44edd0
Caught exception in looking up space point ptr... 0x3b44edd0, 0x7679ca0
Caught exception in looking up space point ptr... 0x35c7a860, 0x35c7a5b0
Caught exception in looking up space point ptr... 0x35c7a5b0, 0x35c7a860
Caught exception in looking up space point ptr... 0x120ed0e0, 0x120ec390
Caught exception in looking up space point ptr... 0x44e0e7e0, 0x120ed0e0
Caught exception in looking up space point ptr... 0x13d69290, 0x3e317f40
Caught exception in looking up space point ptr... 0x3e317f40, 0x39ec3d00
Caught exception in looking up space point ptr... 0x1868a310, 0xf98fff0
Caught exception in looking up space point ptr... 0xf98fff0, 0x1868a310
Caught exception in looking up space point ptr... 0x3beeb5f0, 0x46b78f90
Caught exception in looking up space point ptr... 0x46b78f90, 0x5b5edff0
Caught exception in looking up space point ptr... 0x33bc1f60, 0x33bb50e0
Caught exception in looking up space point ptr... 0x33bb50e0, 0x33bc2440
Caught exception in looking up space point ptr... 0x33c11a60, 0x365eb6e0
Caught exception in looking up space point ptr... 0x365eb6e0, 0x33c093b0
Caught exception in looking up space point ptr... 0x4bb37d00, 0x40711c10
Caught exception in looking up space point ptr... 0x40711c10, 0x4bb37d00
Caught exception in looking up space point ptr... 0x406f9b00, 0x406faa40
Caught exception in looking up space point ptr... 0x406faa40, 0x47f6e6d0
++++>>>> total num hits: 59505, num free: 57715
04-Feb-2026 03:03:10 GMT  Closed output file "np02vd_raw_run042433_0090_df-s04-d0_dw_0_20260203T191619_reco_stage0_20260204T030310_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         170.635       259.755       516.451       210.458       110.551        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.9499e-05    0.000160491   0.000322387   0.000145083    6.706e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                20.9219       23.0872       27.8408       22.4047       1.86414        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.635904      0.666102      0.684075      0.668601      0.012651        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0217838     0.0224817     0.0233627     0.0224084    0.000376194      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0216296     0.0224532     0.0232554     0.0224499    0.000383109      19     
produce:beamevent:BeamEvent                      0.000140104   0.000275381   0.00125275    0.000211909   0.000240397      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.74857       4.28093       4.89449       4.10303      0.373477        19     
produce:ophit:OpHitFinder                         0.0413496     0.0480271     0.0529409     0.0477167    0.00328683       19     
produce:wclsdatavd:WireCellToolkit                 118.161       139.306       155.868       140.007       10.567         19     
produce:gaushit:GausHitFinder                      1.08791       2.02567       3.95113       1.74458      0.788081        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000274677   0.000543329   0.00197463    0.000444647   0.000375122      19     
produce:reco3d:SpacePointSolver                    13.2229       22.2187       37.7947       17.5374       8.20144        19     
produce:hitpdune:DisambigFromSpacePoints          0.140519      0.378926       1.14058      0.268016      0.283297        19     
produce:cluster3d:Cluster3D                        2.1877        59.1267       314.254       4.19284       103.984        19     
[art]:TriggerResults:TriggerResultInserter       2.5308e-05    4.93286e-05   0.000156826   3.4435e-05    3.70368e-05      19     
end_path:out1:RootOutput                          5.64e-06     1.35349e-05   3.9845e-05     1.083e-05    8.25253e-06      19     
end_path:out1:RootOutput(write)                    5.38235       8.50551       19.3224       7.23887       3.4297         19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7451.79 MB
  Peak resident set size usage (VmHWM): 5412.43 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0090_df-s04-d0_dw_0_20260203T191619_reco_stage0_20260204T030310_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0090_df-s04-d0_dw_0_20260203T191619_reco_stage0_20260204T030310_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0090_df-s04-d0_dw_0_20260203T191619_reco_stage0_20260204T030310_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770169167.0,
    "core.end_time": 1770174192.0,
    "core.events": [
      10270,
      10276,
      10282,
      10288,
      10294,
      10300,
      10306,
      10312,
      10318,
      10324,
      10330,
      10336,
      10342,
      10348,
      10354,
      10360,
      10366,
      10372,
      10378
    ],
    "core.event_count": 19,
    "core.first_event_number": 10270,
    "core.last_event_number": 10378,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0090_df-s04-d0_dw_0_20260203T191619.hdf5"
    }
  ]
}
justIN time: 2026-02-04 03:05:42 UTC       justIN version: 01.06.00