justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.174@dunegpschedd01.fnal.gov

Jobsub ID303653.174@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2026-02-04 00:49:07
From worker nodeHostnamewm00.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:48:33
Input filesvd-protodune:np02vd_raw_run042421_0090_df-s04-d0_dw_0_20260203T141941.hdf5
JobscriptExit code0
Real time57m (3478s)
CPU time49m (2942s = 84%)
Max RSS bytes2580746240 (2461 MiB)
Outputting started2026-02-04 00:46:33
Output files
Finished2026-02-04 00:49:07
Saved logsjustin-logs:303653.174-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

70
Caught exception in looking up space point ptr... 0x36d61670, 0x4e4c3e40
Caught exception in looking up space point ptr... 0xb624630, 0x4d471320
Caught exception in looking up space point ptr... 0x4468e6b0, 0xb624630
Caught exception in looking up space point ptr... 0x2c586260, 0x2c587690
Caught exception in looking up space point ptr... 0x2c587690, 0x2c586260
Caught exception in looking up space point ptr... 0x2c586260, 0x539d2400
Caught exception in looking up space point ptr... 0x539d2400, 0x2c586260
Caught exception in looking up space point ptr... 0x3074bdc0, 0x5caede40
Caught exception in looking up space point ptr... 0x5caede40, 0x3074bdc0
Caught exception in looking up space point ptr... 0x29db1980, 0x37b6a480
Caught exception in looking up space point ptr... 0x37b6a480, 0x29db1980
Caught exception in looking up space point ptr... 0x233382e0, 0x23337fc0
Caught exception in looking up space point ptr... 0xd4ffc40, 0x233382e0
Caught exception in looking up space point ptr... 0x22b5a290, 0x41483fc0
Caught exception in looking up space point ptr... 0x294b4080, 0x22b5a290
Caught exception in looking up space point ptr... 0x5d2a4770, 0x5d2a4a60
Caught exception in looking up space point ptr... 0x5d2a4a60, 0x5d2a4770
Caught exception in looking up space point ptr... 0x565bdb90, 0x16ece2e0
Caught exception in looking up space point ptr... 0x16ece2e0, 0x591f0be0
Caught exception in looking up space point ptr... 0x33530b90, 0x53a4d400
Caught exception in looking up space point ptr... 0x53a4d400, 0x2c633710
Caught exception in looking up space point ptr... 0x323951c0, 0x32395350
Caught exception in looking up space point ptr... 0x32395350, 0x32395670
Caught exception in looking up space point ptr... 0x89be320, 0x1527f5e0
Caught exception in looking up space point ptr... 0x1527f5e0, 0x89be320
Caught exception in looking up space point ptr... 0x4587c9e0, 0x100308b0
Caught exception in looking up space point ptr... 0x100308b0, 0x34636840
Caught exception in looking up space point ptr... 0x23d57270, 0x23d56b60
Caught exception in looking up space point ptr... 0x23d56b60, 0x45350eb0
Caught exception in looking up space point ptr... 0x45350eb0, 0x4534f340
Caught exception in looking up space point ptr... 0x2d854370, 0x24770c80
Caught exception in looking up space point ptr... 0x24770c80, 0x2d854370
Caught exception in looking up space point ptr... 0x5cb5cfb0, 0x3b5d6580
Caught exception in looking up space point ptr... 0x3b5d6580, 0x55b4f3a0
Caught exception in looking up space point ptr... 0x55b4f3a0, 0x1474dfc0
Caught exception in looking up space point ptr... 0x1474dfc0, 0x8a61080
Caught exception in looking up space point ptr... 0x8a61080, 0x5cb5cfb0
Caught exception in looking up space point ptr... 0x119d9260, 0x5954e8d0
Caught exception in looking up space point ptr... 0x295b6080, 0x119d9260
Caught exception in looking up space point ptr... 0x45b8b770, 0x3cb89a80
Caught exception in looking up space point ptr... 0x3cb89a80, 0x45b8b770
Caught exception in looking up space point ptr... 0x11b2a370, 0x14fe1c80
Caught exception in looking up space point ptr... 0x5d6e0fe0, 0x11b2a370
Caught exception in looking up space point ptr... 0x5d803a20, 0x3451d050
Caught exception in looking up space point ptr... 0x3451d050, 0x2307dc40
Caught exception in looking up space point ptr... 0x16977a30, 0x2969bd00
Caught exception in looking up space point ptr... 0x2969bd00, 0x4163b4b0
Caught exception in looking up space point ptr... 0x272ce360, 0x31889880
Caught exception in looking up space point ptr... 0x31889880, 0x272ce360
Caught exception in looking up space point ptr... 0x272ce360, 0x3eca4c10
Caught exception in looking up space point ptr... 0x3eca4c10, 0x272ce360
Caught exception in looking up space point ptr... 0x565c00e0, 0x9dcf910
Caught exception in looking up space point ptr... 0xf0d4420, 0x565c00e0
Caught exception in looking up space point ptr... 0x15b64340, 0x12a49b00
Caught exception in looking up space point ptr... 0x12a49b00, 0x15b64340
Caught exception in looking up space point ptr... 0x15b64340, 0x2c7d6050
Caught exception in looking up space point ptr... 0x2c7d6050, 0x15b64340
Caught exception in looking up space point ptr... 0x46af4c90, 0x23db3520
Caught exception in looking up space point ptr... 0x23db3520, 0x46af4c90
Caught exception in looking up space point ptr... 0x46af4c90, 0x46af5cf0
Caught exception in looking up space point ptr... 0x46af5cf0, 0x46af4c90
Caught exception in looking up space point ptr... 0x308482a0, 0x30848440
Caught exception in looking up space point ptr... 0x30848440, 0x308482a0
Caught exception in looking up space point ptr... 0x11e6ace0, 0x2fa4db90
Caught exception in looking up space point ptr... 0x2fa4db90, 0x11e6ace0
++++>>>> total num hits: 44152, num free: 42207
04-Feb-2026 00:46:23 GMT  Closed output file "np02vd_raw_run042421_0090_df-s04-d0_dw_0_20260203T141941_reco_stage0_20260204T004623_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         99.6992       124.985       149.761       124.881       13.542         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       6.6366e-05    9.4855e-05    0.000261504   9.0872e-05    3.5232e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                11.5187       13.6295       15.361        13.925        1.16157        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.489981      0.506237      0.530705      0.505273      0.0109456       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0163193     0.0166691     0.0171014     0.0166814    0.000137337      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0810186     0.0837531     0.0977659     0.0821692    0.00477723       27     
produce:beamevent:BeamEvent                      8.2707e-05    0.000169132    0.0014001    0.000117342   0.000242613      27     
produce:pdvddaphne:DAPHNEReaderPDVD                3.60168       4.2897        4.94957       4.20102      0.393506        27     
produce:ophit:OpHitFinder                         0.0331497     0.046131      0.0562494     0.0460678    0.00649904       27     
produce:wclsdatavd:WireCellToolkit                 66.4115       83.145        101.182       81.6701       9.49933        27     
produce:gaushit:GausHitFinder                     0.704274       1.33681       2.29121       1.28531      0.414491        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000213464   0.000369465   0.00116245    0.000343299   0.00016995       27     
produce:reco3d:SpacePointSolver                    6.19246       11.8181       17.7927       11.1743       3.29423        27     
produce:hitpdune:DisambigFromSpacePoints          0.0631851     0.140965      0.263723      0.134733      0.0525221       27     
produce:cluster3d:Cluster3D                        1.31931       2.85884       5.50163       2.48662       1.04211        27     
[art]:TriggerResults:TriggerResultInserter        2.103e-05    3.90739e-05   0.000105229   3.1189e-05    2.19299e-05      27     
end_path:out1:RootOutput                          3.827e-06    7.40874e-06   2.5298e-05     5.05e-06     4.88496e-06      27     
end_path:out1:RootOutput(write)                    4.5153        7.0827        9.97851       6.89301       1.41308        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4483.57 MB
  Peak resident set size usage (VmHWM): 2580.75 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0090_df-s04-d0_dw_0_20260203T141941_reco_stage0_20260204T004623_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0090_df-s04-d0_dw_0_20260203T141941_reco_stage0_20260204T004623_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0090_df-s04-d0_dw_0_20260203T141941_reco_stage0_20260204T004623_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162525.0,
    "core.end_time": 1770165985.0,
    "core.events": [
      14638,
      14644,
      14650,
      14656,
      14662,
      14668,
      14674,
      14680,
      14686,
      14692,
      14698,
      14704,
      14710,
      14716,
      14722,
      14728,
      14734,
      14740,
      14746,
      14752,
      14758,
      14764,
      14770,
      14776,
      14782,
      14788,
      14794
    ],
    "core.event_count": 27,
    "core.first_event_number": 14638,
    "core.last_event_number": 14794,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0090_df-s04-d0_dw_0_20260203T141941.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:01 UTC       justIN version: 01.06.00