justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303653.56@dunegpschedd01.fnal.gov

Jobsub ID303653.56@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:52:24
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod01
Last heartbeat2026-02-04 01:23:01
From worker nodeHostnamewk13.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:41:03
Input filesvd-protodune:np02vd_raw_run042421_0110_df-s03-d0_dw_0_20260203T143032.hdf5
JobscriptExit code0
Real time1h (5939s)
CPU time1h (5495s = 92%)
Max RSS bytes2716635136 (2590 MiB)
Outputting started2026-02-04 01:20:08
Output files
Finished2026-02-04 01:23:01
Saved logsjustin-logs:303653.56-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

5d280
Caught exception in looking up space point ptr... 0x2c25d280, 0x12c839e0
Caught exception in looking up space point ptr... 0x12c839e0, 0x2c25d280
Caught exception in looking up space point ptr... 0x14d81690, 0x2a4eaa70
Caught exception in looking up space point ptr... 0x2a4eaa70, 0x2a4f7a80
Caught exception in looking up space point ptr... 0x2a4f7a80, 0x642fb5f0
Caught exception in looking up space point ptr... 0x642fb5f0, 0x13dee5d0
Caught exception in looking up space point ptr... 0x549dbf20, 0x529a3400
Caught exception in looking up space point ptr... 0x529a3400, 0x341f5cf0
Caught exception in looking up space point ptr... 0x5ba320c0, 0x12c70c70
Caught exception in looking up space point ptr... 0xd2b6810, 0x5ba320c0
Caught exception in looking up space point ptr... 0x3e940a80, 0x3e93f2d0
Caught exception in looking up space point ptr... 0x3e93f2d0, 0x20c2d200
Caught exception in looking up space point ptr... 0x2f9c3440, 0x13be6920
Caught exception in looking up space point ptr... 0x44f0ec00, 0x2f9c3440
Caught exception in looking up space point ptr... 0x5e664b00, 0x14d84d00
Caught exception in looking up space point ptr... 0x14d84d00, 0x3420aeb0
Caught exception in looking up space point ptr... 0x298e6700, 0x8350210
Caught exception in looking up space point ptr... 0x298e4d50, 0x298e6700
Caught exception in looking up space point ptr... 0x627869c0, 0x32d0ac70
Caught exception in looking up space point ptr... 0x32d0ac70, 0x32d0aae0
Caught exception in looking up space point ptr... 0x10f0f270, 0x447d53f0
Caught exception in looking up space point ptr... 0x447d53f0, 0xe5b11c0
Caught exception in looking up space point ptr... 0x8e614a0, 0x55557040
Caught exception in looking up space point ptr... 0xddce260, 0x8e614a0
Caught exception in looking up space point ptr... 0x2f9cd7d0, 0x4f5beb90
Caught exception in looking up space point ptr... 0x4f5c6dd0, 0x2f9cd7d0
Caught exception in looking up space point ptr... 0x3bd020b0, 0x2b428990
Caught exception in looking up space point ptr... 0x5e56d430, 0x3bd020b0
Caught exception in looking up space point ptr... 0x13f35b30, 0x13f48280
Caught exception in looking up space point ptr... 0x4b9358b0, 0x13f35b30
Caught exception in looking up space point ptr... 0x21b243e0, 0x40e3d830
Caught exception in looking up space point ptr... 0x2f2c1690, 0x21b243e0
Caught exception in looking up space point ptr... 0x403d0a00, 0x403beda0
Caught exception in looking up space point ptr... 0x403beda0, 0x403ac650
Caught exception in looking up space point ptr... 0x30949f70, 0x2e3c5f30
Caught exception in looking up space point ptr... 0x2e3c5f30, 0x3ad42ac0
Caught exception in looking up space point ptr... 0x10f18b00, 0x30949f70
Caught exception in looking up space point ptr... 0x81fb7f0, 0x820d2c0
Caught exception in looking up space point ptr... 0x7edbdf0, 0x81fb7f0
Caught exception in looking up space point ptr... 0x3e399d40, 0x40334fa0
Caught exception in looking up space point ptr... 0x3e3ac490, 0x3e399d40
Caught exception in looking up space point ptr... 0x44674fe0, 0x4e1f0b40
Caught exception in looking up space point ptr... 0x4e1f0b40, 0x4e1f0cd0
Caught exception in looking up space point ptr... 0x5cace890, 0x33713a10
Caught exception in looking up space point ptr... 0x5cacea20, 0x5cace890
Caught exception in looking up space point ptr... 0x3fdeff90, 0x5be02590
Caught exception in looking up space point ptr... 0x3fdf0120, 0x3fdeff90
Caught exception in looking up space point ptr... 0x5c25e9e0, 0x328707f0
Caught exception in looking up space point ptr... 0x328707f0, 0x5bde12c0
Caught exception in looking up space point ptr... 0x40e34d20, 0x40e22a30
Caught exception in looking up space point ptr... 0x40e22a30, 0x40e34b90
Caught exception in looking up space point ptr... 0x40e2fbe0, 0x2e398c40
Caught exception in looking up space point ptr... 0x2e398c40, 0x40e2fa50
Caught exception in looking up space point ptr... 0x2b243aa0, 0x4456fe90
Caught exception in looking up space point ptr... 0x2c00d170, 0x2b243aa0
Caught exception in looking up space point ptr... 0xa230f70, 0x4f997b20
Caught exception in looking up space point ptr... 0x4f997b20, 0xa230f70
Caught exception in looking up space point ptr... 0xa230f70, 0x4e14b0d0
Caught exception in looking up space point ptr... 0x4e14b0d0, 0xa230f70
Caught exception in looking up space point ptr... 0x374d7130, 0x52c50640
Caught exception in looking up space point ptr... 0x52c50790, 0x374d7130
Caught exception in looking up space point ptr... 0x2bb95f60, 0x149beff0
Caught exception in looking up space point ptr... 0x149beff0, 0x2bb95f60
Caught exception in looking up space point ptr... 0x5ca76280, 0x3e73a080
Caught exception in looking up space point ptr... 0x3e73a080, 0x5ca76280
++++>>>> total num hits: 291636, num free: 285746
04-Feb-2026 01:19:49 GMT  Closed output file "np02vd_raw_run042421_0110_df-s03-d0_dw_0_20260203T143032_reco_stage0_20260204T011949_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         151.898       215.584       291.38        206.214       40.4925        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.6921e-05    0.000145854   0.000317579   0.000123277   6.55896e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                12.4899       13.2065       13.8428       13.1801      0.313991        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.488406      0.512617      0.581458      0.507854      0.0218675       27     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0163953     0.0187344     0.0632135     0.0167349    0.00880201       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0810579     0.0829592     0.0985741     0.0816433    0.00431724       27     
produce:beamevent:BeamEvent                      0.000109176   0.000228167   0.00129041    0.000176123   0.000217813      27     
produce:pdvddaphne:DAPHNEReaderPDVD                5.57387       7.23989       8.86772       7.46904      0.910631        27     
produce:ophit:OpHitFinder                         0.0389186     0.0617341     0.098785      0.0653166     0.0169709       27     
produce:wclsdatavd:WireCellToolkit                 88.7486       125.489       152.647       126.41        19.9413        27     
produce:gaushit:GausHitFinder                     0.961765       2.10896       3.94734       2.04298      0.700273        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000348906   0.00058573    0.000833707   0.000545766   0.000128718      27     
produce:reco3d:SpacePointSolver                    6.97666       11.6155       20.1581       11.0145       3.34923        27     
produce:hitpdune:DisambigFromSpacePoints          0.0921973     0.315597       0.63041      0.267571      0.153169        27     
produce:cluster3d:Cluster3D                        1.75781       45.7747       90.7044       53.049        30.3134        27     
[art]:TriggerResults:TriggerResultInserter        2.525e-05    4.80758e-05   0.000115652   4.8315e-05    1.90842e-05      27     
end_path:out1:RootOutput                          4.657e-06    1.62455e-05   0.000166287    9.72e-06     2.98599e-05      27     
end_path:out1:RootOutput(write)                    6.34776       9.12161       12.5743       9.12176       1.81983        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4622.14 MB
  Peak resident set size usage (VmHWM): 2716.64 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0110_df-s03-d0_dw_0_20260203T143032_reco_stage0_20260204T011949_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0110_df-s03-d0_dw_0_20260203T143032_reco_stage0_20260204T011949_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0110_df-s03-d0_dw_0_20260203T143032_reco_stage0_20260204T011949_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162078.0,
    "core.end_time": 1770167994.0,
    "core.events": [
      17894,
      17900,
      17906,
      17912,
      17918,
      17924,
      17930,
      17936,
      17942,
      17948,
      17954,
      17960,
      17966,
      17972,
      17978,
      17984,
      17990,
      17996,
      18002,
      18008,
      18014,
      18020,
      18026,
      18032,
      18038,
      18044,
      18050
    ],
    "core.event_count": 27,
    "core.first_event_number": 17894,
    "core.last_event_number": 18050,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0110_df-s03-d0_dw_0_20260203T143032.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:06:49 UTC       justIN version: 01.06.00