justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303735.0@dunegpschedd01.fnal.gov

Jobsub ID303735.0@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 01:26:31
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 02:41:00
From worker nodeHostnamecn038.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 01:27:32
Input filesvd-protodune:np02vd_raw_run042433_0274_df-s02-d1_dw_0_20260203T202614.hdf5
JobscriptExit code0
Real time1h (4257s)
CPU time1h (4018s = 94%)
Max RSS bytes3370115072 (3213 MiB)
Outputting started2026-02-04 02:38:31
Output files
Finished2026-02-04 02:41:00
Saved logsjustin-logs:303735.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

up space point ptr... 0x380a0e50, 0x398813f0
Caught exception in looking up space point ptr... 0x62eb4560, 0x62eb48a0
Caught exception in looking up space point ptr... 0x62eb48a0, 0x62eb49f0
Caught exception in looking up space point ptr... 0x7194fb40, 0x101684b0
Caught exception in looking up space point ptr... 0x101684b0, 0x7194fb40
Caught exception in looking up space point ptr... 0x7194fb40, 0x4bf19940
Caught exception in looking up space point ptr... 0x4bf19940, 0x7194fb40
Caught exception in looking up space point ptr... 0x7194fb40, 0x7195f270
Caught exception in looking up space point ptr... 0x7195f270, 0x7194fb40
Caught exception in looking up space point ptr... 0x7be5d320, 0x7be5c130
Caught exception in looking up space point ptr... 0x7be5c130, 0x7be5b900
Caught exception in looking up space point ptr... 0x5bfe0fa0, 0x5bfb79f0
Caught exception in looking up space point ptr... 0x5bfe9a80, 0x5bfe0fa0
Caught exception in looking up space point ptr... 0x681b1dc0, 0x40b1cca0
Caught exception in looking up space point ptr... 0x40b1cca0, 0x681b1dc0
Caught exception in looking up space point ptr... 0x675ab9b0, 0x675e2ce0
Caught exception in looking up space point ptr... 0x675e2ce0, 0x675ab9b0
Caught exception in looking up space point ptr... 0xf5fabf0, 0xf5fa810
Caught exception in looking up space point ptr... 0xf5fa810, 0xf5fabf0
Caught exception in looking up space point ptr... 0xbb47540, 0xbb47ec0
Caught exception in looking up space point ptr... 0xbb47ec0, 0xbb47540
Caught exception in looking up space point ptr... 0x71951320, 0x7b909360
Caught exception in looking up space point ptr... 0x7b909360, 0x71951320
Caught exception in looking up space point ptr... 0x9940360, 0x9940500
Caught exception in looking up space point ptr... 0x9940500, 0x3240b490
Caught exception in looking up space point ptr... 0x9940840, 0x9940360
Caught exception in looking up space point ptr... 0x4ecd8ca0, 0x4ecd2750
Caught exception in looking up space point ptr... 0x4ecd2750, 0x4ecd8ca0
Caught exception in looking up space point ptr... 0x197f7a40, 0x197f8770
Caught exception in looking up space point ptr... 0x197f84d0, 0x197f7a40
Caught exception in looking up space point ptr... 0x1810e7b0, 0x1810e6b0
Caught exception in looking up space point ptr... 0x1810e6b0, 0x1810e7b0
Caught exception in looking up space point ptr... 0x9c7a4e0, 0x9c7a070
Caught exception in looking up space point ptr... 0x9c7a070, 0x9c7a4e0
Caught exception in looking up space point ptr... 0x717e29a0, 0x717e2900
Caught exception in looking up space point ptr... 0x717e2900, 0x717e2a90
Caught exception in looking up space point ptr... 0x5fdcb360, 0x5fdcd750
Caught exception in looking up space point ptr... 0x5fdcd750, 0x5fdc9bf0
Caught exception in looking up space point ptr... 0x3abefc10, 0x3abefd70
Caught exception in looking up space point ptr... 0x3abefd70, 0x3abefc10
Caught exception in looking up space point ptr... 0xf22cab0, 0x33142e50
Caught exception in looking up space point ptr... 0x33142e50, 0xf22cab0
Caught exception in looking up space point ptr... 0x5632c220, 0x5632bd70
Caught exception in looking up space point ptr... 0x5632bd70, 0x5632c220
Caught exception in looking up space point ptr... 0x40aa0e70, 0x40aa0d20
Caught exception in looking up space point ptr... 0x40aa0d20, 0x40aa0e70
Caught exception in looking up space point ptr... 0x4b999200, 0x16d14cb0
Caught exception in looking up space point ptr... 0x339b26c0, 0x4b999200
Caught exception in looking up space point ptr... 0x4821d990, 0x8d5be90
Caught exception in looking up space point ptr... 0x8d5be90, 0x2ba2ad70
Caught exception in looking up space point ptr... 0x2ba2ad70, 0x40b4dcf0
Caught exception in looking up space point ptr... 0x61d429c0, 0x61d428c0
Caught exception in looking up space point ptr... 0x61d428c0, 0x61d429c0
Caught exception in looking up space point ptr... 0x4bf18550, 0x101bd0d0
Caught exception in looking up space point ptr... 0x101bd0d0, 0x4bf18550
Caught exception in looking up space point ptr... 0x198cca50, 0x198d0460
Caught exception in looking up space point ptr... 0x198d0460, 0x198cca50
Caught exception in looking up space point ptr... 0x7b97abd0, 0x7b97cf60
Caught exception in looking up space point ptr... 0x7b97cf60, 0x7b97adb0
Caught exception in looking up space point ptr... 0x3b09fa80, 0x3b09f8f0
Caught exception in looking up space point ptr... 0x3b09f8f0, 0x3b09fa80
Caught exception in looking up space point ptr... 0x33a1d8a0, 0x54eee8f0
Caught exception in looking up space point ptr... 0x54eee8f0, 0x33a1d8a0
Caught exception in looking up space point ptr... 0x4bef9010, 0x5ffe4c80
Caught exception in looking up space point ptr... 0x5ffe4c80, 0x4bef9010
Caught exception in looking up space point ptr... 0x3c4c4b30, 0x37bc6050
Caught exception in looking up space point ptr... 0x37bc6050, 0x5fe985f0
++++>>>> total num hits: 103522, num free: 99151
04-Feb-2026 02:38:19 GMT  Closed output file "np02vd_raw_run042433_0274_df-s02-d1_dw_0_20260203T202614_reco_stage0_20260204T023819_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         181.04        218.707       254.056       216.515       19.8058        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.2504e-05    0.000136825   0.000365641   0.000119148   6.1107e-05       19     
produce:tpcrawdecoder:PDVDTPCReader                14.9565       16.1727       17.695        15.8772      0.815895        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.511092      0.547272      0.614744      0.539695      0.0239393       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0166644     0.0192642     0.0386815     0.0168993    0.00516311       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0166621     0.0171144     0.0206191     0.0168909    0.000854102      19     
produce:beamevent:BeamEvent                      0.000117897   0.000274726   0.00133343    0.000175223   0.000307062      19     
produce:pdvddaphne:DAPHNEReaderPDVD                5.66687       6.83546       7.7957        6.87789      0.601763        19     
produce:ophit:OpHitFinder                         0.0363206     0.0465145     0.0681635     0.0435545    0.00932319       19     
produce:wclsdatavd:WireCellToolkit                 137.148       164.747       198.795       163.159       15.6946        19     
produce:gaushit:GausHitFinder                      1.2478        1.87997       2.72825       1.89522      0.389887        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000460602   0.000666339   0.00126647    0.000645623   0.000179279      19     
produce:reco3d:SpacePointSolver                    11.8963       18.3464       24.5297       18.7344       3.66119        19     
produce:hitpdune:DisambigFromSpacePoints          0.147589      0.318151      0.497667      0.309962      0.101581        19     
produce:cluster3d:Cluster3D                        1.38425       3.12717       5.37697       3.28853       1.11384        19     
[art]:TriggerResults:TriggerResultInserter        2.617e-05    4.33302e-05   0.000144677   3.6223e-05    2.48282e-05      19     
end_path:out1:RootOutput                          5.372e-06    2.13999e-05   0.000210019    8.98e-06     4.53857e-05      19     
end_path:out1:RootOutput(write)                    4.95255       6.59406       8.82598       6.32477       1.05979        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5306.65 MB
  Peak resident set size usage (VmHWM): 3370.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0274_df-s02-d1_dw_0_20260203T202614_reco_stage0_20260204T023819_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0274_df-s02-d1_dw_0_20260203T202614_reco_stage0_20260204T023819_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0274_df-s02-d1_dw_0_20260203T202614_reco_stage0_20260204T023819_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770168464.0,
    "core.end_time": 1770172703.0,
    "core.events": [
      31243,
      31249,
      31255,
      31261,
      31267,
      31273,
      31279,
      31285,
      31291,
      31297,
      31303,
      31309,
      31315,
      31321,
      31327,
      31333,
      31339,
      31345,
      31351
    ],
    "core.event_count": 19,
    "core.first_event_number": 31243,
    "core.last_event_number": 31351,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0274_df-s02-d1_dw_0_20260203T202614.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:33:36 UTC       justIN version: 01.06.00