Jobsub ID 36339.15@dunegpschedd02.fnal.gov
| Jobsub ID | 36339.15@dunegpschedd02.fnal.gov |
| Workflow ID | 2329 |
| Stage ID | 1 |
| User name | ykermaid@fnal.gov |
| HTCondor Group | group_dune.prod.mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2025-09-08 11:55:57 |
| Site | UK_QMUL |
| Entry | DUNE_UK_London_QMUL_arcce02 |
| Last heartbeat | 2025-09-08 15:52:38 |
| From worker node | Hostname | cn002.htc.esc.qmul |
| cpuinfo | Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Started | 2025-09-08 14:15:36 |
| Input files | vd-protodune:np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 1h (5556s) |
| CPU time | 1h (4639s = 83%) |
| Max RSS bytes | 2977697792 (2839 MiB) |
| Outputting started | 2025-09-08 15:48:13 |
| Output files | |
| Finished | 2025-09-08 15:52:38 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
fsp> EOS at call=53 with 8
[16:43:45.276] D [ glue ] frame sink sees EOS
[16:43:45.276] D [ pgraph ] <Pgrapher:> graph execution complete
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 8.58 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.36 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.26 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.1 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 6.01 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.65 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.56 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::OmnibusSigProc : 5.16 sec
[16:43:45.276] I [ timer ] Timer: WireCell::Aux::Resampler : 0.32 sec
[16:43:45.276] I [ timer ] Timer: WireCell::Aux::Resampler : 0.29 sec
[16:43:45.276] I [ timer ] Timer: WireCell::Aux::Resampler : 0.29 sec
[16:43:45.276] I [ timer ] Timer: WireCell::Aux::Resampler : 0.27 sec
[16:43:45.276] I [ timer ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:43:45.276] I [ timer ] Timer: WireCell::Gen::Retagger : 0.01 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:45.276] I [ timer ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:43:45.276] I [ timer ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:43:45.276] I [ timer ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:43:45.276] I [ timer ] Timer: wcls::RawFrameSource : 0 sec
[16:43:45.276] I [ timer ] Timer: wcls::FrameSaver : 0 sec
[16:43:45.276] I [ timer ] Timer: Total node execution : 50.93000029027462 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 33747 traces tagged "gauss"
FrameSaver: q=5.92438e+06 n=730322 tag=gauss
wclsFrameSaver: saving 43073 traces tagged "wiener"
FrameSaver: q=6.41321e+06 n=692104 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 1447 XUs and 1536 XVs -> 96 XUVs
C:0 T:1 3242 XUs and 3252 XVs -> 301 XUVs
C:0 T:2 785 XUs and 968 XVs -> 26 XUVs
C:0 T:3 451 XUs and 404 XVs -> 16 XUVs
C:0 T:4 790 XUs and 822 XVs -> 77 XUVs
C:0 T:5 510 XUs and 746 XVs -> 59 XUVs
C:0 T:6 61 XUs and 44 XVs -> 2 XUVs
C:0 T:7 18 XUs and 26 XVs -> 0 XUVs
C:0 T:8 2418 XUs and 1759 XVs -> 143 XUVs
C:0 T:9 427 XUs and 642 XVs -> 38 XUVs
C:0 T:10 658 XUs and 392 XVs -> 47 XUVs
C:0 T:11 2923 XUs and 2806 XVs -> 544 XUVs
C:0 T:12 1181 XUs and 1491 XVs -> 253 XUVs
C:0 T:13 7935 XUs and 15130 XVs -> 1795 XUVs
C:0 T:14 10960 XUs and 35578 XVs -> 3582 XUVs
C:0 T:15 6237 XUs and 8677 XVs -> 1077 XUVs
8056 XUVs total
1621 collection wire objects
8056 potential space points
Neighbour search...
792888 tests to find 344912 neighbours
Iterating with no regularization...
Begin: 4.10066e+08
0 3.66189e+08
1 3.63038e+08
2 3.62525e+08
3 3.62322e+08
Now with regularization...
Begin: 3.52013e+08
0 3.51853e+08
08-Sep-2025 16:44:23 BST Closed output file "np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740_reco_stage1_20250908T154423_keepup.root"
==================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
==================================================================================================================================
Full event 121.575 194.943 458.526 179.783 71.4405 27
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read) 8.3057e-05 0.000114434 0.00023463 0.00010544 3.70829e-05 27
produce:tpcrawdecoder:PDVDTPCReader 11.7001 26.655 42.2167 27.4949 8.16254 27
produce:triggerrawdecoder:PDVDTriggerReader4 0.0330673 0.0392789 0.111064 0.0332364 0.0157616 27
produce:pdvddaphne:DAPHNEReaderPDVD 4.76531 7.18672 11.1609 6.99559 1.44023 27
produce:ophit:OpHitFinder 0.036378 0.0579062 0.0753064 0.0625157 0.0131834 27
produce:opflash:OpFlashFinderVerticalDrift 0.00686522 0.0151048 0.0290499 0.0143778 0.00529524 27
produce:wclsdatavd:WireCellToolkit 50.3889 65.4787 103.574 66.0639 11.1939 27
produce:gaushit:GausHitFinder 0.852251 1.47333 2.1472 1.44988 0.37898 27
produce:nhitsfilter:NumberOfHitsFilter 0.000339992 0.000532441 0.00115452 0.000503855 0.000150558 27
produce:reco3d:SpacePointSolver 6.55292 11.6699 19.0591 11.4307 3.1763 27
produce:hitpdune:DisambigFromSpacePoints 0.111659 0.264625 0.624629 0.238384 0.109281 27
produce:pandora:StandardPandora 20.13 76.0308 330.13 57.3254 66.7481 27
produce:pandoraTrack:LArPandoraTrackCreation 0.562928 1.85239 4.71167 1.63759 0.926021 27
produce:pandoraGnocalo:GnocchiCalorimetry 0.0172361 0.0330845 0.0581738 0.033581 0.010946 27
[art]:TriggerResults:TriggerResultInserter 3.2872e-05 4.78688e-05 0.00012586 4.3516e-05 1.74264e-05 27
end_path:out1:RootOutput 4.282e-06 9.14081e-06 3.6107e-05 7.8e-06 5.97423e-06 27
end_path:out1:RootOutput(write) 3.01633 4.12359 5.75223 3.6685 0.898331 27
==================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4868.39 MB
Peak resident set size usage (VmHWM): 2977.7 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740_reco_stage1_20250908T154423_keepup.root
\tHists: np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740_reco_stage1_20250908T154425_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740_reco_stage1_20250908T154423_keepup.root
Ran successfully
{
"name": "np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740_reco_stage1_20250908T154423_keepup.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "artroot",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "full-reconstructed",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346265.0,
"core.end_time": 1757346265.0,
"core.events": [
1461525,
1461545,
1461565,
1461585,
1461605,
1461625,
1461645,
1461665,
1461685,
1461705,
1461725,
1461745,
1461765,
1461785,
1461805,
1461825,
1461845,
1461865,
1461885,
1461905,
1461925,
1461945,
1461965,
1461985,
1462005,
1462025,
1462045
],
"core.event_count": 27,
"core.first_event_number": 1461525,
"core.last_event_number": 1462045,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740.hdf5"
}
]
}Forming hist metadata
formed
{
"name": "np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740_reco_stage1_20250908T154425_keepup_hists.root",
"namespace": "vd-protodune-det-reco",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_10_00d00",
"core.data_tier": "root-tuple-virtual",
"dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
"dune.campaign": "vd-protodune-reco-keepup-v0",
"core.start_time": 1757346265.0,
"core.end_time": 1757346265.0,
"core.data_stream": "physics",
"core.file_content_status": "good",
"core.file_type": "detector",
"core.run_type": "vd-protodune",
"core.runs": [
39324
],
"core.runs_subruns": [
3932400001
],
"dune.daq_test": false,
"retention.status": "active",
"retention.class": "physics"
},
"parents": [
{
"did": "vd-protodune:np02vd_raw_run039324_2785_df-s02-d3_dw_0_20250908T021740.hdf5"
}
]
} 