While you're still a UK resident, you'll be taxed under the UK system. So yes, you will have to pay interest on your UK building society savings accounts, etc, to the UK Inland Revenue.
But if you spend more than 183 days a year in Cyprus, you should consider becoming resident here as there are a number of tax advantages:
Assuming that you're retired, you will be able to get your private, company & state pensions paid gross in the UK and then pay tax on them here. PricewaterhouseCoopers produce a tax factsheet on Cyprus that you can download by clicking here
(And if you spend more than 183 days a year here in Cyprus you should have registered with the Inland Revenue here).
The situation is more complicated if you're still working. The UK authorities may consider that you're still a UK resident for tax purposes, even though you've registered with the authorities in Cyprus. And even if you're retired, there can be tax issues if you spend more than 90 days in the UK in any tax year.
If you want to explore the various possibilities further, you'll need to take professional advice.