I think you have a good premise for a science fiction story right here: Say some "magic" (i.e. invented) physics quirk allows you to travel both into the future and the past, but all you can do is essentially accelerate and rewind time drastically. You don't "jump" to a time, there's still a physical presence, and colliding would have catastrophic results.
The logistical impacts from that would yield plenty of storytelling material: If you want to travel back in time, you need some ancient cellar that has been undisturbed since the target timeframe. If you want to skip forward, you need to establish that cellar, and round trips are limited by the space available.
That is - essentially - how 2002's _The Time Machine_ showed travel: Alexander's machine was 'stationary' on the earth, but time passed around him in a massively accelerated manner
The logistical impacts from that would yield plenty of storytelling material: If you want to travel back in time, you need some ancient cellar that has been undisturbed since the target timeframe. If you want to skip forward, you need to establish that cellar, and round trips are limited by the space available.