no hard and fast rule here, but here are some guidelines:
1. If it's a classic that has stood the test of time I say hang in there until about page 50 to see if you can handle it. That's about the 1 1/2 hour mark on an audiobook. If you just can't bear it by page 50 then give up, life's too short, man, and maybe you'll pick it up again in a couple of years. It took me 3 bites of the cherry to read Middlemarch and in the end I was glad that I did.
2. If the author's persistent racism or antisemitism or homophobia etc. is just really annoying you then you should feel free to give up immediately. Celine got on my nerves so badly by page 10 of Death on the Instalment Plan that I gave the book up in French and then again years later I gave it up in English at about page 20. The guy's just a dickhead and I didn't want to spend time in his company.
3. If the author's politics beats you over the head also feel free to give up the book. John Le Carre has always been a Manichean left wing ideologue of the Chomsky school but lately his formula (America + Britain = evil) has become tedious and thus his books are predictable and dull. Shame because Smiley is one of the all time great creations. Similarly with other authors who inject their dimwitted political opinions into their novels from the left and right and you can barely read the text for the noise made by the grinding of axes. Feel free to jump ship early.
4. A beach book that "everybody" loves but you can't stand. Zeitgeist and luck usually explain the success of these books. And usually they're bloody terrible. Chapter 3 will suffice. Abandon ship with no guilt after that.
5. A Brief History of Time. I've heard so many people humble bragging about how they never finished A Brief History of Time (Charlie Rose for one) because it's so difficult. It isn't difficult at all. Just read it. Try some David Deutsch if you want difficult. 5 pages a day and you'll be done in two months.
6. The Booker Prize longlist. 5 pages in and you'll have the gist of most of these. I read the first five pages of every book on the longlist and the only two I liked were Marlon James's Seven Killings and A Little Life. At the Sydney Writers Festival I met Marlon James at breakfast and wanted to tell him how impressed I was by the boldness of his temporal shifts and the polyphony of his book but I was too shy and instead I asked him how he was getting his eggs done and he said scrambled.