Dear Bubble,
This is a tough one, but because I'm so old I have experienced exactly what you are going through. I took the chance and left.
I once worked for a little place that no longer exists. But before it went under I got an offer to work at another place for a lot more money and I took it. The job I moved to sucked. Eventually I left that job and went back to the first one. Yes, the money was worse, but I enjoyed the job more and had piece of mind.
Still, in your current situation it's very easy to get complacent and watch the years fly by. Here's what I'd do.
I would look at those other positions from which you are getting offers. Look at their hourly structure and benefits. The structured hours would allow you to gain on not only your budgeting of income but also your time for a social life, which was important enough for you to deem worthy of mention. Do not burn the bridges with your current job and be very forthright about why you are looking elsewhere. Let your boss know it's no reflection on the job, him as a boss or the business. You merely don't see the chances you require to grow as a person both individually and professionally. Let him know you've enjoyed your time and will remember them all fondly.
Then I'd go to the new position. If it goes badly you have paved the way to be able to move back to the current company in that or another capacity. I can't tell you how many times people have left places where I work only to come back later with higher pay and a better position. You are still a young man and can afford to take these chances now. Don't wait until you have a family and mortgage to figure it all out.
In the end it's up to you and your comfort level, but I say go for it!